How do Python imports and __init__.py files work in a package

Python packages typically utilize mechanisms like imports, __init__.py files, and inheritance to create scaleable, and reusable code. In this post, we will look at how these mechanisms work by exploring a piece of code that uses the Netmiko library.

The Netmiko documentation contains this simple example that we will use:

from netmiko import Netmiko

net_connect = Netmiko(
    "cisco1.twb-tech.com",
    username="pyclass",
    password=”pypassword”,
    device_type="cisco_ios",
)

print(net_connect.find_prompt())
net_connect.disconnect()

To summarize the steps in the example:

  • Netmiko library is imported
  • An instance of the Netmiko class is instantiated
  • The instance is used to connect to “cisco1.twb-tech.com”, find the prompt and print the prompt.
  • Netmiko disconnects from “cisco1.twb-tech.com”

Remember: the Netmiko library must be installed using a package manager like pip before it can be used.

Let’s examine each step.

from netmiko import Netmiko

When the Python interpreter sees this import statement it searches through the list of folders in the PYTHONPATH environment variable looking for a folder named “netmiko”. When it finds the folder it looks inside the folder for a file named Netmiko.
When we installed the Netmiko library, a folder called “netmiko” was created inside “site-packages”. One of the default entries in PYTHONPATH is the “site-packages” folder. Therefore the interpreter is able to find the netmiko folder.
But what about the Netmiko file? When we look inside the netmiko folder we do not see anything called “Netmiko”:

/usr/local/lib/python3.6/site-packages/netmiko > ls
__init__.py			citrix				paloalto
__pycache__			coriant				pluribus
_textfsm			dell				py23_compat.py
a10				eltex				quanta
accedian			enterasys			ruckus
alcatel				extreme				scp_functions.py
apresia				f5				scp_handler.py
arista				fortinet			snmp_autodetect.py
aruba				hp				ssh_autodetect.py
avaya				huawei				ssh_dispatcher.py
base_connection.py		juniper				ssh_exception.py
brocade				linux				terminal_server
calix				mellanox			ubiquiti
checkpoint			mrv				utilities.py
ciena				netapp				vyos
cisco				netmiko_globals.py
cisco_base_connection.py	ovs
/usr/local/lib/python3.6/site-packages/netmiko >

How does Python import Netmiko from netmiko when there is no file named Netmiko in the folder?
When the interpreter sees the “from netmiko import………” statement it first looks for an __init__.py file in the netmiko folder and if one exists it executes it.
We can see from the output above that an __init__.py does exist inside the netmiko folder. Therefore the __init__ file is found and executed.
Here is the netmiko.__init__.py file:

from __future__ import unicode_literals
import logging

# Logging configuration
log = logging.getLogger(__name__) # noqa
log.addHandler(logging.NullHandler()) # noqa

from netmiko.ssh_dispatcher import ConnectHandler
from netmiko.ssh_dispatcher import ssh_dispatcher
from netmiko.ssh_dispatcher import redispatch
from netmiko.ssh_dispatcher import platforms
from netmiko.ssh_dispatcher import FileTransfer
from netmiko.scp_handler import SCPConn
from netmiko.cisco.cisco_ios import InLineTransfer
from netmiko.ssh_exception import NetMikoTimeoutException
from netmiko.ssh_exception import NetMikoAuthenticationException
from netmiko.ssh_autodetect import SSHDetect
from netmiko.base_connection import BaseConnection
from netmiko.scp_functions import file_transfer

# Alternate naming
NetmikoTimeoutError = NetMikoTimeoutException
NetmikoAuthError = NetMikoAuthenticationException
Netmiko = ConnectHandler

__version__ = '2.2.2'
__all__ = ('ConnectHandler', 'ssh_dispatcher', 'platforms', 'SCPConn', 'FileTransfer',
           'NetMikoTimeoutException', 'NetMikoAuthenticationException',
           'NetmikoTimeoutError', 'NetmikoAuthError', 'InLineTransfer', 'redispatch',
           'SSHDetect', 'BaseConnection', 'Netmiko', 'file_transfer')

# Cisco cntl-shift-six sequence
CNTL_SHIFT_6 = chr(30)

When the netmiko.__init__ file executes all its import statements execute. One of these statements imports the “ConnectHandler” function from the netmiko.ssh_dispatcher module. 

from netmiko.ssh_dispatcher import ConnectHandler

Then in the “# Alternate naming” section of the netmiko.__init__.py file, the imported ConnectHandler function is assigned to the name Netmiko. 

Netmiko = ConnectHandler

This creates the naming that the interpreter is able to use to execute the “from netmiko import Netmiko” statement.

At this point, the ConnectHandler function has been imported as Netmiko and all the other imports in netmiko.__init__.py have completed.

Before we move on to the next step in the example code, there are some other things to consider. Whenever a module or a function from a module is imported the whole module is executed. This means that when netmiko.__init__.py imports the ConnectHandler function from ssh_dispatcher, ssh_dispatcher is executed. The ssh_dispatcher module contains many import statements, all of which are executed.
Here is the ssh_dispatcher module:

"""Controls selection of proper class based on the device type."""
from __future__ import unicode_literals

from netmiko.a10 import A10SSH
from netmiko.accedian import AccedianSSH
from netmiko.alcatel import AlcatelAosSSH
from netmiko.alcatel import AlcatelSrosSSH
from netmiko.arista import AristaSSH, AristaTelnet
from netmiko.arista import AristaFileTransfer
from netmiko.apresia import ApresiaAeosSSH, ApresiaAeosTelnet
from netmiko.aruba import ArubaSSH
from netmiko.avaya import AvayaErsSSH
from netmiko.avaya import AvayaVspSSH
from netmiko.brocade import BrocadeNetironSSH
from netmiko.brocade import BrocadeNetironTelnet
from netmiko.brocade import BrocadeNosSSH
from netmiko.calix import CalixB6SSH, CalixB6Telnet
from netmiko.checkpoint import CheckPointGaiaSSH
from netmiko.ciena import CienaSaosSSH
from netmiko.cisco import CiscoAsaSSH, CiscoAsaFileTransfer
from netmiko.cisco import CiscoIosSSH, CiscoIosFileTransfer, CiscoIosTelnet, CiscoIosSerial
from netmiko.cisco import CiscoNxosSSH, CiscoNxosFileTransfer
from netmiko.cisco import CiscoS300SSH
from netmiko.cisco import CiscoTpTcCeSSH
from netmiko.cisco import CiscoWlcSSH
from netmiko.cisco import CiscoXrSSH, CiscoXrFileTransfer
from netmiko.citrix import NetscalerSSH
from netmiko.coriant import CoriantSSH
from netmiko.dell import DellForce10SSH
from netmiko.dell import DellOS10SSH, DellOS10FileTransfer
from netmiko.dell import DellPowerConnectSSH
from netmiko.dell import DellPowerConnectTelnet
from netmiko.dell import DellIsilonSSH
from netmiko.eltex import EltexSSH
from netmiko.enterasys import EnterasysSSH
from netmiko.extreme import ExtremeSSH
from netmiko.extreme import ExtremeWingSSH
from netmiko.extreme import ExtremeTelnet
from netmiko.f5 import F5LtmSSH
from netmiko.fortinet import FortinetSSH
from netmiko.hp import HPProcurveSSH, HPProcurveTelnet, HPComwareSSH, HPComwareTelnet
from netmiko.huawei import HuaweiSSH, HuaweiVrpv8SSH
from netmiko.juniper import JuniperSSH, JuniperTelnet
from netmiko.juniper import JuniperFileTransfer
from netmiko.linux import LinuxSSH, LinuxFileTransfer
from netmiko.mellanox import MellanoxSSH
from netmiko.mrv import MrvOptiswitchSSH
from netmiko.netapp import NetAppcDotSSH
from netmiko.ovs import OvsLinuxSSH
from netmiko.paloalto import PaloAltoPanosSSH
from netmiko.pluribus import PluribusSSH
from netmiko.quanta import QuantaMeshSSH
from netmiko.ruckus import RuckusFastironSSH
from netmiko.ruckus import RuckusFastironTelnet
from netmiko.terminal_server import TerminalServerSSH
from netmiko.terminal_server import TerminalServerTelnet
from netmiko.ubiquiti import UbiquitiEdgeSSH
from netmiko.vyos import VyOSSSH


# The keys of this dictionary are the supported device_types
CLASS_MAPPER_BASE = {
    'a10': A10SSH,
    'accedian': AccedianSSH,
    'alcatel_aos': AlcatelAosSSH,
    'alcatel_sros': AlcatelSrosSSH,
    'apresia_aeos': ApresiaAeosSSH,
    'arista_eos': AristaSSH,
    'aruba_os': ArubaSSH,
    'avaya_ers': AvayaErsSSH,
    'avaya_vsp': AvayaVspSSH,
    'brocade_fastiron': RuckusFastironSSH,
    'brocade_netiron': BrocadeNetironSSH,
    'brocade_nos': BrocadeNosSSH,
    'brocade_vdx': BrocadeNosSSH,
    'brocade_vyos': VyOSSSH,
    'checkpoint_gaia': CheckPointGaiaSSH,
    'calix_b6': CalixB6SSH,
    'ciena_saos': CienaSaosSSH,
    'cisco_asa': CiscoAsaSSH,
    'cisco_ios': CiscoIosSSH,
    'cisco_nxos': CiscoNxosSSH,
    'cisco_s300': CiscoS300SSH,
    'cisco_tp': CiscoTpTcCeSSH,
    'cisco_wlc': CiscoWlcSSH,
    'cisco_xe': CiscoIosSSH,
    'cisco_xr': CiscoXrSSH,
    'coriant': CoriantSSH,
    'dell_force10': DellForce10SSH,
    'dell_os10': DellOS10SSH,
    'dell_powerconnect': DellPowerConnectSSH,
    'dell_isilon': DellIsilonSSH,
    'eltex': EltexSSH,
    'enterasys': EnterasysSSH,
    'extreme': ExtremeSSH,
    'extreme_wing': ExtremeWingSSH,
    'f5_ltm': F5LtmSSH,
    'fortinet': FortinetSSH,
    'generic_termserver': TerminalServerSSH,
    'hp_comware': HPComwareSSH,
    'hp_procurve': HPProcurveSSH,
    'huawei': HuaweiSSH,
    'huawei_vrpv8': HuaweiVrpv8SSH,
    'juniper': JuniperSSH,
    'juniper_junos': JuniperSSH,
    'linux': LinuxSSH,
    'mellanox': MellanoxSSH,
    'mrv_optiswitch': MrvOptiswitchSSH,
    'netapp_cdot': NetAppcDotSSH,
    'netscaler': NetscalerSSH,
    'ovs_linux': OvsLinuxSSH,
    'paloalto_panos': PaloAltoPanosSSH,
    'pluribus': PluribusSSH,
    'quanta_mesh': QuantaMeshSSH,
    'ruckus_fastiron': RuckusFastironSSH,
    'ubiquiti_edge': UbiquitiEdgeSSH,
    'ubiquiti_edgeswitch': UbiquitiEdgeSSH,
    'vyatta_vyos': VyOSSSH,
    'vyos': VyOSSSH,
}

FILE_TRANSFER_MAP = {
    'arista_eos': AristaFileTransfer,
    'cisco_asa': CiscoAsaFileTransfer,
    'cisco_ios': CiscoIosFileTransfer,
    'dell_os10': DellOS10FileTransfer,
    'cisco_nxos': CiscoNxosFileTransfer,
    'cisco_xe': CiscoIosFileTransfer,
    'cisco_xr': CiscoXrFileTransfer,
    'juniper_junos': JuniperFileTransfer,
    'linux': LinuxFileTransfer,
}

# Also support keys that end in _ssh
new_mapper = {}
for k, v in CLASS_MAPPER_BASE.items():
    new_mapper[k] = v
    alt_key = k + u"_ssh"
    new_mapper[alt_key] = v
CLASS_MAPPER = new_mapper

new_mapper = {}
for k, v in FILE_TRANSFER_MAP.items():
    new_mapper[k] = v
    alt_key = k + u"_ssh"
    new_mapper[alt_key] = v
FILE_TRANSFER_MAP = new_mapper

# Add telnet drivers
CLASS_MAPPER['brocade_fastiron_telnet'] = RuckusFastironTelnet
CLASS_MAPPER['brocade_netiron_telnet'] = BrocadeNetironTelnet
CLASS_MAPPER['cisco_ios_telnet'] = CiscoIosTelnet
CLASS_MAPPER['apresia_aeos_telnet'] = ApresiaAeosTelnet
CLASS_MAPPER['arista_eos_telnet'] = AristaTelnet
CLASS_MAPPER['hp_procurve_telnet'] = HPProcurveTelnet
CLASS_MAPPER['hp_comware_telnet'] = HPComwareTelnet
CLASS_MAPPER['juniper_junos_telnet'] = JuniperTelnet
CLASS_MAPPER['calix_b6_telnet'] = CalixB6Telnet
CLASS_MAPPER['dell_powerconnect_telnet'] = DellPowerConnectTelnet
CLASS_MAPPER['generic_termserver_telnet'] = TerminalServerTelnet
CLASS_MAPPER['extreme_telnet'] = ExtremeTelnet
CLASS_MAPPER['ruckus_fastiron_telnet'] = RuckusFastironTelnet

# Add serial drivers
CLASS_MAPPER['cisco_ios_serial'] = CiscoIosSerial

# Add general terminal_server driver and autodetect
CLASS_MAPPER['terminal_server'] = TerminalServerSSH
CLASS_MAPPER['autodetect'] = TerminalServerSSH

platforms = list(CLASS_MAPPER.keys())
platforms.sort()
platforms_base = list(CLASS_MAPPER_BASE.keys())
platforms_base.sort()
platforms_str = "\n".join(platforms_base)
platforms_str = "\n" + platforms_str

scp_platforms = list(FILE_TRANSFER_MAP.keys())
scp_platforms.sort()
scp_platforms_str = "\n".join(scp_platforms)
scp_platforms_str = "\n" + scp_platforms_str


def ConnectHandler(*args, **kwargs):
    """Factory function selects the proper class and creates object based on device_type."""
    if kwargs['device_type'] not in platforms:
        raise ValueError('Unsupported device_type: '
                         'currently supported platforms are: {}'.format(platforms_str))
    ConnectionClass = ssh_dispatcher(kwargs['device_type'])
    return ConnectionClass(*args, **kwargs)


def ssh_dispatcher(device_type):
    """Select the class to be instantiated based on vendor/platform."""
    return CLASS_MAPPER[device_type]


def redispatch(obj, device_type, session_prep=True):
    """Dynamically change Netmiko object's class to proper class.

    Generally used with terminal_server device_type when you need to redispatch after interacting
    with terminal server.
    """
    new_class = ssh_dispatcher(device_type)
    obj.device_type = device_type
    obj.__class__ = new_class
    if session_prep:
        obj.session_preparation()


def FileTransfer(*args, **kwargs):
    """Factory function selects the proper SCP class and creates object based on device_type."""
    if len(args) >= 1:
        device_type = args[0].device_type
    else:
        device_type = kwargs['ssh_conn'].device_type
    if device_type not in scp_platforms:
        raise ValueError('Unsupported SCP device_type: '
                         'currently supported platforms are: {}'.format(scp_platforms_str))
    FileTransferClass = FILE_TRANSFER_MAP[device_type]
    return FileTransferClass(*args, **kwargs)

Several of these statements import from the netmiko.cisco module. As before, when this import happens, the interpreter first looks for an __init__.py file in the netmiko.cisco folder. The __init__.py file in netmiko.cisco is found and executed.
Here is the netmiko.cisco.__init__.py file from the netmiko.cisco folder:

from __future__ import unicode_literals
from netmiko.cisco.cisco_ios import CiscoIosBase, CiscoIosSSH, CiscoIosTelnet, CiscoIosSerial
from netmiko.cisco.cisco_ios import CiscoIosFileTransfer
from netmiko.cisco.cisco_ios import InLineTransfer
from netmiko.cisco.cisco_asa_ssh import CiscoAsaSSH, CiscoAsaFileTransfer
from netmiko.cisco.cisco_nxos_ssh import CiscoNxosSSH, CiscoNxosFileTransfer
from netmiko.cisco.cisco_xr_ssh import CiscoXrSSH, CiscoXrFileTransfer
from netmiko.cisco.cisco_wlc_ssh import CiscoWlcSSH
from netmiko.cisco.cisco_s300 import CiscoS300SSH
from netmiko.cisco.cisco_tp_tcce import CiscoTpTcCeSSH

__all__ = ['CiscoIosSSH', 'CiscoIosTelnet', 'CiscoAsaSSH', 'CiscoNxosSSH', 'CiscoXrSSH',
           'CiscoWlcSSH', 'CiscoS300SSH', 'CiscoTpTcCeSSH', 'CiscoIosBase',
           'CiscoIosFileTransfer', 'InLineTransfer', 'CiscoAsaFileTransfer',
           'CiscoNxosFileTransfer', 'CiscoIosSerial', 'CiscoXrFileTransfer']

The netmiko.cisco.__init__.py file contains multiple import statements. One of these statements imports the CiscoIosSSH class from netmiko.cisco.cisco_ios. 

from netmiko.cisco.cisco_ios import CiscoIosBase, CiscoIosSSH, CiscoIosTelnet, CiscoIosSerial

Here is the start of netmiko.cisco.cisco_ios module containing the CiscoIosSSH class:

from __future__ import unicode_literals

import time
import re
import os
import hashlib
import io

from netmiko.cisco_base_connection import CiscoBaseConnection, CiscoFileTransfer


class CiscoIosBase(CiscoBaseConnection):
    """Common Methods for IOS (both SSH and telnet)."""
    def session_preparation(self):
        """Prepare the session after the connection has been established."""
        self._test_channel_read(pattern=r'[>#]')
        self.set_base_prompt()
        self.disable_paging()
        self.set_terminal_width(command='terminal width 511')
        # Clear the read buffer
        time.sleep(.3 * self.global_delay_factor)
        self.clear_buffer()

    def save_config(self, cmd='write mem', confirm=False):
        """Saves Config Using Copy Run Start"""
        return super(CiscoIosBase, self).save_config(cmd=cmd, confirm=confirm)


class CiscoIosSSH(CiscoIosBase):
    """Cisco IOS SSH driver."""
    pass

<truncated>

The CiscoIosSSH class inherits from CiscoIosBase class (also in netmiko.cisco.cisco_ios). 

class CiscoIosSSH(CiscoIosBase):
    """Cisco IOS SSH driver."""
    pass

CiscoIosBase class inherits from the CiscoBaseConnection class.

class CiscoIosBase(CiscoBaseConnection):
    """Common Methods for IOS (both SSH and telnet)."""
    def session_preparation(self):
        """Prepare the session after the connection has been established."""
        self._test_channel_read(pattern=r'[>#]')
        self.set_base_prompt()
        self.disable_paging()
        self.set_terminal_width(command='terminal width 511')
        # Clear the read buffer
        time.sleep(.3 * self.global_delay_factor)
        self.clear_buffer()

    def save_config(self, cmd='write mem', confirm=False):
        """Saves Config Using Copy Run Start"""
        return super(CiscoIosBase, self).save_config(cmd=cmd, confirm=confirm)

CiscoBaseConnection is imported from the netmiko.cisco_base_connection module

from netmiko.cisco_base_connection import CiscoBaseConnection, CiscoFileTransfer

Here is the start of netmiko.cisco_base_connection module containing the CiscoBaseConnection class:

"""CiscoBaseConnection is netmiko SSH class for Cisco and Cisco-like platforms."""
from __future__ import unicode_literals
from netmiko.base_connection import BaseConnection
from netmiko.scp_handler import BaseFileTransfer
from netmiko.ssh_exception import NetMikoAuthenticationException
import re
import time


class CiscoBaseConnection(BaseConnection):
    """Base Class for cisco-like behavior."""
    def check_enable_mode(self, check_string='#'):
        """Check if in enable mode. Return boolean."""
        return super(CiscoBaseConnection, self).check_enable_mode(check_string=check_string)

    def enable(self, cmd='enable', pattern='ssword', re_flags=re.IGNORECASE):
        """Enter enable mode."""
        return super(CiscoBaseConnection, self).enable(cmd=cmd, pattern=pattern, re_flags=re_flags)

<truncated>

The CiscoBaseConnection class inherits from the BaseConnection class (see above)

The BaseConnection class is imported from the netmiko.base_connection module.

from netmiko.base_connection import BaseConnection

Here is the start of the netmiko.base_connection module containing the start of the BaseConnection class:

'''
Base connection class for netmiko

Handles SSH connection and methods that are generically applicable to different
platforms (Cisco and non-Cisco).

Also defines methods that should generally be supported by child classes
'''

from __future__ import print_function
from __future__ import unicode_literals

import io
import re
import socket
import telnetlib
import time
from os import path
from threading import Lock

import paramiko
import serial

from netmiko import log
from netmiko.netmiko_globals import MAX_BUFFER, BACKSPACE_CHAR
from netmiko.py23_compat import string_types, bufferedio_types
from netmiko.ssh_exception import NetMikoTimeoutException, NetMikoAuthenticationException
from netmiko.utilities import write_bytes, check_serial_port, get_structured_data


class BaseConnection(object):
    """
    Defines vendor independent methods.

    Otherwise method left as a stub method.
    """
    def __init__(self, ip='', host='', username='', password='', secret='', port=None,
                 device_type='', verbose=False, global_delay_factor=1, use_keys=False,
                 key_file=None, allow_agent=False, ssh_strict=False, system_host_keys=False,
                 alt_host_keys=False, alt_key_file='', ssh_config_file=None, timeout=100,
                 session_timeout=60, blocking_timeout=8, keepalive=0, default_enter=None,
                 response_return=None, serial_settings=None, fast_cli=False, session_log=None,
                 session_log_record_writes=False, session_log_file_mode='write'):
        """

<truncated>

Now that the cascading imports and class inheritance have completed we have an instance of CiscoIosSSH called net_connect with all the methods from all the inherited classes. This will be important for the next step.

With all of these imports considered we are now ready to look at the next step in the example code:

net_connect = Netmiko(
    "cisco1.twb-tech.com",
    username="pyclass",
    password=”pypassword”,
    device_type="cisco_ios",
)

As explained earlier the alternate naming of ConnectHandler, means that when net_connect = Netmiko(….) is executed it is actually the ConnectHandler function that is executed. The four arguments including “device_type=”cisco_ios” are passed to ConnectHandler. ConnectHandler uses the “device_type=”cisco_ios” argument to select, instantiate and return a new instance of the CiscoIosSSH(…..) class.
Remember that CiscoIosSSH class was imported in the previous step.

As we have seen the CiscoIosSSH class is part of the following inheritance hierarchy:

CiscoIosSSH<-CiscoIosBase<-CiscoBaseConnection<-BaseConnection

This means that when CiscoIosSSH is instantiated the BaseConnection class __init__ () constructor method is executed. The BaseConnection class __init__() constructor method executes, forms a connection to the device and allows the final 2 steps in the example code to complete:

print(net_connect.find_prompt())
net_connect.disconnect()

We will not go into the details of how the final 2 steps complete because it is of little relevance to this post.

In conclusion, we have seen that even with a relatively small package like Netmiko and with a simple piece of code there is a lot going on behind the scenes. A single import command can cause a chain reaction of imports, __init__.py file executions and inheritance.

For a detailed explanation of Python imports visit:
https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html

Leave a Reply

Your email address will not be published. Required fields are marked *