To compare between with and without threading, there are three routers from R1 to R3, the task is to save the show version on each router to all.txt.
Do the tasks without threading
The script runs sequentially on each device, and the total time is 15seconds.
Do the tasks with threading
The time taken to complete the tasks for three routers took 5seconds.
When is threading use?
Threading should be used when the task requires a wait before another task can start, and the tasks do not require high cpu usage. Threading fits the requirement for this scenario.
Connecting the router and writing the data to the file are blocking tasks, before the next same tasks can start and run.
threading runs the connection then executes the command get the data and save to the file all at the same time, this is known as concurrency however all these tasks that runs almost concurrently only has one process.
Synchronization primitive
Because the file all.txt is a shared resource as all threading need to save the data to the same file, in order to ensure the data saved to the file is not corrupted a lock is required to open the file and write the contents into it, after the write is finished the lock is then released for another thread to run the task again.
crypto.py
This script contains functions for creating a symmetric key to decrypt and encrypt contents and save to encrypted contents to the file.
from cryptography.fernet import Fernet
from os.path import exists
# This checks the existence of enc.key.
# which is the symmetric key for encryption and decryption.
# if there is no key create one, but previous encrypted data will be locked forever.
def check_key():
if not exists("enc.key"):
key = Fernet.generate_key()
with open("enc.key", "wb") as file:
file.write(key)
# This encrypts the data and save to a file.
def encrypt(filename, data):
key_byte = read_key()
key = Fernet(key_byte)
ciphertext = key.encrypt(data.encode('utf-8'))
with open(filename, "wb") as cred:
cred.write(ciphertext)
# This decrypts the data retrieved from the encrypted file.
# if the convert_dict is false returns the string.
# if the convert_dict is true then convert the string to dictionary.
def decrypt(filename, convert_dict=False):
key_byte = read_key()
key = Fernet(key_byte)
with open(filename, "rb") as file:
ciphertext = file.read()
# plaintext is the string from the byte slice.
plaintext = key.decrypt(ciphertext).decode('utf-8')
if convert_dict:
# string cannot be converted to dict directly.
# first change the string to json type..
# then convert the json type to dict() object.
import json
return dict(json.loads(plaintext))
else:
return plaintext
# Use the key, the key is a file.
# get the key from the file and load in memory.
def read_key():
check_key()
with open("enc.key", "rb") as file:
key_byte = file.read()
return key_byte
network_threads.py subclass of threading
The network_threads.py is the subclass of threading.Thread class, this is for customizing the run method which is used to execute the task of connecting router and saving the contents to the file.
from netmiko import ConnectHandler
import threading
from time import sleep
class SaveNetworkData(threading.Thread):
def __init__(self, config):
# it can also be threading.Thread.__init__(self)
# super() refers to the parent class
# which is a requirement to have a threading subclass
super().__init__()
# The class parameter is to put in the dictionary config,
# which will be passed into netmiko.
self.config = config
# has to be def run(self), else will not work
def run(self):
# this pause of 0.2s is to ensure there is a short period of time
# before the next thread starts.
sleep(0.2)
# create a lock for the thread to access shared resource.
run_lock = threading.Lock()
with ConnectHandler(**self.config) as conn:
print(f"Connected to {self.config['ip']}")
# this will get the prompt of cisco router such as R1# or R1>
hostname = conn.find_prompt()
print(f"Got the prompt of {self.config['ip']}")
result = conn.send_command("show version")
print(f"Got the show version of {self.config['ip']}")
print("Preparing data and acquiring write lock...")
# concatenate all data into one data object.
# hostname[:-1] will only get the hostname without the > or #.
data = hostname[:-1] + "\n" + 10 * "*" + "\n" + result + "\n"
# Get the lock to access the console to print the message
# and open the all.txt file to write the data into it.
run_lock.acquire()
print(f"Write lock acquired by thread for {self.config['ip']} ")
print(f"Append data extracted from {self.config['ip']} to all.txt")
with open("all.txt", "a") as file:
file.write(data)
print("Done appending releasing write lock...")
# release the lock after data is saved.
run_lock.release()
config.json
This is the base config required for netmiko.
{
"ip": "",
"device_type": "cisco_ios"
}
main.py
The main.py which does the entire task.
from network_threads import SaveNetworkData
from security.crypto import decrypt
import json
from time import time
from os.path import exists
from os import remove
devices = [
"192.168.1.32",
"192.168.1.31",
"192.168.1.30"
]
config_list = list()
threads = list()
if __name__ == "__main__":
if exists("all.txt"):
remove("all.txt")
start = time()
# cred is a dictionary of username and password.
cred = decrypt("cred", convert_dict=True)
with open("devices/config.json", "r") as j:
config = dict(json.load(j))
# adds on username and password dictionary to existing config dictionary.
config.update(cred)
for device in devices:
# copy the config dict to a tmp object.
# else append will not be changed.
tmp = config.copy()
tmp["ip"] = device
# adds on dictionary of device into the list.
config_list.append(tmp)
for cfg in config_list:
t = SaveNetworkData(cfg)
threads.append(t)
t.start()
for thread in threads:
thread.join()
print("All threads finished. Time executed: {} seconds.".format(time() - start))
cyruslab 