[linux]rescan the hard drive after your vm got an expanded storage

The Tufin securechange vm got an expanded storage of 1TB more, however I cannot see it immediately with the fdisk -l command.

I need to rescan it using echo "1" > /sys/class/block/sda/device/rescan, then do fdisk -l, you will see the /dev/sda with the full disk (after expanded). It works immediately..

Advertisements
Posted in Linux | Tagged , , | Leave a comment

[python]Creating security group and inbound rule

This is the extension of Functions of aws automation.I have added some methods to create security groups and apply rules.

In addition to the functions/methods describe here, I have created 4 more methods to accomplish these:

  • Security group creation
  • Inbound rule creation to the security group.

    Demonstration

    create_security_group method
    This method create a security group which allows me to do inbound and outbound rules.

    def create_security_group(client, group_name, description, vpc_id):
        return client.create_security_group(GroupName=group_name, Description=description, VpcId=vpc_id)
    

    cidrip_list_collector method
    This method gets the list of ip address or subnet from user’s input, the user needs to specify the number of source addresses that are expected. The return result is type list.

    def cidrip_list_collector():
        results = []
        process_results = []
        stop = int(input("How many source ip address you want"))
        print("Press enter with empty response to quit.")
        for i in range(0,stop):
            ip = input("Source address:")
            if ip is not "":
                try:
                    ip_network(ip)
                    results.append(ip)
                except ValueError as e:
                    print(e)
            else:
                break
        # collecting the dictionary / dictionaries in a list
        for result in results:
            results_dict = {'CidrIp': result}
            process_results.append(results_dict)
        return process_results
    

    rule_form method
    This method gets these information from user:

    • protocol type either tcp or udp
    • port range, if it is only one port then the start and end ports are the same number. The port is integer type, hence need to use int() so that the list is converted from string to integer.
    • source ip range.This is a list of dictionary or dictionaries of ip addresses.

    The return type is list.

    def rule_form():
        port_list = []
        protocol_response = input("Protocol (tcp/udp)?:")
        if protocol_response.lower() == 'tcp':
            protocol = protocol_response.lower()
        elif protocol_response.lower() == 'udp':
            protocol = protocol_response.lower()
        else:
            print("Invalid choice, this field cannot be empty, hence default to tcp")
            protocol = 'tcp'
        port_range_response = input("Enter your port range, if only one port example 80, write 80,80, \r\n"
                                    "if it is a range like 90-100 write 90,100:").split(',')
        for index in port_range_response:
            port_list.append(index)
        ip_ranges_list = cidrip_list_collector()
        return [
            {
                'IpProtocol': protocol,
                'FromPort': int(port_list[0]),
                'ToPort': int(port_list[1]),
                'IpRanges': ip_ranges_list
            }
        ]
    

    create_inbound_rule method
    This method calls boto3’s authorize_security_group_ingress method.

    Test codes

    if __name__ == '__main__':
        ec2 = get_client('ec2')
        """:type : pyboto3.ec2"""
        vpc_ids = get_attribute_from_vpc('VpcId', ec2.describe_vpcs(Filters=filter)['Vpcs'])
        vpc_ids_menu = resource_menu(vpc_ids)
        while True:
            print(vpc_ids_menu)
            vpc_ids_choice = int(input("Choose a VPC:"))
            if vpc_ids_choice not in vpc_ids_menu.keys():
                print("Invalid choice")
                break
            else:
                if(input("Do you want to create security group?:").lower()) == 'y':
                    group_name = input("Security Group name?:")
                    description = input("Description:")
                    create_sc_response = create_security_group(ec2, group_name, description, vpc_ids_menu[vpc_ids_choice])
                    print("The security group {} is created for {}".format(create_sc_response['GroupId'],
                                                                           vpc_ids_menu[vpc_ids_choice]))
                    create_inbound_rule_response = create_inbound_rule(ec2, create_sc_response['GroupId'], rule_form())
                    print("Security group {} is updated with inbound rule.".format(create_sc_response['GroupId']))
                else:
                    print("Bye..")
                break
    

    A note about is and ==
    == comparison of its value whereas “is” comparison if two objects from the same instance if they are then the boolean is true.

Posted in Python, Scripting | Tagged , , , , , , | Leave a comment

[python]Making a list of dictionaries

Use case
I am trying to write a script for doing security group based on user’s input. the IpRanges is a list of dictionary. I intend to do a method which generate multiple dictionaries with the same key but different values, and put these dictionaries into a list.

ipaddress.ip_network method can evaluate both host ip address and ip subnet, if the value is not a valid subnet or ipv4 address an exception will be thrown, the collection will be based on only valid subnet or host ip address.

from ipaddress import ip_network
results = []
process_results = []
stop = int(input("how many source ip address you want"))
print("Press enter with empty response to quit.")
for i in range(0,stop):
    ip = input("Source address:")
    if ip is not "":
        try:
            ip_network(ip)
            results.append(ip)
        except ValueError as e:
            print(e)
    else:
        break
# collecting the dictionary / dictionaries in a list
for result in results:
    results_dict = {'ip': result}
    process_results.append(results_dict)
print(process_results)

Test case 1: mixture of ip address and subnets
Screenshot 2019-05-08 at 3.57.30 PM

Test case 2: mixture of ip address, subnet and empty input
Screenshot 2019-05-08 at 5.58.47 PM

Test case 3: mixture of ip address, invalid data and subnet
Screenshot 2019-05-08 at 6.00.27 PM
Screenshot 2019-05-08 at 6.01.09 PM

Posted in Python, Scripting | Tagged , , | Leave a comment

[python]Create keypair in EC2

This is a simple code using boto3.

To generate the key pair for EC2 here’s the simple code.

import boto3
from pprint import pprint

ec2 = boto3.client('ec2')
""":type : pyboto3.ec2"""
keyname = input('enter a key name:')
key_pair_creation_response = ec2.create_key_pair(KeyName=keyname)
pprint(keyname)
Posted in Python | Tagged , , , | Leave a comment

[python]Finding your internet gateway id in aws

On previous post, I have this function:

def find_internet_gateway_id(client, vpc_id):
    igw_id = ""
    igw_responses = client.describe_internet_gateways(Filters=filter)
    for i in igw_responses['InternetGateways']:
        for j in i['Attachments']:
            if vpc_id in j['VpcId']:
                igw_id = i['InternetGatewayId']
    return igw_id

This function looks for the internet gateway id associated with your vpc id.

Json response from describe_internet_gateways
The below response is from the describe_internet_gateways method in boto3, here’s the code snippet:

igw_responses = ec2_client.describe_internet_gateways(Filters=filter)
pprint(igw_responses)

The below is a dictionary of two main keys – InternetGateways and ResponseMetadata.

{'InternetGateways': [{'Attachments': [{'State': 'available',
                                        'VpcId': 'vpc-0d5a397499f3bc8b8'}],
                       'InternetGatewayId': 'igw-028b0d9cc6fda54fa',
                       'OwnerId': '203367053771',
                       'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_2'}]},
                      {'Attachments': [{'State': 'available',
                                        'VpcId': 'vpc-00b16be35cdd83960'}],
                       'InternetGatewayId': 'igw-0d05c62094670bdb4',
                       'OwnerId': '203367053771',
                       'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_3'}]},
                      {'Attachments': [{'State': 'available',
                                        'VpcId': 'vpc-072f78e3f64873a46'}],
                       'InternetGatewayId': 'igw-0fd6b4aea334c6555',
                       'OwnerId': '203367053771',
                       'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_1'}]}],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1863',
                                      'content-type': 'text/xml;charset=UTF-8',
                                      'date': 'Mon, 06 May 2019 10:17:59 GMT',
                                      'server': 'AmazonEC2'},
                      'HTTPStatusCode': 200,
                      'RequestId': '394e330b-0c24-4d8f-bcb5-d46d121d9578',
                      'RetryAttempts': 0}}

Now I am only interested in the InternetGateways key, hence by using pprint(igw_responses['InternetGateways'])
I got this:

[{'Attachments': [{'State': 'available', 'VpcId': 'vpc-0d5a397499f3bc8b8'}],
  'InternetGatewayId': 'igw-028b0d9cc6fda54fa',
  'OwnerId': '203367053771',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_2'}]},
 {'Attachments': [{'State': 'available', 'VpcId': 'vpc-00b16be35cdd83960'}],
  'InternetGatewayId': 'igw-0d05c62094670bdb4',
  'OwnerId': '203367053771',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_3'}]},
 {'Attachments': [{'State': 'available', 'VpcId': 'vpc-072f78e3f64873a46'}],
  'InternetGatewayId': 'igw-0fd6b4aea334c6555',
  'OwnerId': '203367053771',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_1'}]}]

The value is a list of three dictionaries, each dictionaries has these keys:

  • Attachments – a list of one dictionary.
  • InternetGatewayId – has a value of IGW ID, which is what i need.
  • OwnerId
  • Tags – a list of one dictionary. To modify the tag, you need create_tags method in boto3.

How do I retrieve the vpc_id?
To access the values of key Attachments, supposed I need to get the vpc_id of cyruslab_igw_2 which is the first item in the list, I need to do this.
igw_responses['InternetGateways'][0]['Attachments'][0]['VpcId']

  • The Top of the entire json structure has two keys: ‘InternetGateways’ and ‘ResponseMetadata’, hence to access the values of ‘InternetGateways’ I use igw_responses['InternetGateways'].
  • The value of the key ‘InternetGateways’ is a list of three dictionaries, ‘cyruslab_igw_2’ is the first item in the list. Hence to access this first item in the list I use igw_responses['InternetGateways][0]
  • In order to get the value of vpc_id, I need to access through the key ‘Attachments’, hence i use igw_responses['InternetGateways'][0]['Attachments'].
  • The value of key 'Attachments' only has ONE dictionary, hence the first item always starts from 0 in a list. I use igw_responses['InternetGateways'][0]['Attachments'][0].
  • Now only the key value pair of 'VpcId', I only want the value hence I access it through the key, I use igw_responses['InternetGateways'][0]['Attachments'][0]['VpcId'].
  • I need to match the value against the supplied value.

These explains the logic of this for...in... loop.

    for i in igw_responses['InternetGateways']:
        for j in i['Attachments']:
            if vpc_id in j['VpcId']:

The VpcId is like a locator to locate which dictionary matches my supplied VpcId, in this example is igw_responses['InternetGateways'][0]. This dictionary has a key 'InternetGatewayId', I only want the value of the key, hence I use igw_responses['InternetGateways'][0]['InternetGatewayId']. This explains the following return value of the function.

igw_id = i['InternetGatewayId']
    return igw_id

afterword
I have been doing network and network security for a decade, I have evolved from a cli fan boy to a scripting enthusiast, my first api experience was with xml data structure, json structure used to intimidate me, however after a few practise I realise working with json structure is easier than xml structure. Manipulating the xml tree with beautifulsoup4 is easier than xml module, however there are at times i need xml module, working with xml structure in python is really hard.... Henceforth if the rest api provides both json and xml, i will prefer to work with json to xml.

Posted in Python, Scripting | Tagged , , , , , | Leave a comment

[python] Functions for aws automation

This post is a documentation of the functions/methods in the python script that will help to automate the AWS.

Declaration

import boto3
from ipaddress import ip_network
from pprint import pprint


filter = [{'Name':'tag:Name', 'Values':['*']}]

The filter can be found in Automated Ramblings.

Resource connection
This function connects to the resource based on the one of your choosing, can be EC2 or S3. It uses the default connection setup by awscli. Which is why I do not need to put in region name, access key id and secret key id.

def get_client(resource_name):
    return boto3.client(resource_name)

Extracting values from VPC
This function looks for the value you are interested such as the vpcid and cidr block. This function returns a list.

def get_attribute_from_vpc(attrib_id, vpc_list):
    # resets the collector.
    collector = []
    for vpc in vpc_list:
        collector.append(vpc[attrib_id])
    return collector

Suggesting a list of subnets based on the cidr block in VPC
This function uses ipaddress module’s ip_network method to list the possible subnets.This function returns a list.

def suggest_subnets(supernet, prefix_len):
    # resets the subnets list
    subnets = []
    for subnet in ip_network(supernet).subnets(new_prefix=prefix_len):
        subnets.append(str(subnet))
    return subnets

Create subnet
This function helps to call the boto3 create_subnet method, I created this function for myself to see the creation requirement easier. It also helps to make my code easier to read and understand.

def create_subnet(client, vpc_id, subnet):
    return client.create_subnet(VpcId=vpc_id, CidrBlock=subnet)

Create tag base on user’s input
This tag is for use in create_tag method in boto3 to name my items in VPC.

def name_resource(chosen_name):
    tag = [
        {
            'Key': 'Name',
            'Value': chosen_name
        }
    ]
    return tag

The menu generator
This function not only generates menu, it’s main purpose is to organise the list into a dictionary, hence this function returns a dictionary. User’s input will be integer, and based on the integer typed in the value which is either a cidrblock or vpcid will be chosen, the value returned can also be used for methods that require cidrblock or vpcid.

def resource_menu(resource_list):
    collector = []
    for resource in resource_list:
        collector.append(resource)
    menu_dict = {}
    for i in range(len(collector)):
        j = i + 1
        menu_dict[j] = collector[i]
    return menu_dict

Create routing table
This function is for explicitly creating a routing table, AWS actually creates routing table after subnet is created. A newly created routing table is just a container, this routing table has to be associated with subnet in order to use, you can add other routes in this table such as a default route.

def create_rtbl(client, vpc_id):
    return client.create_route_table(VpcId=vpc_id)

Create route
This function is for adding new routes, it can be adding a destination subnet of 0.0.0.0/0 which matches all ip addresses usually for default gateway, and also to add the next hop. This function can be used to create default route to the internet gateway.

def create_route(client, rtbl_id, igw_id, dst_net):
    return client.create_route(RouteTableId=rtbl_id, DestinationCidrBlock=dst_net, GatewayId=igw_id)

Associate routing table to subnet
This function associates the created routing table to a subnet.

def associate_route_table(client, rtbl_id, subnet_id):
    return client.associate_route_table(RouteTableId=rtbl_id, SubnetId=subnet_id)

Find gateway id
This function helps to find the internet gateway id that is associated with your vpc id. I will document how the code works on next post.

def find_internet_gateway_id(client, vpc_id):
    igw_id = ""
    igw_responses = client.describe_internet_gateways(Filters=filter)
    for i in igw_responses['InternetGateways']:
        for j in i['Attachments']:
            if vpc_id in j['VpcId']:
                igw_id = i['InternetGatewayId']
    return igw_id

Testing codes: create subnet from suggested list
My code is tested under main, the purpose is to test each functions which I created and see if the expected results can be yielded.

if __name__ == '__main__':
    ec2 = get_client('ec2')
    """:type pyboto3.ec2"""
    vpc_ids = get_attribute_from_vpc('VpcId', ec2.describe_vpcs(Filters=filter)['Vpcs'])
    menu = resource_menu(vpc_ids)
    print(menu)
    choice = int(input("Choose an option:"))
    if choice not in menu.keys():
        print("Invalid choice!")
        print(menu)
    else:
        cidr_ids = get_attribute_from_vpc('CidrBlock', ec2.describe_vpcs(Filters=filter)['Vpcs'])
        cidr_menu = resource_menu(cidr_ids)
        prefix_len = int(input("Choose a prefix length:"))
        cidr_suggestions = suggest_subnets(cidr_menu[choice], prefix_len)
        cidr_suggestions_menu = resource_menu(cidr_suggestions)
        print(cidr_suggestions_menu)
        cidr_choice = int(input("Choose one of the suggestion:"))
        if cidr_choice not in cidr_suggestions_menu.keys():
            print("Invalid subnet! Choose only from the suggested ones.")
        else:
            create_subnet(ec2, menu[choice], cidr_suggestions_menu[cidr_choice])

This code when triggered will look like below.

Testing code: create subnet from suggested list, create routing table, put in a default route into routing table and associate the routing table to a subnet.

if __name__ == '__main__':
    ec2 = get_client('ec2')
    """:type : pyboto3.ec2"""
    vpc_ids = get_attribute_from_vpc('VpcId', ec2.describe_vpcs(Filters=filter)['Vpcs'])
    menu = resource_menu(vpc_ids)
    print(menu)
    choice = int(input("Choose an option:"))
    if choice not in menu.keys():
        print("Invalid choice!")
        print(menu)
    else:
        cidr_ids = get_attribute_from_vpc('CidrBlock', ec2.describe_vpcs(Filters=filter)['Vpcs'])
        cidr_menu = resource_menu(cidr_ids)
        prefix_len = int(input("Choose a prefix length:"))
        cidr_suggestions = suggest_subnets(cidr_menu[choice], prefix_len)
        cidr_suggestions_menu = resource_menu(cidr_suggestions)
        print(cidr_suggestions_menu)
        cidr_choice = int(input("Choose one of the suggestion:"))
        if cidr_choice not in cidr_suggestions_menu.keys():
            print("Invalid subnet! Choose only from the suggested ones.")
        else:
            create_subnet_response = create_subnet(ec2, menu[choice], cidr_suggestions_menu[cidr_choice])
            pprint("{} is created for {}".format(create_subnet_response['Subnet']['SubnetId'],
                                                 create_subnet_response['Subnet']['VpcId']))
            create_rtbl_response = create_rtbl(ec2, menu[choice])
            pprint("{} is created for {}".format(create_rtbl_response['RouteTable']['RouteTableId'],
                                                 create_rtbl_response['RouteTable']['VpcId']))
            create_route(ec2, create_rtbl_response['RouteTable']['RouteTableId'],
                         find_internet_gateway_id(ec2, menu[choice]),
                                                            '0.0.0.0/0')
            pprint("Default route table to IGW is created")
            associate_route_table(ec2, create_rtbl_response['RouteTable']['RouteTableId'],
                                  create_subnet_response['Subnet']['SubnetId'])
            pprint("{} is associated with {}".format(create_rtbl_response['RouteTable']['RouteTableId'],
                                  create_subnet_response['Subnet']['SubnetId']))

Results

Posted in Python, Scripting | Tagged , , , , , , | 1 Comment

[python]Creating subnets based on VPC you have chosen

Background
I am trying to test a few functions of my script for things that can be done after VPC was created. Things such as make subnet either public or private, attach IGW to route table. I have build up some general purpose functions so that I do not need to repeat the same thing over and over again, and also to make my scripting code easier to read.

What this script does is to list the available VPCs to requester, the requester will choose the VPC, then follow by a list of suggested subnets from the main CIDR in the VPC, user will only choose from the suggested list. Once the subnet is chosen, the script proceed to create the request to create subnet for the specific VPC.

Demonstration

Screenshot 2019-05-06 at 12.25.10 PM

Afterword
The script is still in progress, I intend to test a few functions before doing the main code. The main code right now is to test various functions for bugs. The ipaddress module is really cool module for ip address manipulation.

Posted in Python | Tagged , | Leave a comment