[python]Finding your internet gateway id in aws

On previous post, I have this function:

def find_internet_gateway_id(client, vpc_id):
    igw_id = ""
    igw_responses = client.describe_internet_gateways(Filters=filter)
    for i in igw_responses['InternetGateways']:
        for j in i['Attachments']:
            if vpc_id in j['VpcId']:
                igw_id = i['InternetGatewayId']
    return igw_id

This function looks for the internet gateway id associated with your vpc id.

Json response from describe_internet_gateways
The below response is from the describe_internet_gateways method in boto3, here’s the code snippet:

igw_responses = ec2_client.describe_internet_gateways(Filters=filter)
pprint(igw_responses)

The below is a dictionary of two main keys – InternetGateways and ResponseMetadata.

{'InternetGateways': [{'Attachments': [{'State': 'available',
                                        'VpcId': 'vpc-0d5a397499f3bc8b8'}],
                       'InternetGatewayId': 'igw-028b0d9cc6fda54fa',
                       'OwnerId': '203367053771',
                       'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_2'}]},
                      {'Attachments': [{'State': 'available',
                                        'VpcId': 'vpc-00b16be35cdd83960'}],
                       'InternetGatewayId': 'igw-0d05c62094670bdb4',
                       'OwnerId': '203367053771',
                       'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_3'}]},
                      {'Attachments': [{'State': 'available',
                                        'VpcId': 'vpc-072f78e3f64873a46'}],
                       'InternetGatewayId': 'igw-0fd6b4aea334c6555',
                       'OwnerId': '203367053771',
                       'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_1'}]}],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1863',
                                      'content-type': 'text/xml;charset=UTF-8',
                                      'date': 'Mon, 06 May 2019 10:17:59 GMT',
                                      'server': 'AmazonEC2'},
                      'HTTPStatusCode': 200,
                      'RequestId': '394e330b-0c24-4d8f-bcb5-d46d121d9578',
                      'RetryAttempts': 0}}

Now I am only interested in the InternetGateways key, hence by using pprint(igw_responses['InternetGateways'])
I got this:

[{'Attachments': [{'State': 'available', 'VpcId': 'vpc-0d5a397499f3bc8b8'}],
  'InternetGatewayId': 'igw-028b0d9cc6fda54fa',
  'OwnerId': '203367053771',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_2'}]},
 {'Attachments': [{'State': 'available', 'VpcId': 'vpc-00b16be35cdd83960'}],
  'InternetGatewayId': 'igw-0d05c62094670bdb4',
  'OwnerId': '203367053771',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_3'}]},
 {'Attachments': [{'State': 'available', 'VpcId': 'vpc-072f78e3f64873a46'}],
  'InternetGatewayId': 'igw-0fd6b4aea334c6555',
  'OwnerId': '203367053771',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_igw_1'}]}]

The value is a list of three dictionaries, each dictionaries has these keys:

  • Attachments – a list of one dictionary.
  • InternetGatewayId – has a value of IGW ID, which is what i need.
  • OwnerId
  • Tags – a list of one dictionary. To modify the tag, you need create_tags method in boto3.

How do I retrieve the vpc_id?
To access the values of key Attachments, supposed I need to get the vpc_id of cyruslab_igw_2 which is the first item in the list, I need to do this.
igw_responses['InternetGateways'][0]['Attachments'][0]['VpcId']

  • The Top of the entire json structure has two keys: ‘InternetGateways’ and ‘ResponseMetadata’, hence to access the values of ‘InternetGateways’ I use igw_responses['InternetGateways'].
  • The value of the key ‘InternetGateways’ is a list of three dictionaries, ‘cyruslab_igw_2’ is the first item in the list. Hence to access this first item in the list I use igw_responses['InternetGateways][0]
  • In order to get the value of vpc_id, I need to access through the key ‘Attachments’, hence i use igw_responses['InternetGateways'][0]['Attachments'].
  • The value of key 'Attachments' only has ONE dictionary, hence the first item always starts from 0 in a list. I use igw_responses['InternetGateways'][0]['Attachments'][0].
  • Now only the key value pair of 'VpcId', I only want the value hence I access it through the key, I use igw_responses['InternetGateways'][0]['Attachments'][0]['VpcId'].
  • I need to match the value against the supplied value.

These explains the logic of this for...in... loop.

    for i in igw_responses['InternetGateways']:
        for j in i['Attachments']:
            if vpc_id in j['VpcId']:

The VpcId is like a locator to locate which dictionary matches my supplied VpcId, in this example is igw_responses['InternetGateways'][0]. This dictionary has a key 'InternetGatewayId', I only want the value of the key, hence I use igw_responses['InternetGateways'][0]['InternetGatewayId']. This explains the following return value of the function.

igw_id = i['InternetGatewayId']
    return igw_id

afterword
I have been doing network and network security for a decade, I have evolved from a cli fan boy to a scripting enthusiast, my first api experience was with xml data structure, json structure used to intimidate me, however after a few practise I realise working with json structure is easier than xml structure. Manipulating the xml tree with beautifulsoup4 is easier than xml module, however there are at times i need xml module, working with xml structure in python is really hard.... Henceforth if the rest api provides both json and xml, i will prefer to work with json to xml.

Advertisements
Posted in Python, Scripting | Tagged , , , , , | Leave a comment

[python] Functions for aws automation

This post is a documentation of the functions/methods in the python script that will help to automate the AWS.

Declaration

import boto3
from ipaddress import ip_network
from pprint import pprint


filter = [{'Name':'tag:Name', 'Values':['*']}]

The filter can be found in Automated Ramblings.

Resource connection
This function connects to the resource based on the one of your choosing, can be EC2 or S3. It uses the default connection setup by awscli. Which is why I do not need to put in region name, access key id and secret key id.

def get_client(resource_name):
    return boto3.client(resource_name)

Extracting values from VPC
This function looks for the value you are interested such as the vpcid and cidr block. This function returns a list.

def get_attribute_from_vpc(attrib_id, vpc_list):
    # resets the collector.
    collector = []
    for vpc in vpc_list:
        collector.append(vpc[attrib_id])
    return collector

Suggesting a list of subnets based on the cidr block in VPC
This function uses ipaddress module’s ip_network method to list the possible subnets.This function returns a list.

def suggest_subnets(supernet, prefix_len):
    # resets the subnets list
    subnets = []
    for subnet in ip_network(supernet).subnets(new_prefix=prefix_len):
        subnets.append(str(subnet))
    return subnets

Create subnet
This function helps to call the boto3 create_subnet method, I created this function for myself to see the creation requirement easier. It also helps to make my code easier to read and understand.

def create_subnet(client, vpc_id, subnet):
    return client.create_subnet(VpcId=vpc_id, CidrBlock=subnet)

Create tag base on user’s input
This tag is for use in create_tag method in boto3 to name my items in VPC.

def name_resource(chosen_name):
    tag = [
        {
            'Key': 'Name',
            'Value': chosen_name
        }
    ]
    return tag

The menu generator
This function not only generates menu, it’s main purpose is to organise the list into a dictionary, hence this function returns a dictionary. User’s input will be integer, and based on the integer typed in the value which is either a cidrblock or vpcid will be chosen, the value returned can also be used for methods that require cidrblock or vpcid.

def resource_menu(resource_list):
    collector = []
    for resource in resource_list:
        collector.append(resource)
    menu_dict = {}
    for i in range(len(collector)):
        j = i + 1
        menu_dict[j] = collector[i]
    return menu_dict

Create routing table
This function is for explicitly creating a routing table, AWS actually creates routing table after subnet is created. A newly created routing table is just a container, this routing table has to be associated with subnet in order to use, you can add other routes in this table such as a default route.

def create_rtbl(client, vpc_id):
    return client.create_route_table(VpcId=vpc_id)

Create route
This function is for adding new routes, it can be adding a destination subnet of 0.0.0.0/0 which matches all ip addresses usually for default gateway, and also to add the next hop. This function can be used to create default route to the internet gateway.

def create_route(client, rtbl_id, igw_id, dst_net):
    return client.create_route(RouteTableId=rtbl_id, DestinationCidrBlock=dst_net, GatewayId=igw_id)

Associate routing table to subnet
This function associates the created routing table to a subnet.

def associate_route_table(client, rtbl_id, subnet_id):
    return client.associate_route_table(RouteTableId=rtbl_id, SubnetId=subnet_id)

Find gateway id
This function helps to find the internet gateway id that is associated with your vpc id. I will document how the code works on next post.

def find_internet_gateway_id(client, vpc_id):
    igw_id = ""
    igw_responses = client.describe_internet_gateways(Filters=filter)
    for i in igw_responses['InternetGateways']:
        for j in i['Attachments']:
            if vpc_id in j['VpcId']:
                igw_id = i['InternetGatewayId']
    return igw_id

Testing codes: create subnet from suggested list
My code is tested under main, the purpose is to test each functions which I created and see if the expected results can be yielded.

if __name__ == '__main__':
    ec2 = get_client('ec2')
    """:type pyboto3.ec2"""
    vpc_ids = get_attribute_from_vpc('VpcId', ec2.describe_vpcs(Filters=filter)['Vpcs'])
    menu = resource_menu(vpc_ids)
    print(menu)
    choice = int(input("Choose an option:"))
    if choice not in menu.keys():
        print("Invalid choice!")
        print(menu)
    else:
        cidr_ids = get_attribute_from_vpc('CidrBlock', ec2.describe_vpcs(Filters=filter)['Vpcs'])
        cidr_menu = resource_menu(cidr_ids)
        prefix_len = int(input("Choose a prefix length:"))
        cidr_suggestions = suggest_subnets(cidr_menu[choice], prefix_len)
        cidr_suggestions_menu = resource_menu(cidr_suggestions)
        print(cidr_suggestions_menu)
        cidr_choice = int(input("Choose one of the suggestion:"))
        if cidr_choice not in cidr_suggestions_menu.keys():
            print("Invalid subnet! Choose only from the suggested ones.")
        else:
            create_subnet(ec2, menu[choice], cidr_suggestions_menu[cidr_choice])

This code when triggered will look like below.

Testing code: create subnet from suggested list, create routing table, put in a default route into routing table and associate the routing table to a subnet.

if __name__ == '__main__':
    ec2 = get_client('ec2')
    """:type : pyboto3.ec2"""
    vpc_ids = get_attribute_from_vpc('VpcId', ec2.describe_vpcs(Filters=filter)['Vpcs'])
    menu = resource_menu(vpc_ids)
    print(menu)
    choice = int(input("Choose an option:"))
    if choice not in menu.keys():
        print("Invalid choice!")
        print(menu)
    else:
        cidr_ids = get_attribute_from_vpc('CidrBlock', ec2.describe_vpcs(Filters=filter)['Vpcs'])
        cidr_menu = resource_menu(cidr_ids)
        prefix_len = int(input("Choose a prefix length:"))
        cidr_suggestions = suggest_subnets(cidr_menu[choice], prefix_len)
        cidr_suggestions_menu = resource_menu(cidr_suggestions)
        print(cidr_suggestions_menu)
        cidr_choice = int(input("Choose one of the suggestion:"))
        if cidr_choice not in cidr_suggestions_menu.keys():
            print("Invalid subnet! Choose only from the suggested ones.")
        else:
            create_subnet_response = create_subnet(ec2, menu[choice], cidr_suggestions_menu[cidr_choice])
            pprint("{} is created for {}".format(create_subnet_response['Subnet']['SubnetId'],
                                                 create_subnet_response['Subnet']['VpcId']))
            create_rtbl_response = create_rtbl(ec2, menu[choice])
            pprint("{} is created for {}".format(create_rtbl_response['RouteTable']['RouteTableId'],
                                                 create_rtbl_response['RouteTable']['VpcId']))
            create_route(ec2, create_rtbl_response['RouteTable']['RouteTableId'],
                         find_internet_gateway_id(ec2, menu[choice]),
                                                            '0.0.0.0/0')
            pprint("Default route table to IGW is created")
            associate_route_table(ec2, create_rtbl_response['RouteTable']['RouteTableId'],
                                  create_subnet_response['Subnet']['SubnetId'])
            pprint("{} is associated with {}".format(create_rtbl_response['RouteTable']['RouteTableId'],
                                  create_subnet_response['Subnet']['SubnetId']))

Results

Posted in Python, Scripting | Tagged , , , , , , | 1 Comment

[python]Creating subnets based on VPC you have chosen

Background
I am trying to test a few functions of my script for things that can be done after VPC was created. Things such as make subnet either public or private, attach IGW to route table. I have build up some general purpose functions so that I do not need to repeat the same thing over and over again, and also to make my scripting code easier to read.

What this script does is to list the available VPCs to requester, the requester will choose the VPC, then follow by a list of suggested subnets from the main CIDR in the VPC, user will only choose from the suggested list. Once the subnet is chosen, the script proceed to create the request to create subnet for the specific VPC.

Demonstration

Screenshot 2019-05-06 at 12.25.10 PM

Afterword
The script is still in progress, I intend to test a few functions before doing the main code. The main code right now is to test various functions for bugs. The ipaddress module is really cool module for ip address manipulation.

Posted in Python | Tagged , | Leave a comment

[python]Check if subnet is within the range of a supernet

See this reference.
I have found something really cool about the ipaddress module. I have a use case to help requester to decide which subnets to use. So here’s the thing…

Screenshot 2019-05-05 at 10.40.58 PM

Posted in Python, Scripting | Tagged | Leave a comment

[python]Update a list to a dictionary

the code below enumerate the list of vpc-id and make them into dictionary, the purpose is to make use of this dictionary to give a menu option for user.

import boto3


vpc_id_collector = []
menu_dict = {}
i=0
filter = [{'Name':'tag:Name', 'Values':['*']}]
ec2_client = boto3.client('ec2')
""":type : pyboto3.ec2"""
vpcs = ec2_client.describe_vpcs(Filters=filter)['Vpcs']
for vpc in vpcs:
    vpc_id_collector.append(vpc['VpcId'])
for i in range(len(vpc_id_collector)):
    j = i + 1
    menu_dict[j] = vpc_id_collector[i]
print(menu_dict)
Posted in Python | Tagged , | Leave a comment

[python]Get VPC ID

I am looking for a solution to get the vpcid so that i can start another script to add routes and subnets for the chosen vpc.
See this post, which makes the below code possible.

import boto3
from pprint import pprint


filter = [{'Name':'tag:Name', 'Values':['*']}]
ec2_client = boto3.client('ec2')
""":type : pyboto3.ec2"""
vpcs = ec2_client.describe_vpcs(Filters=filter)['Vpcs']
for vpc in vpcs:
    pprint(vpc['VpcId'])

Results

The json structure after the filter was applied

[{'CidrBlock': '192.168.1.0/24',
  'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-0e252bc4c542b4f28',
                               'CidrBlock': '192.168.1.0/24',
                               'CidrBlockState': {'State': 'associated'}}],
  'DhcpOptionsId': 'dopt-9f1249f8',
  'InstanceTenancy': 'default',
  'IsDefault': False,
  'OwnerId': '203367053771',
  'State': 'available',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_1'}],
  'VpcId': 'vpc-072f78e3f64873a46'},
 {'CidrBlock': '192.168.2.0/24',
  'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-01861ed0994c5b920',
                               'CidrBlock': '192.168.2.0/24',
                               'CidrBlockState': {'State': 'associated'}}],
  'DhcpOptionsId': 'dopt-9f1249f8',
  'InstanceTenancy': 'default',
  'IsDefault': False,
  'OwnerId': '203367053771',
  'State': 'available',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_2'}],
  'VpcId': 'vpc-0d5a397499f3bc8b8'}]

The above json output is a list of two dictionaries.
First is the below, take the above json output is stored in a variable named “response”, then the below will be response[0]:

{'CidrBlock': '192.168.1.0/24',
  'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-0e252bc4c542b4f28',
                               'CidrBlock': '192.168.1.0/24',
                               'CidrBlockState': {'State': 'associated'}}],
  'DhcpOptionsId': 'dopt-9f1249f8',
  'InstanceTenancy': 'default',
  'IsDefault': False,
  'OwnerId': '203367053771',
  'State': 'available',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_1'}],
  'VpcId': 'vpc-072f78e3f64873a46'}

See response[1]:

{'CidrBlock': '192.168.2.0/24',
  'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-01861ed0994c5b920',
                               'CidrBlock': '192.168.2.0/24',
                               'CidrBlockState': {'State': 'associated'}}],
  'DhcpOptionsId': 'dopt-9f1249f8',
  'InstanceTenancy': 'default',
  'IsDefault': False,
  'OwnerId': '203367053771',
  'State': 'available',
  'Tags': [{'Key': 'Name', 'Value': 'cyruslab_2'}],
  'VpcId': 'vpc-0d5a397499f3bc8b8'}

Python uses for in statement to enumerate each item in a list.
So for vpc in vpcs will mean vpc = vpcs[0] then vpc = vpcs[1], until the entire item is enumerated within the for loop. As mentioned the result after filtered, is a list of two dictionaries, in order to get the value, I need to reference its key.
Hence for vpc in vpcs means vpc = vpcs[0], vpc is the first dictionary in the list, I only interested in vpc id, hence vpc['VpcId] and so on.. until the entire list is enumerated.

Posted in Python | Tagged , , | Leave a comment

[python]Create VPC part 2

Background
This post is an improvement to this.
The goal is to:

  • Create VPC.
  • Create internet gateway.
  • Attach internet gateway to vpc.
  • Name the internet gateway and vpc.

About parameters in create_tags
This method accepts

  • Resources of type list or tuple.
  • Tags of type dictionary.

The code works but is ugly and cannot scale, for demonstrating purpose it is ok… But I need to refine the code so that it is modular, such as creating classes and to be used repetitively.

import boto3
import ipaddress


# Initialize values
# Empty the list
vpc_resource_names = []
# Get user's subnet in CIDR format
user_vpc_network = input("Enter the network for VPC:")
# Get user's VPC name.
user_vpc_name = input("Enter VPC name:")
# Get user's internet gateway name.
user_vpc_igw_name = input("Enter internet gateway name:")

# this has to use twice, for creating tag to be used for create_tags method.
def create_tag(resource_name):
    tag = [
        {
            'Key': 'Name',
            'Value': resource_name
        }
    ]
    return tag

try:
    # Specify to change EC2 instance
    ec2_instance = boto3.client('ec2')
    # pyboto3 autocomplete
    """:type : pyboto3.ec2"""
    try:
        # test the validity of subnet, recommended to use regex instead of ipaddress.
        valid_cidr = str(ipaddress.ip_network(user_vpc_network))
        try:
            # Creates VPC with subnet specified.
            vpc_cidr_response = ec2_instance.create_vpc(CidrBlock=valid_cidr)
            # See response in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.create_vpc
            # Case sensitive.
            vpc_id = vpc_cidr_response['Vpc']['VpcId']
            # Include VPC ID to the list, required for create_tags method.
            vpc_resource_names.append(vpc_id)
            # Create name for VPC
            ec2_instance.create_tags(Resources=vpc_resource_names,
                                     Tags=create_tag(user_vpc_name))
            print("{} is created successfully".format(vpc_id))
            try:
                # Create an internet gateway.
                igw_create_response = ec2_instance.create_internet_gateway()
                # See response https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Vpc.attach_internet_gateway
                igw_id = igw_create_response['InternetGateway']['InternetGatewayId']
                # Overwrites the VPC ID value with IGW ID.
                vpc_resource_names[0] = igw_id
                # Name the IGW.
                ec2_instance.create_tags(Resources=vpc_resource_names,
                                         Tags=create_tag(user_vpc_igw_name))
                print("{} is created, attempting to attach to {}".format(igw_id, vpc_id))
                try:
                    ec2_instance.attach_internet_gateway(InternetGatewayId=igw_id, VpcId=vpc_id)
                    print("{} is successfully attached to {}".format(igw_id, vpc_id))
                except BaseException:
                    print("It seems there is a problem attaching internet gateway to VPC.")
            except BaseException:
                print("It seems there is a problem creating internet gateway.")
        except BaseException as e:
            print("It seems there is a problem creating VPC. {}".format(e))
    except ValueError:
        print("Possibly invalid subnet")
except BaseException as e:
    print(e)

Results


Results with all inputs.


Results with subnet and IGW name.


Results with only subnet.

Screenshot 2019-05-05 at 5.11.32 AM.png
Result with no input. Subnet is mandatory to create VPC, in essence a VPC is similar to VRF, has its own routing table. If it is exactly like VRF then each VPC can have the same subnets as well.

Posted in Python, Scripting | Tagged , , , , , | Leave a comment