Q: Converting hostname to IP from a dictionary for netmiko connection? - python-2.7

In the past, I was able to do something simple like this to read a file
of IP addresses that I would iterate through to SSH to each one
# Grab the list of devices from a text file
devices = open('./devices.txt','r').read().split('\n')
# Connect to each router and do a show run command
for device in devices:
net_connect = ConnectHandler(device_type="cisco_ios", ip=device, username="myusername",
password=password)
This time, however, I need to do something a bit more complex.
I pull some data from a JSON source where I can derive a "hostname."
But I cannot connect to the "hostname." If the hostnames were in DNS
that would be easier but sadly they are not.
So.. I have a list of hosts to IPs that I figured I could use and pull
into a dictionary.
But now I somehow need to match up the hostname that gets derived from
the JSON data to match against a switchname in the hosts.csv so that I can
then basically convert hostname to IP so that I can then iterate through each
device in devices to SSH into each one.
This is all I have so far. I'm stuck at this point. Not sure how to match
things up to get the IP to use in my net_connect statement.
# Creat dict mapping hostnames to IP address for devices
with open(r'hosts.csv', 'r') as f:
for line in f:
switch = {}
switch_line = line.split(',')
switch = {
'ip': switch_line[1],
'switchname': str(switch_line[0]).strip('\n')
# Define list of devices to connect to and the config changes to be made
pa = []
# This data is coming from a JSON source
for device in devices:
if device['switchParent']:
hostname = device['switchParent']
else:
hostname = device['destDevice']
# hostname technically is the device I need to connect
# however it needs to resolve to an IP from the switch dict earlier
net_connect = ConnectHandler(device_type="cisco_ios", ip=hostname,
username="myusername", password=password)

Assuming that switchname has a unique ip address then you should parse the hosts.csv file this way:
# Create dict mapping hostnames to IP address for devices
switch = {}
with open(r'hosts.csv', 'r') as f:
for line in f:
switch_line = line.split(',')
switch[str(switch_line[0].strip())] = switch_line[1]
Then you go through the JSON data source:
# This data is coming from a JSON source
for device in devices:
if device['switchParent']:
hostname = device['switchParent']
else:
hostname = device['destDevice']
# hostname technically is the device I need to connect
# however it needs to resolve to an IP from the switch dict earlier
net_connect = ConnectHandler(device_type="cisco_ios", ip=switch[hostname],
username="myusername", password=password)
That should work IF hostname is expected to be one of the keys from the switch dict.

Related

Capturing Data of a User Visiting my website

How can I capture data of someone visiting my website via a shortened link? I want to be able to capture the OS, IP, divide ID etc. Would capturing the user agent string be the best option? What are some other ways? In my case I'm building a website with DJAGNO.
To get the IP, you can use the HTTP_X_FORWARDED_FOR or REMOTE_ADDR headers from request.META
For instance:
FORWARDED_HEADER = "HTTP_X_FORWARDED_FOR"
REMOTE_HEADER = "REMOTE_ADDR"
headers = request.META
ip_header = FORWARDED_HEADER if FORWARDED_HEADER in headers else REMOTE_HEADER
ip_address = headers.get(ip_header)
For the user agent you can use the HTTP_USER_AGENT header:
USER_AGENT_HEADER = "HTTP_USER_AGENT"
user_agent = request.META.get(USER_AGENT_HEADER)

When deploying Django into AWS Fargate how do you add the local ip into ALLOWED_HOSTS

I am testing out deploying my Django application into AWS's Fargate Service.
Everything seems to run, but I am getting Health Check errors as the Application Load Balancer is sending requests to my Django application using the Local Ip of the host. This give me an Allowed Host error in the logs.
Invalid HTTP_HOST header: '172.31.86.159:8000'. You may need to add '172.31.86.159' to ALLOWED_HOSTS
I have tried getting the Local ip at task start up time and appending it to my ALLOWED_HOSTS, but this fails under Fargate:
import requests
EC2_PRIVATE_IP = None
try:
EC2_PRIVATE_IP = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4', timeout = 0.01).text
except requests.exceptions.RequestException:
pass
if EC2_PRIVATE_IP:
ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
Is there a way to get the ENI IP Address so I can append it to ALLOWED_HOSTS?
Now this works, and it lines up with the documentation, but I don't know if it's the BEST way or if there is a BETTER WAY.
My containers are running under the awsvpc network mode.
https://aws.amazon.com/blogs/compute/under-the-hood-task-networking-for-amazon-ecs/
...the ECS agent creates an additional "pause" container for each task before starting the containers in the task definition. It then sets up the network namespace of the pause container by executing the previously mentioned CNI plugins. It also starts the rest of the containers in the task so that they share their network stack of the pause container. (emphasis mine)
I assume the
so that they share their network stack of the pause container
Means we really just need the IPv4 Address of the pause container. In my non-exhaustive testing it appears this is always Container[0] in the ECS meta: http://169.254.170.2/v2/metadata
With those assumption in play this does work, though I don't know how wise it is to do:
import requests
EC2_PRIVATE_IP = None
METADATA_URI = os.environ.get('ECS_CONTAINER_METADATA_URI', 'http://169.254.170.2/v2/metadata')
try:
resp = requests.get(METADATA_URI)
data = resp.json()
# print(data)
container_meta = data['Containers'][0]
EC2_PRIVATE_IP = container_meta['Networks'][0]['IPv4Addresses'][0]
except:
# silently fail as we may not be in an ECS environment
pass
if EC2_PRIVATE_IP:
# Be sure your ALLOWED_HOSTS is a list NOT a tuple
# or .append() will fail
ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
Of course, if we pass in the container name that we must set in the ECS task definition, we could do this too:
import os
import requests
EC2_PRIVATE_IP = None
METADATA_URI = os.environ.get('ECS_CONTAINER_METADATA_URI', 'http://169.254.170.2/v2/metadata')
try:
resp = requests.get(METADATA_URI)
data = resp.json()
# print(data)
container_name = os.environ.get('DOCKER_CONTAINER_NAME', None)
search_results = [x for x in data['Containers'] if x['Name'] == container_name]
if len(search_results) > 0:
container_meta = search_results[0]
else:
# Fall back to the pause container
container_meta = data['Containers'][0]
EC2_PRIVATE_IP = container_meta['Networks'][0]['IPv4Addresses'][0]
except:
# silently fail as we may not be in an ECS environment
pass
if EC2_PRIVATE_IP:
# Be sure your ALLOWED_HOSTS is a list NOT a tuple
# or .append() will fail
ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
Either of these snippets of code would then in in the production settings for Django.
Is there a better way to do this that I am missing? Again, this is to allow the Application Load Balancer health checks. When using ECS (Fargate) the ALB sends the host header as the Local IP of the container.
In fargate, there is an environment variable injected by the AWS container agent:${ECS_CONTAINER_METADATA_URI}
This contains the URL to the metadata endpoint, so now you can do
curl ${ECS_CONTAINER_METADATA_URI}
The output looks something like
{
"DockerId":"redact",
"Name":"redact",
"DockerName":"ecs-redact",
"Image":"redact",
"ImageID":"redact",
"Labels":{ },
"DesiredStatus":"RUNNING",
"KnownStatus":"RUNNING",
"Limits":{ },
"CreatedAt":"2019-04-16T22:39:57.040286277Z",
"StartedAt":"2019-04-16T22:39:57.29386087Z",
"Type":"NORMAL",
"Networks":[
{
"NetworkMode":"awsvpc",
"IPv4Addresses":[
"172.30.1.115"
]
}
]
}
Under the key Networks you'll find IPv4Address
Putting this into python, you get
METADATA_URI = os.environ['ECS_CONTAINER_METADATA_URI']
container_metadata = requests.get(METADATA_URI).json()
ALLOWED_HOSTS.append(container_metadata['Networks'][0]['IPv4Addresses'][0])
An alternative solution to this is to create a middleware that bypasses the ALLOWED_HOSTS check just for your healthcheck endpoint, eg
from django.http import HttpResponse
from django.utils.deprecation import MiddlewareMixin
class HealthEndpointMiddleware(MiddlewareMixin):
def process_request(self, request):
if request.META["PATH_INFO"] == "/health/":
return HttpResponse("OK")
I solved this issue by doing this:
First i've installed this middleware that can handle CIDR masks on top of ALLOWED_HOSTS: https://github.com/mozmeao/django-allow-cidr
With this middleware i can use a env var like this:
ALLOWED_CIDR_NETS = ['192.168.1.0/24']
So you need to find out the subnets you had configured on your ECS Service Definition, for me it was: 10.3.112.0/24 and 10.3.111.0/24.
You add that to your ALLOWED_CIDR_NETS and you're good to go.

get_host() vs META['REMOTE_ADDR'] for security reasons

I am working on a Django project where users will have custom info given to them depending on their location. In order to do this, I use their IP address to identify their country. In order to keep data in the database consistent, I need to make sure I have an accurate IP.
I understand that using META usually uses headers sent by the client's browser, but I don't know if that applies to the REMOTE_ADDR attribute.
TLDR: what is the difference between HttpRequest.get_host() and HttpRequest.META['REMOTE_ADDR']?
The difference between HttpRequest.get_host() and HttpRequest.META['REMOTE_ADDR'] is that the first one checks IP in the following headers in order of decreasing preference:
HTTP_X_FORWARDED_HOST
HTTP_HOST
SERVER_NAME combined with SERVER_PORT
whereas the second one check the IP in the header REMOTE_ADDR.
There is a huge difference in the type of information returned: get_host() will give you the name of the server hosting your application, not the IP of the client.
More in detail, here is the implementation of get_host():
def get_host(self):
"""Returns the HTTP host using the environment or request headers."""
# We try three options, in order of decreasing preference.
if settings.USE_X_FORWARDED_HOST and (
'HTTP_X_FORWARDED_HOST' in self.META):
host = self.META['HTTP_X_FORWARDED_HOST']
elif 'HTTP_HOST' in self.META:
host = self.META['HTTP_HOST']
else:
# Reconstruct the host using the algorithm from PEP 333.
host = self.META['SERVER_NAME']
server_port = str(self.META['SERVER_PORT'])
if server_port != ('443' if self.is_secure() else '80'):
host = '%s:%s' % (host, server_port)
allowed_hosts = ['*'] if settings.DEBUG else settings.ALLOWED_HOSTS
domain, port = split_domain_port(host)
if domain and validate_host(domain, allowed_hosts):
return host
else:
msg = "Invalid HTTP_HOST header: %r." % host
if domain:
msg += "You may need to add %r to ALLOWED_HOSTS." % domain
raise DisallowedHost(msg)
If you want to check for client IP address, here are some headers that could be worth checking (see Getting the client IP address: REMOTE_ADDR, HTTP_X_FORWARDED_FOR, what else could be useful?):
REMOTE_ADDR
HTTP_X_FORWARDED_FOR
HTTP_CLIENT_IP
HTTP_X_FORWARDED_FOR can be comma delimited list of IPs
HTTP_X_FORWARDED
HTTP_X_CLUSTER_CLIENT_IP
HTTP_FORWARDED_FOR
HTTP_FORWARDED
If you don't know which one to pick (if not all), you could log those headers and pragmatically add new checkings over time.

Loop in Django Model

I have a IP Block calculator web application that will print a range of IP addresses based on slashes. However it only saves the last record in the range into the text field. I want to be able to save it all to the text field.
I am using python-ipy with my code. Look at the last for loop "rangeip",
Here is my code:
#ip block and range save function
def save(obj, *args, **kwargs):
subnet = unicode(obj.subnet)
first = IP(obj.ip_start + subnet).net()
broadcast = IP(obj.ip_start + subnet).broadcast()
print first
print broadcast
obj.broadcast_ip = broadcast
ip_block = IP(obj.ip_start + subnet)
ip_block.WantPrefixLen = 3
ip = IP(obj.ip_start + subnet)
for gateway in ip[1]:
obj.gateway_ip = gateway
print gateway
#rangeip for loop
for rangeip in ip:
obj.ip_range = rangeip
print rangeip
super(IP_block, obj).save(*args, **kwargs)
This is what I would like, to be able to save the list into a text field, it only saves the last ip: 192.168.1.31
Example I would like:
192.168.1.1
192.168.1.0
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
192.168.1.7
192.168.1.8
192.168.1.9
192.168.1.10
192.168.1.11
192.168.1.12
192.168.1.13
192.168.1.14
192.168.1.15
192.168.1.16
192.168.1.17
192.168.1.18
192.168.1.19
192.168.1.20
192.168.1.21
192.168.1.22
192.168.1.23
192.168.1.24
192.168.1.25
192.168.1.26
192.168.1.27
192.168.1.28
192.168.1.29
192.168.1.30
192.168.1.31
Any help is greatly appreciate it.
obj.ip_range = rangeip is an assignment. You are replacing the content of obj.ip_range with the new rangeip.
What you need to do is:
obj.ip_range += "%s"%rangeip

GeoIP always shows the server's IP address

I am trying to use GeoIP, but I have a problem when I use REMOTE_ADDR. The IP shown is that of my server and not the client.
from django.contrib.gis.geoip import GeoIP
Example context:
g = GeoIP()
ip = self.request.META.get('REMOTE_ADDR')
context['my_ip'] = ip # this display ip client
context['pais_anuncio'] = g.country_code('ip') # this display ip my server.
What am I doing wrong, Thank you.
My guess is, since you are passing the string 'ip', it is defaulting to your server's IP. Try passing in the variable ip, like this:
context['pais_anuncio'] = g.country_code(ip)