I cannot find the server with IP 54.240.3.17 in the IP ranges list published by Amazon https://ip-ranges.amazonaws.com/ip-ranges.json
According to ip-location it is in Seattle, but it should be in EU West.I know that the ip-location is normally not that reliable, but does anyone know why it is not in the IP ranges list?
As the list of public AWS IP-ranges is pretty long and matching an IP-address with a network can be error prone I wrote a little script to check if the IP-address is really not included in the list:
#!/usr/bin/env python3
import ipaddress
import json
import urllib.request
IP_ADDRESS_TO_CHECK = ipaddress.IPv4Address('54.240.3.17')
response = urllib.request.urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json')
json_response = json.loads(response.read())
for ip in json_response['prefixes']:
if IP_ADDRESS_TO_CHECK in ipaddress.IPv4Network(ip['ip_prefix']):
print(ip)
Running this does indeed verify that the IP address is currently not included in the list of public IPv4 ranges published by AWS.
I can only guess why that's the case, but it seems likely that AWS just hasn't updated the list yet to include the network the IP-address is part of. Note that the currently served list is already three days old, as you can see from the embedded createDate:
"createDate": "2019-10-18-19-53-09",
Related
I would like to identify the public IP of the terraform execution environment and add it to the "source_range of the GCP firewall delivery. Objective to allow access only through this address
I am currently manually editing the values in the terraform.tfvars file
For example:
public_ip_address {
default = "xx.xx.xx.xx"
}
I would like to automate this process, but I have not found a way that does not use a request for an external system, for example on the platform http://ipv4.icanhazip.com
Is there a way to do these things?
Thank you for reading my question.
As John Hanley mentioned in his comment you will need to make an external request if the machine your Terraform client is running on has a private IP address assigned to its NIC.
However, you can make the external request (to eg. https://ifconfig.me) from your main.tf by using curl (or any other command-line utility in your system) through an external data source:
main.tf:
# Fetch the external IP address via an HTTPS service with curl
data "external" "curlip" {
program = ["sh", "-c", "echo '{ \"extip\": \"'$(curl -s https://ifconfig.me)'\" }'"]
}
# Reference curl's result within your resource
public_ip_address {
default = data.external.curlip.result["extip"]
...
You will then probably need to run terraform init so it installs the required hashicorp/external plugin, and after that it will fetch the external IP address with curl every time you run terraform apply.
I have some Rails 6 applications, deployed at AWS, via Opsworks.
After upgrading to Rails 6 the app blocks the health check of its own instance and it causes the load balancer to take the instance down.
I would like to know how to whitelist all my EC2 instances automatically with dynamic IP addresses? Instead of adding one by one to config/application.rb?
Thanks
Rails.application.configure do
# Whitelist one hostname
config.hosts << "hostname"
# Whitelist a test domain
config.hosts << /application\.local\Z/
# config.hosts.clear
end
The work-around that worked for me was
config.hosts.clear
I posted this question a while back. A safer solution would be reading the IP addresses from environmental variables that can be set from the AWS console.
config.hosts << ENV["INSTANCE_IP"]
config.hosts << ENV["INSTANCE_IP2"]
...
config.hosts << ENV["INSTANCE_IPn"]
At least in this way it does not require a new git commit every time the IP address changes when the instance has a dynamic IP.
Simple solution is to allow the Health Checker user agent, add this to your production.log
config.host_authorization = {
exclude: ->(request) { request.user_agent =~ /ELB-HealthChecker/ }
}
Looks like it has been resolved in the latest versions atleast works on 6.1 and above
https://guides.rubyonrails.org/configuring.html#actiondispatch-hostauthorization
You can exclude certain requests from Host Authorization checks by setting config.host_configuration.exclude:
# Exclude requests for the /healthcheck/ path from host checking
Rails.application.config.host_configuration = {
exclude: ->(request) { request.path =~ /healthcheck/ }
}
I would like to upload files to S3 using boto3.
The code will run on a server without DNS configured and I want that the upload process will be routed through a specific network interface.
Any idea if there's any way to solve these issues?
1) add the end point addresses for s3 to /etc/hosts, see this list http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
2) configure a specific route to the network interface, see this info on superuser
https://superuser.com/questions/181882/force-an-application-to-use-a-specific-network-interface
As for setting a network interface, I did a workaround that allows to set the source ip for each connection made by boto.
Just change awsrequest.py AWSHTTPConnection class with the following:
a) Before init() of AWSHTTPConnection add:
source_address = None
b) Inside the init() add:
if AWSHTTPConnection.source_address is not None:
kwargs["source_address"] = AWSHTTPConnection.source_address
Now, from your code you should do the following before you start using boto:
from botocore.awsrequest import AWSHTTPConnection
AWSHTTPConnection.source_address = (source_ip_str, source_port)
Use source_port = 0 in order to let OS choose random port (you probably want this option, see python socket docs for more details)
I want to accept Bitcoin on my site. I assign each incoming sale a Bitcoin public key/address from a pool of non-used addresses.
I add records to the pool of non-used addresses by generating 1000 receiving addresses on a separate computer using MultiBit and then importing them into the table. I do this as often as I need when I am running out of addresses.
My question is this:
What is the best way or API (and most simple, that does not require bitcoind installation?!) to monitor incoming deposits to a list of addresses to which I don't have public keys for? Basically I would need a cron to check for incoming transactions to these addresses so I can detect payment acceptance.
If you use btc.blocker.io's API you can find the balance of any address use the url like so
http://btc.blockr.io/api/v1/address/info/PublicAdressGoesHere
It will have an outcome something close to this
{"status":"success","data":{"address":"198aMn6ZYAczwrE5NvNTUMyJ5qkfy4g3Hi","is_unknown":false,"balance":8000.00176957,"balance_multisig":0,"totalreceived":8000.00176957,"nb_txs":30,"first_tx":{"time_utc":"2009-02-22T10:50:53Z","tx":"0f0fbcc18fd0d090ad3402574df8404cec1176bc000f9aa0dc19f8d832ff94db","block_nb":"5219","value":400,"confirmations":385428},"last_tx":{"time_utc":"2015-11-25T00:47:46Z","tx":"77bfb2a8098508646980195c7885baf710c1b30b83cfb7432c6de01a1afe1bc7","block_nb":"385201","value":0.000135,"confirmations":5446},"is_valid":true},"code":200,"message":""}
if you want to read the data in python try using this code. It will out put the data into a file called data.txt And it takes data in from a file called address.txt. only do one address at a time
import urllib2.urlopen
with open("address.txt","r") as file:
address = str(file.read())
data = urllib2.urlopen("http://btc.blockr.io/api/v1/address/info/" + address)
with open("data.txt", "w") as a:
a.write(str(data.read()))
hope this helps!
I'm trying to put a set of EC2 instances behind a couple of Varnish servers. Our Varnish configuration very seldom changes (once or twice a year) but we are always adding/removing/replacing web backends for all kinds of reasons (updates, problems, load spikes). This creates problems because we always have to update our Varnish configuration, which has led to mistakes and heartbreak.
What I would like to do is manage the set of backend servers simply by adding or removing them from an Elastic Load Balancer. I've tried specifying the ELB endpoint as a backend, but I get this error:
Message from VCC-compiler:
Backend host "XXXXXXXXXXX-123456789.us-east-1.elb.amazonaws.com": resolves to multiple IPv4 addresses.
Only one address is allowed.
Please specify which exact address you want to use, we found these:
123.123.123.1
63.123.23.2
31.13.67.3
('input' Line 2 Pos 17)
.host = "XXXXXXXXXXX-123456789.us-east-1.elb.amazonaws.com";
The only consistent public interface ELB provides is its DNS name. The set of IP addresses this DNS name resolves to changes over time and with load.
In this case I would rather NOT specify one exact address - I would like to round-robin between whatever comes back from the DNS. Is this possible? Or could someone suggest another solution that would accomplish the same thing?
Thanks,
Sam
You could use a NGINX web server to deal with the CNAME resolution problem:
User-> Varnish -> NGNIX -> ELB -> EC2 Instances
(Cache Section) (Application Section)
You have a configuration example in this post: http://blog.domenech.org/2013/09/using-varnish-proxy-cache-with-amazon-web-services-elastic-load-balancer-elb.html
Juan
I wouldn't recommend putting an ELB behind Varnish.
The problem lies on the fact that Varnish is resolving the name
assigned to the ELB, and it’s caching the IP addresses until the VCL
get’s reloaded. Because of the dynamic nature of the ELB, the IPs
linked to the cname can change at any time, resulting in Varnish
routing traffic to an IP which is not linked to the correct ELB
anymore.
This is an interesting article you might like to read.
Yes, you can.
in your default.vcl put:
include "/etc/varnish/backends.vcl";
and set backend to:
set req.backend = default_director;
so, run this script to create backends.vcl:
#!/bin/bash
FILE_CURRENT_IPS='/tmp/elb_current_ips'
FILE_OLD_IPS='/tmp/elb_old_ips'
TMP_BACKEND_CONFIG='/tmp/tmp_backends.vcl'
BACKEND_CONFIG='/etc/varnish/backends.vcl'
ELB='XXXXXXXXXXXXXX.us-east-1.elb.amazonaws.com'
IPS=($(dig +short $ELB | sort))
if [ ! -f $FILE_OLD_IPS ]; then
touch $FILE_OLD_IPS
fi
echo ${IPS[#]} > $FILE_CURRENT_IPS
DIFF=`diff $FILE_CURRENT_IPS $FILE_OLD_IPS | wc -l`
cat /dev/null > $TMP_BACKEND_CONFIG
if [ $DIFF -gt 0 ]; then
COUNT=0
for i in ${IPS[#]}; do
let COUNT++
IP=$i
cat <<EOF >> $TMP_BACKEND_CONFIG
backend app_$COUNT {
.host = "$IP";
.port = "80";
.connect_timeout = 10s;
.first_byte_timeout = 35s;
.between_bytes_timeout = 5s;
}
EOF
done
COUNT=0
echo 'director default_director round-robin {' >> $TMP_BACKEND_CONFIG
for i in ${IPS[#]}; do
let COUNT++
cat <<EOF >> $TMP_BACKEND_CONFIG
{ .backend = app_$COUNT; }
EOF
done
echo '}' >> $TMP_BACKEND_CONFIG
echo 'NEW BACKENDS'
mv -f $TMP_BACKEND_CONFIG $BACKEND_CONFIG
fi
mv $FILE_CURRENT_IPS $FILE_OLD_IPS
I wrote this script to have a way to auto update the vcl once a new
instance comes up or down.
it requires that the .vcl has an include to backend.vcl
This script is just a part of the solution, the tasks should be:
1. get new servername and IP (auto scale) can use AWS API cmds to do that, also via bash
2. update vcl (this script)
3. reload varnish
The script is here
http://felipeferreira.net/?p=1358
Other pepole did it in different ways
http://blog.cloudreach.co.uk/2013/01/varnish-and-autoscaling-love-story.html
You don get to 10K petitions if had to resolve an ip on each one. Varnish resolve ips on start and do not refresh it unless its restarted o reloaded. Indeed varnish refuses to start if found two ip for a dns name in a backend definition, like the ip returned for multi-az ELBs.
So we solved a simmilar issue placing varnish in front of nginx. Nginx can define an ELB as a backend so Varnish backend is a local nginx an nginx backend is the ELB.
But I don't feel comfy with this solution.
You Could make the ELB in your private VPC so that it would have a local ip. This way you don't have to use any DNS kind of Cnames or anything which Varnish does not support as easily.
Using internal ELB does not help the problem, because it usually have 2 Internal IP's!
Backend host "internal-XXX.us-east-1.elb.amazonaws.com": resolves to multiple IPv4 addresses.
Only one address is allowed.
Please specify which exact address you want to use, we found these:
10.30.10.134
10.30.10.46
('input' Line 13 Pos 12)
What I am not sure is if this IPs will remain always the same or they can change? anyone?
I my previous answer (more than three years ago) I hadn't solve this issue, my [nginx - varnish - nxinx ] -> ELB solution worked until ELB changes IPs
But from some time ago we are using the same setup but with nginx compiled with jdomain plugin
So the idea is to place a nginx in the same host that varnish an there configure the upstream like this:
resolver 10.0.0.2; ## IP for the aws resolver on the subnet
upstream backend {
jdomain internal-elb-dns-name port=80;
}
that upstream will automatically reconfigure the upstream ips the IP if the ELB changes its addresses
It might not be a solution using varnish but it works as expected