The problem:
My url xyz.co is getting resolved into an ugly AWS public DNS name such as ec2-11-22-33-44.ap-southeast-2.compute.amazonaws.com. It doesn't stick to xyz.co.
Here's what I did:
I have set up my Route 53 configuration according to http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/MigratingDNS.html, so I created an A record pointing to the IP address and a CNAME alias record to allow for www.xyz.co. The domain is sitting with godaddy and the name servers are configured to the AWS delegation set.
The instance itself sits in the default VPC. I double-checked and DNS resolution and DNS host names are both active.
I'm a bit stuck here with this. Any help would be highly appreciated!
Cheers,
Bruno
What you are seeing isn't actually related to name resolution.
It's impossible for DNS to change what appears in the address bar of the web browser -- DNS and web browsers simply do not interact in a way that makes such behavior possible. Your URL is not "getting resolved to" this new value via anything DNS-related, since DNS, configured correctly or incorrectly, can't impact what shows up there, on its own.
The fact that navigating to the IP address has the same impact backs up this assertion.
What you are seeing is not related in any way to DNS or Route 53 or even EC2 or VPC. Your web server is, for whatever reason, configured to redirect incoming requests with any other hostname... over to the hostname you are subsequently seeing in the address bar (which is the one you don't like).
You should notice this in your web server's log. It will be issuing a 301 or 302 redirect on the initial request.
You should also be able to verify this yourself with the curl command line utility. Here, a server accessed as "www.example.com" is redirecting the browser to use its preferred address, "example.com." (Hostnames and addresses are sanitized, but the output is otherwise unmodified.)
$ curl -v www.example.com
* Rebuilt URL to: www.example.com/
* Hostname was NOT found in DNS cache
* Trying 203.0.113.139...
* Connected to www.example.com (203.0.113.139) port 80 (#0)
The next block of output is the request sent to the web server.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: www.example.com
> Accept: */*
>
The http response from the web server includes a redirect.
< HTTP/1.1 301 Moved Permanently
< Content-length: 0
< Location: http://example.com/
< Connection: close
<
* Closing connection 0
If we were using a browser instead of a command line tool, this would cause the address bar to change to the new value, and establish a new connection to the web server (which might actually be the same one, or a different one... in this case, it's the same).
In spite of the fact that I had typed http://www.example.com into my browser, it would now show only http://example.com/. The same thing would happen if I typed in the IP address if my server was configured to redirect everything to one hostname, as yours appears to be. In my case, it's deliberately configured to do something else.
The above should illustrate that you do not actually have a DNS issue, and explain the mechanism that's causing this to occur (because you may find this to be something useful to do deliberately in the future, as my web servers do -- any www.* request gets stripped and rewritten without the www).
The issue is with your web server, telling the browser to use a different hostname. How to fix that will depend on what web server you are running and why it thinks the redirect is necessary.
Related
I am following the instructions at https://cloud.google.com/cdn/docs/setting-up-cdn-with-ex-backend-internet-neg and https://medium.com/the-innovation/how-to-enable-google-cdn-for-custom-origin-websites-google-cdn-for-external-websites-56e3fe66cca9 to setup Google Cloud CDN for my website www.datanumen.org.
For the "Fully qualified domain name and port" in "New network endpoint", I choose www.datanumen.org.
All others are same as the above two articles, I use HTTP protocol for all the communications. Finally I get a frontend IP address 34.96.69.82. So I try to visit http://34.96.69.82/, but get a default "SORRY" web page instead of the contents from www.datanumen.org. Why?
Also later I plan to update the DNS A Record for www.datanumen.org so that datanumen.org will points to 34.96.69.82 instead of its current IP address. I am just curious that if I do that, then since what I put in "Fully qualified domain name and port" in "New network endpoint" is www.datanumen.org, will it cause the following deadloop:
a user visit www.datanumen.org
Based on DNS A record, he will go to 34.96.69.82(frondend)
The frontend will request data from backend, and the endpoint is www.datanumen.org,
Based on DNS A record, the backend end point will also solved to 34.96.69.82.
Thus will cause a deadloop for ever?
Update:
For the 1st question, I find the solution. My website is hosted on a server with shared IP. In article https://cloud.google.com/cdn/docs/setting-up-cdn-with-ex-backend-internet-neg, it asks me to add "Host" to the request header, which is used to identify the actual site to be accessed when the request reaches the original server. In my previous configuration, I thought this step is useless so I just skip it. After adding the "Host" field, now I can visit my website properly with the IP address given by Google.
You already fixed your fist issue so you're at a point when you can successfully access your site using an IP address.
Right now all you need is a CDN enabled - if it's not you can do it in the "backend" section of the "load balancing" page.
Have a look also at Network Endpoint Groups documentation to see how to create a load balancer utilising them (the only way you can use GCP's CDN for external site).
To answer your second question - there will be no loop - your site works properly.
Since you're not using secure HTPPS (only HTTP) then you don't have to worry about SSL certificates and the only thing that remains for you to do is to direct your domain to your load-balancer's IP and you're done.
If you encounter any issues or just want to check if CDN is working correctly then have a look at CDN troubleshooting page.
Most simple way to verify if it's working is to use curl: curl -s -D - -o /dev/null http://example.com/style.css and see if you have Cache-Control line present in the output:
HTTP/1.1 200 OK
Date: Tue, 16 Feb 2016 12:00:31 GMT
Content-Type: text/css
Content-Length: 1977
Cache-Control: max-age=86400,public
Via: 1.1 google
However I recommend using HTTPS and SSL certificates for the security reasons - it's much harder to spoof the traffic/listen to the between the site and the client. It's not mandatory though.
I have a bit problem with my site.
So setup is ElasticBeanstalk(NGINX) + Cloudflare
But each day around 4AM I have direct IP attack to my server.
Around 300 requests in 1-2 minutes.
Bot try to access some resources like
GET /phpMyadmi/index.php HTTP/1.1
GET /shaAdmin/index.php HTTP/1.1
POST /htfr.php HTTP/1.1
For now all of them going to 80 or 8080 ports.
And successfully handled by Nginx configuration that redirect it to example:443
server {
listen 80 default_server;
listen 8080 default_server;
server_name _;
return 301 https://example.com$request_uri;
}
server {
listen 443 ssl;
server_name example.com;
ssl on;
...
So questions are,
have many site owners/devOps face the same attack. What is your action to prevent such attacks.
For now it is handled very well and did not affect on server work, should I worry about it? Or just filter out logs with /phpmy/ pattern and forgot about it.
Before this attacks I have request with method PROPFIND, should I blocked it for security reasons? It is handled by default server for now.
I know that I can use Cloudflare Argotunel or ELB + WAF. But I am not really want to do it for now.
I have found one solution on stackoverflow. Is whitelist of all cloudflare ips. But i think it is not a good one.
Also another solution that should work I guess it is to check Host header, and compare it with 'example.com'.
To answer your specific questions:
Every public IP receives unwanted traffic like you describe, sadly its pretty normal. This isnt really an attack as such, its just a bot looking for signs of specific weaknesses, or otherwise trying to provoke a response that contains useful data. This data is no doubt later used in actual attacks, but its basically automated recognisance on a potentially massive scale.
This kind of script likely isnt trying to do any damage, so as long your server is well configured & fully patched its not a big concern. However these kinds of scans are first step towards launching an attack - by identifying services & application versions with known vulnerabilities - so its wise to keep your logs for analysis.
You should follow the principle of least privilege. PROPFIND is related to WebDAV - if you dont use it, disable it (or better white list the verbs you do support and ignore the rest).
If your site is already behind CloudFlare then you really should firewall access to your IP so only Cloudflares IPs can talk to your server. Those IPs do change, so I would suggest a script to download the latest from https://www.cloudflare.com/ips-v4 and have it periodically update your firewall. Theres a slightly vuage help article from CloudFlare on the subject here: https://support.cloudflare.com/hc/en-us/articles/200169166-How-do-I-whitelist-Cloudflare-s-IP-addresses-in-iptables-
If for whatever reason you cant firewall the IP, your next best option is something like fail2ban (www.fail2ban.org) - its a log parser that can manipulate the firewall to temporarily or permanently block an IP address based on patterns found in your log files.
A final thought - id advise against redirecting from your IP to your domain name - your telling the bot/hackers your URL - which they can then use to bypass the CDN and attack your server directly. Unless you have some reason to allow HTTP/HTTPS traffic to your IP address, return a 4XX (maybe 444 a " Connection Closed Without Response") instead of redirecting when requests hit your IP. You should then create a separate server block to handle your redirects, but only have it respond to genuine named URLs.
I have used ShimmerCat with sc-tool to connect to my development sites as described here, and everything has worked always like a charm with it, but I also wanted to follow the "old way" configuring my /etc/hosts. In this case I had a small problem, the server ran ok, and I could access to my development site (let's say that I used https://www.example.com:4043/), but I'm also using a reverse proxy as described on this article, and on the config file reference. It redirects to a Django app I'm using. Let's say it is my devlove.yaml config file:
---
shimmercat-devlove:
domains:
www.example.com:
root-dir: site
consultant: 8080
cache-key: xxxxxxx
api.example.com:
port: 8080
The problem is that when I try to access to a URL that requests the API, a 404 response is sent from the API. Let me try to explain it through an example. I try to access to https://www.example.com:4043/country/, and on this page I do a request to the API: /api/<country>/towns/, then the API endpoint is returning a 404 response so it is not finding this URL, which does not happen when using Google Chrome with sc-tool. I had set both domains www.example.com, and api.example.com on my /etc/hosts. I have been trying to solve it, but without any luck, is there something I'm missing? Any help will be welcome. Thanks in advance.
With a bit more of data, we may be able to find the issue. In the meantime, here is a list of troubleshooting tips:
Possible issue: DNS is cached in browser, /etc/hosts is not being used (yet)
This can happen if somehow your browser has not done a DNS lookup since before you changed your /etc/hosts file. Then the connection is going to a domain in the Internet that may not have the API endpoint that you are calling.
Troubleshooting: Check ShimmerCat's log for the requests. If this is the issue, closing and opening the browser may solve the issue.
Possible issue: the host header is incorrect
ShimmerCat uses the Host header in HTTP/1.1 requests and the :authority header in HTTP/2 requests to distinguish the domains. It always discards any port number present in them. If these headers are not set or are set to a domain other than the ones ShimmerCat is configured to listen, the server will consider the situation so despicable that it will just close the connection.
Troubleshooting: This is not a 404 error, but a connection close (if trying to connect un-proxied, directly to the SSL port where ShimmerCat is listening), or a Socks Connection Failed (if trying to connect through ShimmerCat's built-in SOCKS5 proxy). In the former case, the server will print the message "Rejected request to Just https://some-domain-or-ip/some/path" in his log, using the actual value for the domain, or "Rejected request to Nothing", if no header was present. The second case is more complicated, because the SOCKS5 proxy is before the HTTP routing algorithm.
In any case, the browser will put a red line in the network panel of the developer tools. If you are accessing the server using curl, like this:
curl -k -H host:api.incorrect-domain.com https://127.0.0.1:4043/contents/blog/data-density/
or like
curl -k -H host:api.incorrect-domain.com
(notice the --http2 parameter in the second form), you will get a response:
curl: (56) Unexpected EOF
Extra-tip: There is a field for the network address in the browser's developer tools. Check it, it may tell you something!
Possible issue: something gets messed up when passing the request to the api back-end.
API backends are also sensitive to the host header, and to additional things like authentication cookies and request parameters.
Troubleshooting: A way to diagnose things is invoking ShimmerCat using the --show-proxied-headers command-line option. It makes ShimmerCat to report the proxied headers to the log:
Issuing request with headers :authority: api.example.com
:method: GET
:path: /my/api/endpoint/path/
:scheme: https
accept: */*
user-agent: curl/7.47.0
Possible issue: there are two instances or more of ShimmerCat running
...and they are using different configurations. ShimmerCat uses port sharing among several processes to increase availability. A downside of this is that is perfectly possible to mistakenly start ShimmerCat, forget about stopping it, and start it again after changing some configuration bit. The two instances will be running at the same time, and any of them will pick connections made to the listening port.
Troubleshooting: Shutdown all instances of ShimmerCat, then double-check there are none running by using the corresponding form of the ps command, and start the server with the configuration you want.
I have the following problem and I am struggling to find if a solution exists for it, or what the best practice is here.
I have a site example.com, and multiple servers with different IP addresses around the world. I am seeing the following behaviour in my browser (Chrome) - for simplicity lets say I only have 2 IP addresses for now.
I connect to example.com and data is served from IP address A.B.C.D (server 1). After 40 seconds or, any subsequent request (GET/POST) to example.com then resolves to W.X.Y.Z (server 2). My issue is that I have a cookie based web session on server 1, and server 2 knows nothing about that session. There is no kind of back-end replication I can do to sync state between both servers.
Is there any way I can force the browser to only connect to a single server once a server has served the first page? I am using RR DNS with multiple A records at the moment. Would switching to CNAME solve this problem?
One solution I was thinking of was having each server reply with a configured domain in the http headers (e.g. server1 would reply with X-HEADER: server1.example.com, server2 would reply with X-HEADER: server2.example.com) and then force the browser to make requests to these. I would then have a single IP address for server1.example.com, and another for server2.example.com. Does this break same-origin policy though? If I am on example.com can I send GET/POST/PUT etc. to server1.example.com?
I'd really appreciate any advice on this - I'm so confused!
I want to recover from IP address change. My website used to be running fine, but after I allocated elastic IP address, it stopped working I want the website to be running again with new IP address.
So my question is: What steps must I consider and where should I look at first to recover from IP address change?
I can't access that website with old DNS address or new DNS address. Is this DNS issue?
Notes that might be helpful:
My web server seems to be using nginx, and I tried changing the configuration file with new dns, but that doesn't seem to do it.
When I access new domain name, it redirects to old domain name.
when I check with curl using curl -I http://localhost:80, it says HTTP/1.1 301 Moved Permanently
Just as an add on. The problem is this configuration change in your nginx file:
server {
listen 80;
server_name my.old.domain.com;
...
}
Has to be changed to:
server {
listen 80;
server_name my.new.domain.com;
...
}
So the DNS change can take up to 48 hours to propagate (it usually takes 10-30 minutes from experience, depending on your DNS provider)
When the DNS server wasn't propagated nginx would still redirect you to my.old.domain.com
Hope this helps.
Okay, I think it was just time issue.
After changing web server configuration (nginx), I waited around 28 hours and it suddenly started working again.
Source:
Subdomain IP address changed but client still getting directed to old site