I am trying to make a web scraper but my ubuntu instance is getting banned. I think I am scraping the website a little too fast. I've slowed down the requests but I am still banned. To fix this I assigned the instance with a new IP by releasing my IP and reassigning it one with Elastic IP but it is still banned. How can I assign a new IP for my ubuntu instance to stop it from being banned? It does not seem like reassigning IPs with ElasticIP is the solution.
I dont want to terminate my instance as I would have to setup the new instance again. This instance is under a VPC as well
Unless you know for sure that the website owner is banning just specific IPs, then getting a new IP probably won't help.
There are plenty of websites that block the entire AWS IP range.
When you stop an instance and start it again, you get a new ip address and the entire setup is intact. You wont have to set it up again
Related
I am trying to upgrade my hosting from the 1GB RAM to the 2GB RAM instance from an existing Plesk Lightsail snapshot to another Plesk instance. I was able to create a larger instance and load the snapshot successfully. I was able to access the files via FTP and the databases were all present as expected. However, when I logged into Plesk via {NewStaticIP}:8443/ (using the same credentials as I did with my old instance), I was able to login but the IP address shown was still the old one.
The mistake I made was that for my old instance, I had not assigned a static IP and was using the default public IP provided by Lightsail. If I had defined a static IP to the old one, it would have been easy. I would detach the static IP from the old and attach it to the new and there would be no need to make any change in the DNS records even.
However now, even if I change the A record to point to the new Static IP I've assigned to the new instance (I use Cloudflare), it propagates instantly but shows me a "Web server is down" (Cloudflare error 521). This is confusing me because the server has been set up and I can access it via FTP. Ports 80, 443, 8443, 8447, 21, 22 are all open as well.
When I try to access the website using the static IP, it shows an error that the server refused to connect.
In addition, via Plesk Tools, I was able to change the IP address shown for that server but I also read that it's not the best way to change the address of the server so not sure if this helps or not.
Any help regarding this would be much appreciated. Thank you...
I stopped by EC2 instance to do some maintenance and after I started I got a new IP. bummer. I made an elastic IP to make sure it does not change again, but i still have people calling the old IP, is there any way to claim that recently lost IP associated to my Ec2.
No, you can't get that old public IP back. Try to always use DNS names to avoid this kind of issues in the future.
Look at this official answer to a similar question.
I had a compute engine instance, I shut down that instance from command line. Now the thing is when I started it back the IP got changed, which I guess it was because it was ephemeral.
Now I wanted to ask as I am having now new IP. Does my system got changed or just my IP, because if my previous system is changed then I think that all of my data which I had on that compute engine will be still there and if my previous system got assigned to a new person then maybe he can see my data which will be a security breach.
The second question is, is there any way I can get my previous IP assigned to my new instance which I had before.
You're conflating a couple of things here. Networking in GCP is a purely software-defined configuration. As such, the IP of your machine is defined by software and is in no way dependent on things like your instance itself or the disk attached to the instance (where your data actually lives).
To answer your actual questions:
When the machine is restarted, a new ephemeral IP is assigned. Your data is on the disk attached to the machine, and nobody else has access to it.
You have no control over ephemeral IP addresses. If you want to maintain an IP address between restarts, you need to reserve a static IP and assign it to your instance.
Here's what's bothering me. Is there a better way than sending emails to devs that the ip address for their dev server has changed after the instance is stopped and started?
I was thinking of a single small instance that has an elastic ip which the devs can log in using terminal, and ssh again to the internal ip address of the dev server. Is that effective?
Does it mean that the devs need to be informed of the change every time?
It's unclear exactly what you are saying "there's a new public dns for the server"? -thanks for the comment, that's clearer what you mean! It's the aws domain name in the format "ec2-54-222-213-143.eu-west-1.compute.amazonaws.com" you are referring to
You are asking how can these name/address changes be managed?
Generally speaking for fixing these kinds of problems there are a couple of things to be aware of
Firstly, if it is the public ip address that is changing instead of an ephemeral public ip address use an elastic ip. This will stay the same and can be transferred from an old instance to a new instance. Please read http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html about the differences between "Elastic IP" and normal public IP addresses on AWS
Secondly, if you are concerned about maintenance of the dns records that map the ip addresses to the domain names then it is possible to automate the updates to aws route53. I have used the aws cli command "route53 change-resource-record-sets" for this and also CloudFormation
Automating events to occur on instance start up does take a little research of the available APIs and hooks for example see this answer with a simple use of cloud-init Using cloud-init user data
i've registered a .com domain name. At the Amazon Web Services account i own, I have already set up the DNS zone,i've changed the nameservers at my registrar's panel and i've created an A-record in my AWS DNS zone,too. I think i've done all the preparation needed. But my website is not opening!
This is not a DNS propagation time-requiring issue,by the time i did all the above stuff about 5 days ago (DNS had enough time to be refreshed globally in any ISP). Also via ipduh.com i can see that all the nameservers are correctly configured and recognised, as well as the *.mydomain.com A record which points at my AWS instance's IP.
What possibly would be wrong guys? :/ i've done anything i know and i've followed also the directions i've found in SO and i had no luck till now :/
Any suggestion and help would be highly appreciated :D
Thank you in advance guys!
I'm going to assume that the DNS is set up properly, and that the A record is pointing at the IP address assigned to your instance.
If this is a new AWS account, you're probably running in a VPC. Did you make sure that you allocated a public IP address to the instance? If your IP is 10.something, that's the internal, private IP address and you won't be able to use that. You'll need to allocate an Elastic IP and associate it with your instance, then update your DNS settings.
Next, make sure that the web server is up & running? If you log into the instance, what happens if you wget localhost? You might not get the page you're expecting if you're running multiple name-based virtualhosts, but you should get the index page for the default web site.
OK, so how you're sure the web server is running. Next thing to do is check the security rules. When you created your instance, you had give it the name of a security group. The default is, strangely enough, called "defaut". Take a look and see if port 80 is open. If not, open it up to the world (0.0.0.0) and see if you can access the web site now.
None of this helps? Reboot your instance and see if it starts working when it comes back up - it's possible that you're on a bad host, and rebooting will bring it up on different hardware.