Good Morning i have questions surrounding AWS
Question One
Can i create a single instance that is used specifically for DEV that can be used by multiple developers EG one instance but can cater for multiple projects baseline for ENV should be PHP for example
how do i then specify an instance for a client to be able to test based on question one
how do i then provision a instance for a clients live environment
I want to create template that i can use for this or do i need to create a instance every time to complete the above tasks
Question Two
Can i achieve the above using Elastic beanstalk is it cheaper and more practical
Thank you
I would think you just simply launch one instance for each of your use cases, set the necessary security group and public ips(elastic IP) to allow for those users. And then production environment in its on VPC or at least subnet and with a back up in a second availability zone?
Hard to give any specific advise without more details on the respective requirements.
I would like to setup a Ray cluster to use Rtune over 4 gpus on AWS. But each gpu belongs to a different member of our team. I have scoured available resources for an answer and found nothing. Help ?
In order to start a Ray cluster using instances that span multiple AWS accounts, you'll need to make sure that the AWS instances can communicate with each other over the relevant ports. To enable that, you will need to modify the AWS security groups for the instances (though be sure not to open up the ports to the whole world).
You can choose which ports are needed via the arguments --redis-port, --redis-shard-ports, --object-manager-port, and --node-manager-port to ray start on the head node and just --object-manager-port, and --node-manager-port on the non-head nodes. See the relevant documentation.
However, what you're trying to do sounds somewhat complex. It'd be much easier to use a single account if possible, in which case you could use the Ray autoscaler.
We have a VPN setup with two static routes
10.254.18.0/24
10.254.19.0/24
We have a problem that we can only ever communicate from AWS - to one of the above blocks at a time. At some times it is .18 and at other times it is .19 - I cannot figure out what is the trigger.
I never have any problem communicating from either of my local subnets out to aws at the same time.
Kinda stuck here. Any suggestions?
What have we tried? Well the 'firewall' guys said they dont see anything being blocked. But I read another post here that stated the same thing and the problem still ended up being the firewall.
Throughout the course of playing with this the "good" subnet has flipped 3 times. Meaning
Right now I can talk to .19 but not .18
10 min ago I could talk to .18 but not .19
It just keeps flipping.
We've been able to get this resolved. We changed the static routes configured in AWS from:
10.254.18.0/24
10.254.19.0/24
To use instead:
10.254.18.0/23
This will encompass all the addresses we need and has resolved the issue. Here was Amazon's response:
Hello,
Thank you for contacting AWS support. I can understand you have issues
with reaching your two subnets: 10.254.18.0/24 and 10.254.19.0/24 at
the same time from AWS.
I am pretty sure I know why this is happening. On AWS, we can accept
only one SA (security association) pair. On your firewall, the
"firewall" guys must have configured policy based VPN. In policy/ACL
based VPN, if you create following policys for eg: 1) source
10.254.18.0/24 and destination "VPC CIDR" 2) source 10.254.19.0/24 and destination "VPC CIDR"
OR 1) source "10.254.18.0/24, 10.254.19.0/24" and destination "VPC CIDR"
In both the cases, you will form 2 SA pairs as we have two different
source mentioned in the policy/ACL. You just have to use source as
"ANY" or "10.254.0.0/16" or "10.254.0.0/25", etc. We would prefer if
you can use source as "ANY" then micro-manage the traffic using
VPN-filters if you are using Cisco ASA device. How to use VPN-filters
is given in the configuration file for CISCO ASA. If you are using
some other device then you will have to find a solution accordingly.
If your device supports route based VPN then I would advice you to
configure route based VPN. Route based VPNs always create only one SA
pair.
Once you find a solution to create only one ACL/Policy on your
firewall, you will be able to reach both the networks at the same
time. I can see multiple SA formation on your VPN. This is the reason
why you cannot reach both the subnets at the same time.
If you have any additional questions feel free to update the case and
we will respond to them.
I'm looking for a way to pick the best AWS region to host a Proof of Concept installation for a potential customer in India.
For this, I'd like to try to ping the customer's web site (I verified that it's hosted in India, I assume by the customer itself since that's part of their business) from multiple AWS regions and see which one gives best results.
I found multiple tools which would allow me to run ping from my own browser to multiple AWS locations (e.g. https://cloudharmony.com/speedtest, http://www.cloudping.info/) but none which will allow me to ping between all AWS regions and a specific third party.
Does such a tool exist, or is my only option to run up an EC2 instance in each region and try to ping from it?
You might want to check the answers to this very similar question.
Keep in mind that not all regions have all AWS services available at this time, so make sure the region you pick has all the services that you plan to use. Also, Amazon has said that an India region is in the works.
When writing a web app with Django or such, what's the best way to connect to dynamic EC2 instances, such as a cluster of Redis or memcache instances? IP addresses change between reboots, etc. Elastic IPs are limited to 5 by default - what are some other options for auto-discovering/auto-updating which machines are available?
Late answer, but use Boto: http://boto.cloudhackers.com/en/latest/index.html
You can use security groups, tags, and other means to hit the EC2 API and pick the instances/IPs for each thing (DB Server, caching server, etc.) at load-time. We do this with great success in deployment, and are moving that way with our Django settings.py, as well.
One method that I heard mentioned recently in an AWS webinar was to store this sort of information in SimpleDB. Essentially, you would use SimpleDB as the central configuration location, and each instance that you launch would register its IP etc. with this configuration, so you would always have a complete description of all of your instances in one place. I haven't seen this in practice so I don't know what the best practices would be exactly, but the idea sounds reasonable. I suppose you could use SNS or something to signal all the other instances whenever the configuration changes, so everyone could refresh their in-memory cache of the configuration.
I don't know the AWS administrative APIs yet really, but there's probably an API call to list your EC2 instances, at which point you could use some sort of custom protocol to ping each of them and ask it what it is -- part of the memcache cluster, Redis, etc.
I'm having a similar problem and didn't found a solution yet because we also need to map Load Balancers addresses.
For your problem, there are two good alternatives:
If you are not using EC2 micro instances or load balancers, you should definitely use Amazon Virtual Private Cloud, because it lets you control instances IPs and routing tables (check all limitations before using this service).
If you are only using EC2 instances, you could write a script that uses the EC2 API tools to run the command ec2-describe-instances to find all instances and their public/private IPs. Then, the script could parameterize instances names to hosts and update /etc/hosts. Finally, you should put the script in the crontab of every computer/instance that need to access the EC2 instances (see ec2-describe-instances).
If you want to stay with EC2 instances (I'm in the same boat, I've read that you can do such things with their VPC or use an S3 bucket or something like that.) but with EC2, I'm in the middle of writing stuff like this...it's all really simple up till the part where you need to contact the server with a server from your data center or something. The way I'm doing it currently is using the API to create the instance and start it...then once its ready, I contact the server to execute a powershell script that I have on the server....the powershell renames the computer and reboots it...that takes care of needing the hostname and MAC for our data center firewalls. I haven't found a way yet to remotely rename a computer.
As far as knowing the IP, the elastic IPs are the way to go. They say you're only allowed 5 and gotta apply for more but we've been regularly requesting more and they give em to us..we're up to like 15 now and they haven't complained yet.
Another option if you dont' want to do all the computer renaming and such...you could use DHCP and set your computer up so when it boots it gets the computer name and everything from DHCP....I'm not sure how to do this exactly, I've come across very smart people telling me that's the way to do it during my research for Amazon.
I would definitely recommend that you get into the Amazon API...I've been working with it for less than a month and I can do all kinds of crazy things. My code can detect areas of our system that are getting stressed, spin up 10 amazon servers all configured to act as whatever needs stress relief, and be ready to send jobs to all in less than 7 minutes. Brings a tear to my eye.
The documentation is very complete...the API itself is a work of art and a joy to program against...I've very much enjoyed working with it. (and no, i dont' work for them lol)
Do it the traditional way: with DNS. This is what it was built for, so use it! When a machine boots, have it ask for the domain name(s) related to its function, and use that for your configuration. If it stops responding, re-resolve the DNS (or just do that periodically anyway).
I think route53 and the elastic load balancing stuff can be used to do this, if you want to stick to Amazon solutions.