Is there a best practice around separating environments in AWS?
I've got a solution that employs the following services:
Lambda
SNS
SQS
DyanmoDB
API Gateway
S3
IAM
We're not live yet, but we're getting close. By the time we go-live, I'd like a proper production, test, and development environment with a "reasonable" amount of isolation between them.
Separate account per environment
Single Account and separate VPC per environment
I read the article AWS NETWORKING, ENVIRONMENTS AND YOU by Charity Majors. I'm down with segmentation via VPC, but I don't know that all the services in my stack are VPC scoped? Here are some of my requirements:
Limit Service Name Collision (for non global services)
Establish a very clear boundary between environments
Eventually, grant permissions at the environment level
I am using an AWS Organization.
P.S. Apologies if this isn't the right forum for the question. If there is something better, just let me know and I'll move it.
I recommend one AWS account per environment. The reasons, in no particular order:
security: managing complex IAM policies to create boundaries within a single account is really hard; conversely, one account per environment forces boundaries by default: you can enable cross account access but you have to be very deliberate
auditing access to your different environments is more difficult when all activity happens in the same account
performance: some services don't have the same performance characteristics when operating in VPC vs non-VPC (ie. Lambda cold starts increased latency when operating in VPC)
naming: instead of using the AWS account id to identify the environment you're operating in, you have to add prefixes or suffixes to all the resources in the account - this is a matter of preference but nonetheless..
compliance: if you ever need to adhere to some compliance standard such as HIPAA which imposes strict restrictions on how long you can hold on to data and who can access data, it becomes really difficult to prove which data is production and which data is test etc. (this goes back to #1 and #2 above)
cost control: in dev, test, staging environments you may want to give people pretty wide permissions to spin up new resources but put low spending caps to prevent accidental usage spikes; conversely in a production account you'll want restricted ability to spin up resources but higher spending caps; easy to enforce via separate account - not so much in the same account
Did I miss anything? Possibly! But these are the reasons why I would use separate accounts.
By the way - I am NOT advocating against using VPCs. They exist for a reason and you should definitely use VPCs for network isolation. What I am trying to argue is that anybody who also uses other services such as DynamoDb, Lambda, SQS, S3 etc - VPCs are not really the way to isolate resources, IMO.
The downsides to one account per stage that I can think of are mostly around continuous deployment if you use tools that are not flexible enough to be able to deploy to different accounts.
Finally, some people like to call on billing as a possible issue but really, wouldn’t you want to know how much money you spend on Production vs Staging vs Development ?!
Avoid separate accounts for each environment to avoid additional complexity and obstacles in accessing shared resources.
Try rather using:
resource groups
tagging
as recommended by AWS:
https://aws.amazon.com/blogs/startups/managing-resources-across-multiple-environments-in-aws/
The account separation is recommended by the AWS Well Architected Framework security pillar.
Related
I am very new with AWS and wanted to clear my concept on AWS services. I have read that that AWS has plenty of services that can also be accessed through API. A service is basically a software program. Then why are services not available in all regions. If my customers are from India, I can buy the EC2 instance from Asia but why should I choose service from USA East. Again, why does AWS provide regions for End Points. They could have installed all the services in all their regions - assuming that they are only software programs and not hardware resources.
Latency is not a big problem for you, I think, you can choose the best price options for your sources. If latency big a problem, you must choose the region/zone near your target market. Better understanding read this doc.
AWS Services operate on multiple levels and are all exposed through APIs.
Some services operate at a global scope (e.g. Identity and Access Management or Route53), most on a regional level (e.g. S3) and others somewhere between the region and availability zone (EC2, RDS, VPC...).
AWS uses the concept of a region for multiple purposes, one of the major drivers being fault isolation. Something breaking in Ireland (eu-west-1) shouldn't stop a service in Frankfurt (eu-central-1) from operating. Latency is another driver here. Since physics is involved, higher distances also increase the latency, which makes things like replication more tricky. Data residency and other compliance aspects are also a good reason to compartmentalize services.
Services being regional results in their endpoints being regional as well.
As to not every service being available in every region: Hardware availability is part of the reason, it doesn't make sense to have the more obscure hardware for niche use cases (think GroundStation, their satellite control service) in all regions. Aside from that, there are most likely some financial aspects involved as well, as global scale and complexity come at a cost and if demand isn't sufficient, it may not make sense to roll out a service everywhere.
My employer is asking me what hours I want to use AWS VMs.
They don't want to grant me full corporate access, because in the past people have shut down mission critical instances by mistake.
I'd like the flexibility to start/stop my own instance and not be reliant on asking someone else to extend the hours on an adhoc basis, as I often work odd hours into the night if I am on a roll with something.
Other than the expense of a 24/7 use case, is there a more cost effective capability that I can point the gatekeeper too, that would allow this sort of flexibility?
At the moment, I'm pretty naive on the AWS front.. I just use the VMs I've been given to use.
BTW: I think there are issues about having them in certain domains - so I can't just have my own individual account.
Thanks in advance for your advice.
I think there are issues about having them in certain domains - so I can't just have my own individual account.
This is what AWS Organizations is for: you have your own account, but it's tied to the corporate account and can be granted access to perform certain functions.
You don't describe what you're creating these instances for, but I'm going to assume that it's development testing. In that case, you would work entirely within your own sandbox, and be unable to affect the mission-critical resources. If there's a need for explicit domain naming, they can delegate authority for a sub-domain, and if necessary use CNAMEs link hosts in that sub-domain to the parent domain.
If you need to do production support work, such as bringing up a new production machine, they can create a role that grants you permission to do so -- probably one that allows you to start machines but not stop them.
At the moment, I'm pretty naive on the AWS front
Unfortunately, it sounds like they are as well. I think the best thing you can do is point them at the Organizations doc.
I would like to know a system by which I can keep track of multiple aws accounts, somewhere around 130+ accounts with each account containing around 200+ servers.
I wanna know methods to keep track of machine failure, service failure etc.
I also wanna know methods by which I can automatically turn up a machine if the underlying hardware failed or the machine terminated while on spot.
I'm open to all solutions including chef/terraform automation, healing scripts etc.
You guys will be saving me a lot of sleepless nights :)
Thanks in advance!!
This is purely my take on implementing your problem statement.
1) Well.. for managing and keeping track of multiple aws accounts you can use AWS Organization. This will help you manage centrally with one root account all the other 130+ accounts. You can enable consolidated billing as well.
2) As far as keeping track of failures... you may need to customize this according to your requirements. For example: You can build a micro service on top of docker containers or ecs whose sole purpose is to keep track of failures, generate a report and push to s3 on a daily basis.You can further create a dashboard using AWS quicksight out of this reports in S3.
There can be another micro service which will rectify the failures. It just depends on how exhaustive and fine grained you want your implementation to be.
3) For spawning instances when spot instances are terminated, it can be achieved through you simple autoscaling configurations. Here are some of the articles you may want to go through which will give you some ideas:
Using Spot Instances with On-Demand instances
Optimizing Spot Fleet+Docker with High Availability
AWS Organisations are useful for management. You can also look at multiple account billing strategy and security strategy. A shared services account with your IAM users will make things easier.
Regarding tracking failures you can set up automatic instance recovery using CloudWatch. CloudWatch can also have alerts defined that will email you when something happens you don't expect, though setting them up individually could be time consuming. At your scale I think you should look into third party tools.
An infrastructure (include both dev and prod environments) for an application has been made on an AWS account that is quite big, includes 15 instances,... Now, we're gonna make a new infrastructure for another application. I would like to know if it's better to create another AWS account for the new project. What would be the advantages?
Although I prefered to have separated account for each environment than projects but as the first project is made on one account compeletly, so I think the only better way is to atleast create another AWS account for the new project.
Plus,in any case, is there any easy way to transfer production env to another account inorder to separate the environments?
Any suggestion would be appreciated.
I'm not sure as to the circumstances in your case but I imagine having a separate account for each environment does give you more control and less room for error.
If you're working alone, try to determine this for yourself whether the effort is worth it. Should you be part of a team or even leading a team, if someone has access to the 'global' aws account with both the development and production instances, errors can easily be made. If you're consuming the AWS API for example and terminate the wrong instance... Food for thought.
Another reason would be that you will need to become very very granular with your IAM roles should you wish to worth with a global account with each environment in it to keep some level of control.
Lastly, cloudwatch will give you nice detailed reports on how your instances are doing and when you have all environments in their respective AWS accounts, it becomes a quick way to see which servers are operating in which fashion.
If all your environments are in the same account, this can become quite confusing as to which instances are production / development.
TLDR, it is good practise to split up the different environments to keep a higher level of control and overview.
Today (I know I'm answering a very old question), AWS makes it easy and very useful to group accounts into Organizations.
For a big setup, this means you can consolidate billing, reservations and other reductions, as well as many security and compliance aspects, while keeping each account operationally separate. While it may be some overhead for a small setup it will be less overhead than trying to keep separate two development teams that are using one account, and extra costs are small to none.
In short, there are a number of very significant advantages and as far as I can see no significant downsides to separating different spheres of responsibility into different accounts.
I would want to have different environments in AWS. At first I thought of differentiating environments by Tags, tags on AWS Resources. But then I cannot restrict users to change Tags of the machine. What that means is, if I allow them ec2:CreateTags, they can not only create tag, but also change tag of any of the resources, since cannot apply a condition on it - say for example if it belongs to a particular VPC or subnet. If I don't allow them the previlege to create tag, then they can launch an instance but their tags are not applied and hence any further operation on the instance is not permitted.
If I want to distinguish between environments by VPC-ID, then for operations such as ec2:StartInstance cannot apply a condition to allow the operation only in a specific VPC-ID, but can conditionally allow based on Resource Tag which for reasons in previous paragraph is not convincing.
On AWS documentation it mentions
One approach to maintaining this separation was discussed in the Working with AWS Access Credentials, that is, to use different accounts for development and production resources.
So it is possible to have one Paying Account for several other accounts which themselves are Paying Accounts? I still don't think multiple accounts for just different environments is a good idea.
How do you generally differentiate among environments for enforcing policies?
Thanks.
Different accounts is the way to go. There are so many places you'll want to create isolation that you'll make yourself crazy trying to do it within one account. Think about it - there's network controls, all the IAM permissions for all the services, access control lists, tags that have the limitations you describe, and on and on. Real isolation comes from putting things in different accounts for now.
The last thing you want is some weakness in your dev environment to pivot into your production environment - end of story. Consider also the productivity benefit of separating prod and dev accounts... you'll never break a prod system from a mistake or experiment in development.
Consolidated billing is the answer to paying for it all. Easy to setup and track. If you need more reporting, look into CloudAbility.
Where this gets really interesting is in the space of multiple production and multiple dev environments. There are a lot of opinions on isolation there. Some people combine all prod and dev into two accounts, and some put every prod and dev into their own. It all depends on your risk profile. Just don't end up like CloudSpaces.
It is possible to do consolidated billing, where one account is billed for its own usage + the AWS usage for any other linked account. However, you can not split that bill (e.g. have the master account only pay for EC2 services on a linked account, while having the linked account pay for it's other usage like S3, etc.).
As for differentiating between environments, I've used different security groups for each one (dev, staging, production) as an alternative to tags, but there are limitations when it comes to enforcing policies. The best option to have full policy control is to use different accounts.
I would suggest go with with one VPC and use Security Groups for isolation. As your AWS infra grows, you will need Directory Services (Name Servers, User Directory, VM Directory, Lookup services etc.). If you have two VPCs, sharing the Directory Services will not be easy. Also if you need Code Repository (e.g. GitHub) or Build tools (e.g. Jenkins) having three separate VPCs for DEV, Staging and Production will make things really complicated.