I would like to know a system by which I can keep track of multiple aws accounts, somewhere around 130+ accounts with each account containing around 200+ servers.
I wanna know methods to keep track of machine failure, service failure etc.
I also wanna know methods by which I can automatically turn up a machine if the underlying hardware failed or the machine terminated while on spot.
I'm open to all solutions including chef/terraform automation, healing scripts etc.
You guys will be saving me a lot of sleepless nights :)
Thanks in advance!!
This is purely my take on implementing your problem statement.
1) Well.. for managing and keeping track of multiple aws accounts you can use AWS Organization. This will help you manage centrally with one root account all the other 130+ accounts. You can enable consolidated billing as well.
2) As far as keeping track of failures... you may need to customize this according to your requirements. For example: You can build a micro service on top of docker containers or ecs whose sole purpose is to keep track of failures, generate a report and push to s3 on a daily basis.You can further create a dashboard using AWS quicksight out of this reports in S3.
There can be another micro service which will rectify the failures. It just depends on how exhaustive and fine grained you want your implementation to be.
3) For spawning instances when spot instances are terminated, it can be achieved through you simple autoscaling configurations. Here are some of the articles you may want to go through which will give you some ideas:
Using Spot Instances with On-Demand instances
Optimizing Spot Fleet+Docker with High Availability
AWS Organisations are useful for management. You can also look at multiple account billing strategy and security strategy. A shared services account with your IAM users will make things easier.
Regarding tracking failures you can set up automatic instance recovery using CloudWatch. CloudWatch can also have alerts defined that will email you when something happens you don't expect, though setting them up individually could be time consuming. At your scale I think you should look into third party tools.
Related
I have two EC2 instances in a single AWS account. Both are running for different application services. Now, I want to make billing separate for that particular account. So, I can get the exact spent and charges for the applications and can manage my account as per that for separate accounting purposes.
Is it possible? If not, then can anyone suggest me a better way to achieve this?
I'm planning to handle the different AWS accounts for both services. But, it will be hard to manage so, I can't prefer that option.
For more, In Google Cloud, they're providing to handle different billing accounts within the same google cloud account. So, I think this concept might be available on AWS also.
Thanks in advance for any little bit of help or suggestion.
You can't get separate bills for different sets of resources within the same AWS account. However, you can filter out the costs for different sets of resources using tags. By using a distinct tag (or set of tags) in the resources you allocate to each application, you can get a breakdown of the cost for each application in billing reports and cost explorer. See the documentation for details and steps on how to set it up - https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html
I believe it's the same with Google Cloud as well - although you can have multiple billing accounts within your google account (the hierarchy in GC is different to AWS, and an AWS account is more similar to a GC project than a GC account), a project within your account must have one billing account and does not support multiples.
By this -
Now, I want to make billing separate for that particular account.
do you mean you already have 2 different AWS accounts? If yes, you can get billing details as per account if they become part of same Organization. Check AWS Organizations docs for more info.
With Orgs, you can view bills of different accounts from one account itself if it is part of org. Also your accounts get discounts for services.
I am completely new to the implementation of cloud solutions. I've just started taking AWS training courses.
But I already have a very fundamental question about the flow of development in cloud projects:
How do you go about developing solutions without incurring costs? I know that there are free tiers, but in practice you need a lot of unfree elements. Especially when working with infrastructure-as-code approaches (e.g. CloudFormation), it can happen that every time you try out the templates, costs can be incurred immediately.
Is there maybe something like a sandbox mode or how else do you go about it in practice?
Outside of the AWS Free Tier you will be billed for creating services.
The best way to keep costs as low as possible is to combing the lowest priced settings (such as instance class) with removing resources you're not using after you're complete. I understand that this will cost, however many resources are now moving to per second billing (where you normally have to pay for at least the first minute) so the cost is kept low.
Additionally when dealing with some services (such as EC2, ECS, Fargate and ECR) you can make use of spot instances to pay sometimes as low as 10% of the original cost which will help to reduce these resources.
To ensure you can recreate resources when you want them use infrastructure as code to reroll out as you need the resources (CloudFormation or Terraform are great offerings for this).
Finally be on the lookout for AWS conferences, they are a great way to pickup AWS credits for attending which will offset your bill against most AWS services.
Is there a best practice around separating environments in AWS?
I've got a solution that employs the following services:
Lambda
SNS
SQS
DyanmoDB
API Gateway
S3
IAM
We're not live yet, but we're getting close. By the time we go-live, I'd like a proper production, test, and development environment with a "reasonable" amount of isolation between them.
Separate account per environment
Single Account and separate VPC per environment
I read the article AWS NETWORKING, ENVIRONMENTS AND YOU by Charity Majors. I'm down with segmentation via VPC, but I don't know that all the services in my stack are VPC scoped? Here are some of my requirements:
Limit Service Name Collision (for non global services)
Establish a very clear boundary between environments
Eventually, grant permissions at the environment level
I am using an AWS Organization.
P.S. Apologies if this isn't the right forum for the question. If there is something better, just let me know and I'll move it.
I recommend one AWS account per environment. The reasons, in no particular order:
security: managing complex IAM policies to create boundaries within a single account is really hard; conversely, one account per environment forces boundaries by default: you can enable cross account access but you have to be very deliberate
auditing access to your different environments is more difficult when all activity happens in the same account
performance: some services don't have the same performance characteristics when operating in VPC vs non-VPC (ie. Lambda cold starts increased latency when operating in VPC)
naming: instead of using the AWS account id to identify the environment you're operating in, you have to add prefixes or suffixes to all the resources in the account - this is a matter of preference but nonetheless..
compliance: if you ever need to adhere to some compliance standard such as HIPAA which imposes strict restrictions on how long you can hold on to data and who can access data, it becomes really difficult to prove which data is production and which data is test etc. (this goes back to #1 and #2 above)
cost control: in dev, test, staging environments you may want to give people pretty wide permissions to spin up new resources but put low spending caps to prevent accidental usage spikes; conversely in a production account you'll want restricted ability to spin up resources but higher spending caps; easy to enforce via separate account - not so much in the same account
Did I miss anything? Possibly! But these are the reasons why I would use separate accounts.
By the way - I am NOT advocating against using VPCs. They exist for a reason and you should definitely use VPCs for network isolation. What I am trying to argue is that anybody who also uses other services such as DynamoDb, Lambda, SQS, S3 etc - VPCs are not really the way to isolate resources, IMO.
The downsides to one account per stage that I can think of are mostly around continuous deployment if you use tools that are not flexible enough to be able to deploy to different accounts.
Finally, some people like to call on billing as a possible issue but really, wouldn’t you want to know how much money you spend on Production vs Staging vs Development ?!
Avoid separate accounts for each environment to avoid additional complexity and obstacles in accessing shared resources.
Try rather using:
resource groups
tagging
as recommended by AWS:
https://aws.amazon.com/blogs/startups/managing-resources-across-multiple-environments-in-aws/
The account separation is recommended by the AWS Well Architected Framework security pillar.
We are developing a way of automating the process of running Dataproc on GCP (everything from starting a cluster to submitting a job, and terminating the cluster) for genomic analyses. Ultimately, we want to be aware of how much is being spent for each of these automated “runs” of Dataproc, especially as we open up our program to other users. Therefore, we would also like to automate (if possible) the process of attributing Dataproc costs to a certain GCP user (“run”). That way, at the end of each month, we can see how much was spent per user and specifically for what personal project. What is a good strategy for implementing this?
I have considered creating a separate GCP project for each user/client and tying them to a particular billing account, but is there perhaps a better way of managing this?
The recommended way is to apply labels to your resouces [1]. Billing data can be filtered and segmented by labels [2].
In your example, both user and client would be labels on Cluster and Job resources. Only Cluster resources are billed.
If you automation is really just a one-shot create-cluster/submit-job(s)/delete-cluster flow, then consider Workflow Templates for automating this. Specifically Inline templates are tailor made for this kind of automation.
[1] https://cloudplatform.googleblog.com/2015/10/using-labels-to-organize-Google-Cloud-Platform-resources.html
[2] https://cloud.google.com/billing/docs/how-to/export-data-bigquery
An infrastructure (include both dev and prod environments) for an application has been made on an AWS account that is quite big, includes 15 instances,... Now, we're gonna make a new infrastructure for another application. I would like to know if it's better to create another AWS account for the new project. What would be the advantages?
Although I prefered to have separated account for each environment than projects but as the first project is made on one account compeletly, so I think the only better way is to atleast create another AWS account for the new project.
Plus,in any case, is there any easy way to transfer production env to another account inorder to separate the environments?
Any suggestion would be appreciated.
I'm not sure as to the circumstances in your case but I imagine having a separate account for each environment does give you more control and less room for error.
If you're working alone, try to determine this for yourself whether the effort is worth it. Should you be part of a team or even leading a team, if someone has access to the 'global' aws account with both the development and production instances, errors can easily be made. If you're consuming the AWS API for example and terminate the wrong instance... Food for thought.
Another reason would be that you will need to become very very granular with your IAM roles should you wish to worth with a global account with each environment in it to keep some level of control.
Lastly, cloudwatch will give you nice detailed reports on how your instances are doing and when you have all environments in their respective AWS accounts, it becomes a quick way to see which servers are operating in which fashion.
If all your environments are in the same account, this can become quite confusing as to which instances are production / development.
TLDR, it is good practise to split up the different environments to keep a higher level of control and overview.
Today (I know I'm answering a very old question), AWS makes it easy and very useful to group accounts into Organizations.
For a big setup, this means you can consolidate billing, reservations and other reductions, as well as many security and compliance aspects, while keeping each account operationally separate. While it may be some overhead for a small setup it will be less overhead than trying to keep separate two development teams that are using one account, and extra costs are small to none.
In short, there are a number of very significant advantages and as far as I can see no significant downsides to separating different spheres of responsibility into different accounts.