Azure Resource group is one thing which ties one complete stack together, so in case you want to know what your complete stack looks like just open up your RG and all resources are there(not talking about very big stacks ).
Do we have any service in AWS like this ??
AWS also has Resource Groups. It's not identical to Azure resource groups. But it will work for your need. You need to use tags for the resources and then group them using AWS Resource Groups. Tags are very powerful and widely used in AWS.
An AWS Cloudformation template creates a "Stack" with all the resources defined in the template. In the AWS console what you see under the Cloudformation service are these stacks. They seem to me to be very much like an Azure Resource Group. The life cycle of the resources in the template are managed by the stack. If you delete the stack all the resources are deleted; very much like an Azure Resource Group.
As mentioned in comments you can group together AWS resources using resource groups. If you are looking for more than mere grouping then there is AWS OpsWorks Stacks. AWS OpsWorks Stacks is nothing but chef under the hood. Here you can get the full view of associated resources and you can manage as well.
I've been playing in Azure for a while with Kubernetes (AKS) and Terraform. I was used to deploying an Azure resource group and everything in it:
Key vault
storage account
DataBricks
Data Factory
SQL Server (or some other database)
Simple enough. So, I just tried to deploy a similar data stack in AWS. It's not at all the same, and a lot more effort.
The VPC seems to be the first place to start. You'll also have to
think about CIDR addresses too.
Subnets (you'll be defining CIDR blocks here too) and deciding availability zones come next.
You'll need security group
You'll also need IAM resources and to attach the relevant roles/policies. In Azure, once permissions and Service Principals are in place, you don't have to worry about these things.
All this is before you've deployed anything.
Now that's my immediate feeling trying to replicate what I was doing on Azure. I didn't let how long it takes to deploy these resources on AWS (in the EU region - YMMV) compared to Azure 🙄 cloud my judgement either...
A Cloudformation 'stack' was created when I tested using the AWS EKSCLI command line tool to provision a kubernetes cluster and nodes. Deleting a stack on Cloudformation will remove all the associated resources, like if you were to remove a resource group on Azure.
I think using Terraform is probably a good idea, which is the conclusion you may arrive at after having to chase down why you can't delete a certain resource because something is still using it (and it's not all in the same place (resource group) like in Azure.
Related
Please how can get the list of all services I am using.
I have gone to Service Quotas at
https://ap-east-1.console.aws.amazon.com/servicequotas/home?region=ap-east-1
on the dashboard. I could see a list of Items e.g. EC2, VPC, RDS, Dynamo etc but I did not understand what is there.
As I did not request for some of the services I am seeing I even went into budget at
https://console.aws.amazon.com/billing/home?region=ap-east-1#/budgets
and also credits. Maybe I can get the services I have been given credits to use
https://console.aws.amazon.com/billing/home?region=ap-east-1#/budgets?
Also, how can I stop any service which I do not want?
The Billing service is not giving me tangible information also. I do not want the bill to pile up before I start taking needed steps.
Is there a location where I can see all services I am using or maybe there is a code I can enter somewhere which would produce such result?
You can use AWS Config Resource Inventory feature.
AWS Config will discover resources that exist in your account, record their current configuration, and capture any changes to these configurations. Config will also retain configuration details for resources that have been deleted. A comprehensive snapshot of all resources and their configuration attributes provides a complete inventory of resources in your account.
https://aws.amazon.com/config/
There is not an easy answer on this one, as there is not an AWS service that you can use to do this out of the box (yet).
There are some AWS services that you can use to get you close, like:
AWS Config (as suggested by #kepils)
Another option is to use Resource Groups and Tagging to list all resources within a region within account (as described in this answer).
In both cases however, the issue is that both Config and Resource Groups come with the same limitation - they can't see all AWS services on their own.
Another option would be to use a third party tool to do this, if your end goal is to find what do you currently have running in your account like aws-inventory or cloudmapper
On the second part of your question on how to stop any services which you don't want you can do the following:
Don't grant excessive permissions to your users. If someone needs to work on EC2 instances, then their IAM role and respective policy should allow only that instead of for example full access.
You can limit the scope and services permitted for use within account by creating Service Control Policies which are allowing only the specific resources you plan to use.
Set-up an AWS Budget Notifications and potentially AWS Budget Actions.
Had a question regarding infrastructure as code on AWS.
Wondering how to do this (the process of deploying) and also why is this an efficient method for architecture? Also, are there other methods that should be looked at over this?
I am looking to deploy this for a startup I am working for and need assistance in getting this going. Any help is appreciated.
Thank you.
From What is AWS CloudFormation?:
AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; AWS CloudFormation handles all of that.
So, instead of manually creating each bit of architecture (network, instances, queues, storage, etc), you can define them in a template and CloudFormation will deploy them. It is smart enough to mostly know the correct order of creation (eg creating the network before creating an Amazon EC2 instance within the network) and it can also remove resources when the 'stack' is no longer required.
Other benefits:
The template effectively documents the infrastructure
Infrastructure can be checked into a source code repository, and versioned
Infrastructure can be repeatedly and consistently deployed (eg Test environment matches Production environment)
Changes can be made to the template and CloudFormation can update the 'stack' by just deploying the changes
There are tools (eg https://former2.com/) that can generate the template from existing infrastructure, or just create it from code snippets taken from the documentation.
Here's an overview: Simplify Your Infrastructure Management Using AWS CloudFormation - YouTube
There are also tools like Terraform that can deploy across multiple cloud services.
I am exploring backing up our AWS services configuration to a backup disk or source control.
Only configs. eg -iam policies, users, roles, lambdas,route53 configs,cognito configs,vpn configs,route tables, security groups etc....
We have a tactical account where we have created some resources on adhoc basis and now we have a new official account setup via cloud formation.
Also in near future planning to migrate tactical account resources to new account either manually or using backup configs.
Looked at AWS CLI, but it is time consuming. Any script which crawls through AWS and backup the resources?
Thank You.
The "correct" way is not to 'backup' resources. Rather, it is to initially create those resources in a reproducible manner.
For example, creating resources via an AWS CloudFormation template allows the same resources to be deployed in a different region or account. Only the data itself, such as the information stored in a database, would need a 'backup'. Everything else could simply be redeployed.
There is a poorly-maintained service called CloudFormer that attempts to create CloudFormation templates from existing resources, but it only supports limited services and still requires careful editing of the resulting templates before they can be deployed in other locations (due to cross-references to existing resources).
There is also the relatively recent ability to Import Existing Resources into a CloudFormation Stack | AWS News Blog, but it requires that the template already includes the resource definition. The existing resources are simply matched to that definition rather than being recreated.
So, you now have the choice to 'correctly' deploy resources in your new account (involves work), or just manually recreate the ad-hoc resources that already exist (pushes the real work to the future). Hence the term Technical debt - Wikipedia.
I want to understand the exact limitations of OpsWorks- things that we simply may not be able to do or not optimally. That would require to go to the next level - CloudFormation. Of course we can use OpsWorks + CloudFormation - get best of both worlds - a best practice.
Some limitations of OpsWorks that I am aware of are - it cant provision everything - like EMR, S3 etc - but have never found on exhaustive list. Also OpsWorks the AutoScaling configuration has limitations. And CloudFormation lets us version control the environment unlike OpsWorks. Do understand that there is good amount of overlap and CloudFormation does add to complexity.
There was a previous discussion but the demarcations were not made clear.
OpsWorks is a totally different service from CloudFormation.
OpsWorks is focused at managing applications layered as stacks, and taking advantadge of chef recipes for setting up and deploying applications.
CloudFormation is a descriptive language to create sets of AWS infrastructure.
It may look obvious, but the thing is that whenever you prefer to manage an application and its deployment cycle, the OpsWorks service is better suited. You can of course use cloud formation to define entire applications and layers in OpsWorks and that will allow you to replicate entire applications layer sets (for test environments and so)
The only good way to learn the boundaries of every service is using them for your needs, then you will find where Opsworks is strong and where Cloudformation complements or allows you to automate your Opsworks setups.
About versioning, CF lets you version the infrastructure stack, which may have nothing to do with the version of code that you are managing through opsworks.
regards
In many cases CloudFormation is used with OpsWorks; where CloudFormation is used to provision the infrastructure and OpsWorks to configure the resources created. OpsWorks (via Chef or Puppet) provides a richer set of abilities when it comes to configuring the application stack then the simple shell scripting offered by CloudFormation. So, generally CloudFormation is used to deploy AWS resources, and OpsWorks is used to do the detailed configuration of the application(s)/operating system(s).
In some ways one can say that CloudFormation is focused more with a collection of AWS infrastructure resources rather than the application itself; but as you have alluded to some things can be done by both services interchangeably.
This AWS link may help where it states:
“Compared to AWS CloudFormation, AWS OpsWorks Stacks supports a narrower range of application-oriented AWS resource types including Amazon EC2 instances, Amazon EBS volumes, Elastic IPs, and Amazon CloudWatch metrics.”
and this is certainly not a limitation for CloudFormation.
I am creating an AWS stack inside a VPC using CloudFormation and need to create ElastiCache clusters on it. I have investigated and there is no support in CloudFormation to create cache clusters in VPCs.
Our "workaround" was to to create the cache cluster when some "fixed" instance (like a bastion for example) bootstrap using CloudInit and AWS AmazonElastiCacheCli tools (elasticache-create-cache-subnet-group, elasticache-create-cache-cluster). Then, when front end machines bootstrap (we are using autoscaling), they use elasticache-describe-cache-clusters to get cache cluster nodes and update configuration.
I would like to know if you have different solutions to this problem.
VPC support has now been added for Elasticache in Cloudformation Templates.
To launch a AWS::ElastiCache::CacheCluster in your VPC, create a AWS::ElastiCache::SubnetGroup that defines which subnet in your VPC you want Elasticache and assign it to the CacheSubnetGroupName property of AWS::ElastiCache::CacheCluster.
You workaround is a reasonable one (and shows that you seem to be in control of your AWS operations already).
You could improve on your custom solution eventually by means of the dedicated CustomResource type, which are special AWS CloudFormation resources that provide a way for a template developer to include resources in an AWS CloudFormation stack that are provided by a source other than Amazon Web Services. - the AWS CloudFormation Custom Resource Walkthrough provides a good overview of what this is all about, how it works and what's required to implement your own.
The benefit of using this facade for a custom resource (i.e. the Amazon ElastiCache cluster in your case) is that its entire lifecycle (create/update/delete) can be handled in a similar and controlled fashion just like any officially supported CloudFormation resource types, e.g. resource creation failures would be handled transparently from the perspective of the entire stack.
However, for the use case at hand you might actually just want to wait for official support becoming available:
AWS has announced VPC support for ElastiCache in the context of the recent major Amazon EC2 Update - Virtual Private Clouds for Everyone!, which boils down to Default VPCs for (Almost) Everyone.
We want every EC2 user to be able to benefit from the advanced networking and other features of Amazon VPC that I outlined above. To enable this, starting soon, instances for new AWS customers (and existing customers launching in new Regions) will be launched into the "EC2-VPC" platform. [...]
You don’t need to create a VPC beforehand - simply launch EC2
instances or provision Elastic Load Balancers, RDS databases, or
ElastiCache clusters like you would in EC2-Classic and we’ll create a
VPC for you at no extra charge. We’ll launch your resources into that
VPC [...] [emphasis mine]
This update sort of implies that any new services will likely be also available in VPC right away going forward (else the new EC2-VPC platform wouldn't work automatically for new customers as envisioned).
Accordingly I'd expect the CloudFormation team to follow suit and complete/amend their support for deployment to VPC going forward as well.
My solution for this has been to have a controller process that polls a message queue, which is subscribed to the SNS topic which I notify CloudFormation events to (click advanced in the console when you create a CloudFormation stack to send notifications to an SNS Topic).
I pass the required parameters as tags to AWS::EC2::Subnet and have the controller pick them up, when the subnet is created. I execute the set up when a AWS::CloudFormation::WaitConditionHandle is created, and use the PhysicalResourceId to cURL with PUT to satisfy a AWS::CloudFormation::WaitCondition.
It works somewhat, but doesn't handle resource deletion in ElastiCache, because there is no AWS::CloudFormation::WaitCondition analogue for stack deletion. That's a manual operation procedure wth my approach.
The CustomResource approach looks more polished, but requires an endpoint, which I don't have. If you can put together an endpoint, that looks like the way to go.