I am trying to programmatically recreate a bunch of AWS resources that were created/configured manually via AWS consoles.
The AWS consoles do a lot for you.
For example, you can create a Lambda function with an Api-Gateway trigger in about 10 seconds using the AWS console.
The console is doing a lot of magic under the covers, defining and configuring resources such as policies, stages, permissions, models, etc.
In theory, CloudTrail is supposed to allow me to see what exactly is happening under the covers, but it seems to be silent in this case (i.e. Lambda function with Api-Gateway trigger).
I can play hide and seek and do extensive dumps using the CLI to list stages, policies, export api definitions, etc. etc. and look for the differences but is there an easier way? - like some way to trace the REST calls that the console is creating when it does all its magic?
Note: CloudFormer could have helped but it is only half-written software (Hey Amazon!) and only covers about a third of the resources I have defined. Does embracing Cloudformation imply not using these great time-saving consoles?
CloudFormation and other Infrastructure as code services are there to lessen the clicks you make while using AWS console or any other cloud console to manage your resources.
But these come in handy when you have to spin up resources which will be having almost the same configurations and software stack.
If you use CloudFormation you will be able to define the policies according to your need, which OS image to use, which stack to install etc. etc. it provides you minute control over your resources.
I suggest if you have to deploy these resources multiple times then create a CloudFormation template and use it.
So, I would suggest that rather than finding a way to recreate the code from your current infrastructure, create a CloudFormation template and use it for future needs.
If you need something easier than your current flow, this is it, as you just have to write your required configuration once.
Hashicorp Terraform is also a good alternative to AWS CloudFormation. You can use Terraforming to export the current infrastructure into Terraform readable files.
Related
I would like to understand the need of tools like Terraform. When we do have Cloudformation template available and one can create/update all AWS services with that , What is the point in using a service like Terraform.
Please Suggest.
CloudFormation (CFN) and Terraform (CF) are both Infrastructure as Code (IaC) development tools.
However, CFN is only for AWS. You can't use it with Azure, GCP or anything else outside of AWS ecosystem. In contrast, TF is cloud agnostic. You can use it across not only multiple cloud providers, but also to work with non-cloud products, such as docker, various databases and even domino pizza if you want.
So the main advantage of TF is that once you learn it only once, you can apply it to a number of cloud providers. CFN is only useful in AWS, and once you stop using CFN, you have to learn something new to work with other cloud.
There are also difference in how TF and CFN work. Both have their strengths and weekends. For example:
when you deploy using CFN all resources are available to view in one central location in AWS along with template's source code. Whereas with TF there is no such place. If you login to the AWS console, you have no idea what was created by TF, what was the source code used, etc.
TF has loops and complex data structures and condtions, while CFN does not.
CFN has creation policies and update policies, TF has not.
You can control access to CFN using CFN policies and IAM policies. You can't do same with TF as it "lives" outside of AWS.
There are a couple of reasons why you might choose Terraform over CloudFormation:
Vendor Agnostic: There might be a point in the future where you need to migrate your cloud infrastructure. This could be due to several reasons (e.g. costs, regulatory compliance, etc.). With Terraform you are still able to use the same tool to deploy the new infrastructure. With smart use of Terraform modules you can even leave large parts of your infrastucture as code repository in tact.
Support for other tools: This also builds a bit on the previous point, but Terraform can deploy a lot more then just AWS resources. For example, you can use Terraform to orchestrate the deployment of an EC2 machine that is then configured with Ansible. Or you could use Terraform to deploy applications on top of your Kubernetes cluster. While CloudFormation supports custom resources via the creation of custom Lambdas, it is quite a lot of work to maintain.
Wider ecosystem: Due to the Open Source nature of Terraform, there is a huge ecosystem of tools that help you solve all kinds of issues, such as testing the infrastructure as code or building in compliance in a continuous fashion.
Arguably a better language: Personally I think Terraform is a way more suited for Infrastructure as Code then CloudFormation. Terraform has a lot more flexibility build in to the language (HCL) and their module system allows for a lot more composability then what can be achieved in CloudFormation.
I am familiar with AWS SDKs (Python/Java) and the need for the same for a lot of services like S3, DDB, KMS etc.
Are there any valid use cases for using the AWS Java SDK e.g. to programmatically spin off an EC2 instance or an EMR cluster or Redshift cluster or RDS instance or for that matter any resource that requires setting up of an infrastructure/cluster?
If AWS console is not an option, and programmatic access is all we have then, don't we always end up using AWS CLI for corresponding services or CloudFormation or Terraform for that matter?
Generally its best practice to manage any infrastructure/service deployments by using an infrastructure as code solutions such as CloudFormation, CDK (which generates CloudFormation stacks under the hood) or Terraform.
Whilst you could use the SDK to create this services (and create a solution that is similar to the solutions above) you will be building a lot of functionality that other services have already created which would put more ownership on you to fix it if you want to support another service etc.
The SDKs simply provide every AWS API interaction in a programmatic way, even under the hood CloudFormation and Terraform will likely be using the SDKs to programmatically create the resources (although I am speculating) but would then add additional functionality such as state management and drift detection on top of this.
I only have seen services being created via the SDKs when that service is not available in the selected tool, and even then it would generally be wrapped within that tool (such as custom resources for CloudFormation).
In summary, yes you could use the SDK to generate these but unless there is a specific usecase to use the SDK I would advise using a tool that already manages this so you can focus more on your infrastructure/applications.
The AWS CLI is built using the AWS SDK for Python. Terraform is built using the AWS SDK for GoLang. You may want to stick with higher level infrastructure-as-code tools, but those tools wouldn't exist without the SDKs, and if you wanted to build a tool like that you would most likely build it on top of one of the SDKs.
There are also quite a few use-cases I've seen discussed here on StackOverflow for performing infrastructure automation through AWS Lambda, for example periodically stopping and starting RDS instances, where using one of the AWS SDKs in the Lambda code would likely be much easier than trying to get Terraform to run inside a Lambda function.
Would anybody have any clues as to how AWS CloudFormation works under the hood?
Also, would anybody know an open-source equivalent to AWS CF (and I don't mean tooling that may be using CloudFormation)?
It's clearly a powerful orchestrator, but I'd be keen to explore the inner workings of such tools.
AWS Cloudformation has multiple pre-defined set of schemas for each of the components that are supported. When you upload a Cloudformation template for creating resources, it performs the below steps:
It validates the templates against the schema
It generates dynamic form for gathering parameters
It validates the values of parameters
Once it has all it needs, you can click Create to begin with the resource creation
Under the hood, it starts creation of resources using the internal coding for which is keeps echoing the status and progress continuously on the console.
We need to understand here that internally Cloudformation in itself is a product that does use AWS SDK/CLI as needed. However, under the hood, it maintains its own data to compare the attributes and resources when you run an update.
An open source alternative to this is Terraform. Terraform is the most widely used open source replacement of Cloudformation. Terraform is known for its Cloud independent architecture. Terraform works with multiple clouds with minimal changes in the templates.
The under-the-hood working of terraform involves creation of a State file/directory where it stores the current state of any stack identified uniquely by the name provided by the user. Terraform creates resources majorly using Python SDK (boto3) and some other APIs as needed. We need to pass the access key and secrets to the Terraform configuration in order to enable it to access the AWS Cloud environment.
If you are looking to build a smart new alternative, it should be fairly simple considering that AWS strictly follows standard design patterns in its SDK and CLI interface design. This makes it easier to convert template into executable code.
More information about working of Cloudformation can be found here
I recently started working with AWS and IaC, I'm using Cloudformation to provision my AWS resources, but I discovered that AWS provide both a SDK and a CDK to enable you to provision resources programmatically instead of plain json/yaml.
But based on the documentation I did not really understand how they differ, can someone explain me how they differ and for what use case you should use what?
CDK: Is a framework to model and provision your infrastructure or stack. Stack can consist of a database for ex: DynamoDB, S3 Bucket, Lambda, API Gateway etc. It provides a facility to write code to create an infrastructure in AWS. Also called Infrastructure as code.
Check here
SDK: These are the code libraries provided by Amazon in various languages, like Java, Python, PHP, Javascript, Typescript etc. These libraries help interact with AWS services (like creating data in DynamoDB) which you either create through CDK or console. SDKs simplify using AWS services in your application with an API.
Check here
AWS SDK is a library primarily to ease the access to the AWS services by handling for you the data (de)serialization, credentials management, failure handling, etc. Perhaps, for specific scenarios, you could use the AWS SDK as the infrastructure as a code tool, however it could be cumbersome as it is not the intended usage of the library.
Based on the https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html, dedicated tools for the IaC are AWS CloudFormation and AWS CDK.
AWS CDK is an abstraction on top of CloudFormation. CDK scripts are in fact transformed to the CloudFormation definitions when scripts are synthesized.
The difference can be best described on an example: Imagine that for each lambda function in your stack you want to create an error CloudWatch alarm and connect to the SNS topic.
With CloudFormation you will either a) need to write a pretty much similar bunch of yaml/json definitions for each lambda function to ensure the monitoring, b) use the nested stack templates, c) use CloudFormation modules.
With CDK you can write a generic code construct - class or method, which can create the alarm for the given lambda function and create the SNS alarm action for given topic.
In other words, CDK helps you generalize and re-use your IaC in a very familiar way to how you develop your business code. The code is shorter and more readable than the CF definitions.
The difference is even more remarkable when you need to set up similar resources in different AWS regions and when you have different AWS account per environment. You can manage all AWS accounts and regions with a single CDK codebase.
Some background first: CloudFormation is Amazon's solution for an “Infrastructure as Code” approach to managing the definition, provisioning and deployment of a bunch of resources across accounts/regions. This is done by using their declarative yaml/json-based template language to define it all, and then executing the templates through various means (console, cli, APIs...). More info:
white paper: https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html
faq: https://aws.amazon.com/cloudformation/faqs/
There are other popular IaC solutions or tools to help achieve it more easily out there, such as Terraform and Kubernetes (container orchestration that also uses declarative templates to define desired states).
Potential benefits of IaC: At a high level, you can better track & audit your infra, reuse definitions/processes, make all your changes in a more consistent manner, faster thanks to all the automation and assurances you can get with an infra-as-code approach. You may be familiar with these as mentioned in previous answers and more, such as:
version controlling your infrastructure definitions,
more efficient and logically complex ways of constructing templates,
ability to write tests against them,
do diffs (see "change sets") before making real infra changes with the templates,
detect when live infra differs from your definitions,
automate rollbacks,
and lots of other state management assistance through a framework like CF that might be needed when performing regular ops duties.
CDK:
This is for helping to automate CloudFormation as part of an IaC approach to provisioning and deploying resources. It lets you use various popular programming languages to help with the creation, testing, and management of your CF setup. Some of AWS’s motivations: “YAML is an excellent format for describing the desired state of your cluster, but it is does not have primitives for expressing logic and reusable abstractions.“ “AWS CDK uses the familiarity and expressive power of programming languages for modeling your applications.”
More info: https://docs.aws.amazon.com/cdk/v2/guide/home.html
However, Amazon knows about other solutions, and happily points them out on the main CDK page now, downplaying its original connection to CF. You don't need to use CloudFormation if you don't want to; specifically, they mention you can use the same CDK constructs with the help of:
cdktf for Terraform maintained by its creators, Hashicorp
cdk8s for Kubernetes by AWS. re: “We realized this was exactly the same problem our customers had faced when defining their applications through CloudFormation templates, a problem solved by the AWS Cloud Development Kit (AWS CDK), and that we could apply the same design concepts from the AWS CDK to help all Kubernetes users.”
SDK:
AWS has an API for all of their services, and the various SDKs give you access to them. For example, I can use AWS’s Java SDK to manage an API Gateway. If I wanted to script some custom deployment process, I could do so with the SDK, managing all the state, etc. myself. You could probably even re-implement the CloudFormation service with the various underlying APIs... The APIs have varying levels of documentation though. E.g. CloudFormation Java APIs are only mentioned in the raw API reference, not the friendlier Developer Guide.
I find that the difference for me is that the CDK codifies the CloudFormation JSON/YAML. First response, is great ya okay in code but the benefit on the code side of things is you can write unit testing against the code. Therefore you get to build that sense of security or insurance policy against the provisioned services in the CDK.
There are other ways to test CF, however, with a dev background, this feels more comfortable.
Is there any known downside to creating resources on aws through the CLI? Is it more reliable/easier/error prone/largely accepted/recommended to use one method over the other? While setting up recurring scripts, is there a reason why i would want to use CloudFormation or the AWS Console over the AWS CLI to run commands directly?
For example, if I were to create an ECS Fargate Task Definition, is there any reason why I might want to use AWS CloudFormation or the Console over AWS CLI? Cli syntax is straightforward and easy to use, and there are a few things (like setting up event rules/targets for a fargate task specifically) that are not supported via cloudformation yet.
The AWS CLI and AWS CloudFormation are two different tools that can be used to create infrastructure on AWS. The CLI is more powerful and has finer grained control than CloudFormation. CloudFormation makes it very easy to use yaml or json text files that can describe an entire enterprise in the cloud.
One of the strong benefits of CloudFormation is the automatic support for rolling back changes if anything fails while deploying a stack. The CLI in comparison would require you to figure out the details of what went wrong and how to get back to where your state was. Updating infrastructure using CloudFormation is another benefit. Make the change in the template and update the stack.
For small setups, using the CLI is fine. However, once you get past launching an EC2 instance and start building VPCs, Instances, KeyPairs, Security Groups, RDS, etc. etc. you will find that the CLI has some real limitations: mostly being too manual of a process, not easily repeatable, difficult to put the process into version control, ....
If you are constantly building, testing and deleting complex setups, CloudFormation is absolutely one of the best tools from AWS. Note that there are a number of third party solutions that have a huge number of followers such as Bamboo, Octopus, Jenkins, Chef, etc.
If your job is SysOps or DevOps then you absolutely want to master the CLI and CloudFormation. These are amazing tools for working with AWS. Also master Beanstalk, maybe OpsWorks and one of the third party tools like Jenkins.