Is it possible to use AWS CloudFormation Templates to validate that already existing AWS resources are configured properly? What I would like to do is create a JSON file according to the AWS EC2 Instance resource specification and validate that the instance is configured properly using BOTO3. I haven't been able to find this type of functionality in the AWS API, and was wondering if there was something that I missed, or there is a 3rd party tool that will do this for me.
Thank you for your time.
I don't believe this is possible - you'd run into some interesting issues in a lot of cases, as CloudFormation doesn't always name resources exactly the way you'd expect. Cloudformation is meant more for building resources and maintaining them, not verifying existing outside-of-cloudformation resources.
Cool idea for a tool though, not sure if anything like that already exists. It should be pretty easy to write something that snapshots various resources, then compares them over time using boto.
Good luck! If you write a tool (or find one) let me know - though almost all of our resources are managed in CloudFormation already (via stacker).
Related
Recently, while trying to build a terraform IaC, I found that I couldn’t get the API Gateway to route to the Lambda properly. It turned out that when using the console AWS automatically assigns the permissions the gateway needs for the Lambda, but with IaC in terraform this must be assigned explicitly.
The above is understandable but for a newbie, to both AWS and terraform, confusing.
Is there documentation which explains the required components within an infrastructure connection, such as that above?
I know of the AWS docs and the terraform docs are particularly well thought out but none of it actually explains (as far as I’ve seen) that a certain resource is required in any particular (however common or obscure) setup. Inferring these connections from general searching is not a great replacement.
I don't think that there is a documentation that lists "all of the required components" in one single page/area. But you can get different pieces of information from different docs, and as you mentioned AWS and Terraform do both a great job at this.
Talking about AWS, in the case of permissions in API gateway, I can think of two useful links (the 1st one is referenced from the 2nd one though):
How API Gateway resource policies affect authorization workflow
Control access for invoking an API
I agree in the fact that sometimes it's a lot of guesses to translate AWS into terraform if you don't really know what you are trying to achieve. Usually when I am blocked on something that "should theoritically work" in IaC vs AWS console, I step back from the problem and try to figure out what kind of components am I really trying to glue together in AWS world. Then usually things become more obvious.
Because in terraform it's really creating small independant pieces of infrastructure and make them work together. Comparing with other IaC, in my experience it's a lot more granular than CloudFormation for instance.
A personal practice that helps me figure out things faster is to read every single intro doc of the components I am working on in Terraform. For instance, if I am writing lambda in terraform IaC, I would quickly read all the lambda_xxxx_yyyy intro parts to get less stuck and react faster when something fails. It usually works for me.
I haven't see such a documentation, but I can share my work-around for similar cases.
You can make changes you need using AWS console - manually, using UI. Then you can define resources you just created in your TF files, defining only/required required set of properties, even random values will work. Then you import what you created manually into resources you defined.
By running terraform plan you will see the differences, that will allow you to adjust your TF files accordingly.
After few iterations you will replicate what you have just done in the UI using TF. As a final test you can manually revert your changes, run terraform apply and ensure that everything works as expected.
Would anybody have any clues as to how AWS CloudFormation works under the hood?
Also, would anybody know an open-source equivalent to AWS CF (and I don't mean tooling that may be using CloudFormation)?
It's clearly a powerful orchestrator, but I'd be keen to explore the inner workings of such tools.
AWS Cloudformation has multiple pre-defined set of schemas for each of the components that are supported. When you upload a Cloudformation template for creating resources, it performs the below steps:
It validates the templates against the schema
It generates dynamic form for gathering parameters
It validates the values of parameters
Once it has all it needs, you can click Create to begin with the resource creation
Under the hood, it starts creation of resources using the internal coding for which is keeps echoing the status and progress continuously on the console.
We need to understand here that internally Cloudformation in itself is a product that does use AWS SDK/CLI as needed. However, under the hood, it maintains its own data to compare the attributes and resources when you run an update.
An open source alternative to this is Terraform. Terraform is the most widely used open source replacement of Cloudformation. Terraform is known for its Cloud independent architecture. Terraform works with multiple clouds with minimal changes in the templates.
The under-the-hood working of terraform involves creation of a State file/directory where it stores the current state of any stack identified uniquely by the name provided by the user. Terraform creates resources majorly using Python SDK (boto3) and some other APIs as needed. We need to pass the access key and secrets to the Terraform configuration in order to enable it to access the AWS Cloud environment.
If you are looking to build a smart new alternative, it should be fairly simple considering that AWS strictly follows standard design patterns in its SDK and CLI interface design. This makes it easier to convert template into executable code.
More information about working of Cloudformation can be found here
I recently started working with AWS and IaC, I'm using Cloudformation to provision my AWS resources, but I discovered that AWS provide both a SDK and a CDK to enable you to provision resources programmatically instead of plain json/yaml.
But based on the documentation I did not really understand how they differ, can someone explain me how they differ and for what use case you should use what?
CDK: Is a framework to model and provision your infrastructure or stack. Stack can consist of a database for ex: DynamoDB, S3 Bucket, Lambda, API Gateway etc. It provides a facility to write code to create an infrastructure in AWS. Also called Infrastructure as code.
Check here
SDK: These are the code libraries provided by Amazon in various languages, like Java, Python, PHP, Javascript, Typescript etc. These libraries help interact with AWS services (like creating data in DynamoDB) which you either create through CDK or console. SDKs simplify using AWS services in your application with an API.
Check here
AWS SDK is a library primarily to ease the access to the AWS services by handling for you the data (de)serialization, credentials management, failure handling, etc. Perhaps, for specific scenarios, you could use the AWS SDK as the infrastructure as a code tool, however it could be cumbersome as it is not the intended usage of the library.
Based on the https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html, dedicated tools for the IaC are AWS CloudFormation and AWS CDK.
AWS CDK is an abstraction on top of CloudFormation. CDK scripts are in fact transformed to the CloudFormation definitions when scripts are synthesized.
The difference can be best described on an example: Imagine that for each lambda function in your stack you want to create an error CloudWatch alarm and connect to the SNS topic.
With CloudFormation you will either a) need to write a pretty much similar bunch of yaml/json definitions for each lambda function to ensure the monitoring, b) use the nested stack templates, c) use CloudFormation modules.
With CDK you can write a generic code construct - class or method, which can create the alarm for the given lambda function and create the SNS alarm action for given topic.
In other words, CDK helps you generalize and re-use your IaC in a very familiar way to how you develop your business code. The code is shorter and more readable than the CF definitions.
The difference is even more remarkable when you need to set up similar resources in different AWS regions and when you have different AWS account per environment. You can manage all AWS accounts and regions with a single CDK codebase.
Some background first: CloudFormation is Amazon's solution for an “Infrastructure as Code” approach to managing the definition, provisioning and deployment of a bunch of resources across accounts/regions. This is done by using their declarative yaml/json-based template language to define it all, and then executing the templates through various means (console, cli, APIs...). More info:
white paper: https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html
faq: https://aws.amazon.com/cloudformation/faqs/
There are other popular IaC solutions or tools to help achieve it more easily out there, such as Terraform and Kubernetes (container orchestration that also uses declarative templates to define desired states).
Potential benefits of IaC: At a high level, you can better track & audit your infra, reuse definitions/processes, make all your changes in a more consistent manner, faster thanks to all the automation and assurances you can get with an infra-as-code approach. You may be familiar with these as mentioned in previous answers and more, such as:
version controlling your infrastructure definitions,
more efficient and logically complex ways of constructing templates,
ability to write tests against them,
do diffs (see "change sets") before making real infra changes with the templates,
detect when live infra differs from your definitions,
automate rollbacks,
and lots of other state management assistance through a framework like CF that might be needed when performing regular ops duties.
CDK:
This is for helping to automate CloudFormation as part of an IaC approach to provisioning and deploying resources. It lets you use various popular programming languages to help with the creation, testing, and management of your CF setup. Some of AWS’s motivations: “YAML is an excellent format for describing the desired state of your cluster, but it is does not have primitives for expressing logic and reusable abstractions.“ “AWS CDK uses the familiarity and expressive power of programming languages for modeling your applications.”
More info: https://docs.aws.amazon.com/cdk/v2/guide/home.html
However, Amazon knows about other solutions, and happily points them out on the main CDK page now, downplaying its original connection to CF. You don't need to use CloudFormation if you don't want to; specifically, they mention you can use the same CDK constructs with the help of:
cdktf for Terraform maintained by its creators, Hashicorp
cdk8s for Kubernetes by AWS. re: “We realized this was exactly the same problem our customers had faced when defining their applications through CloudFormation templates, a problem solved by the AWS Cloud Development Kit (AWS CDK), and that we could apply the same design concepts from the AWS CDK to help all Kubernetes users.”
SDK:
AWS has an API for all of their services, and the various SDKs give you access to them. For example, I can use AWS’s Java SDK to manage an API Gateway. If I wanted to script some custom deployment process, I could do so with the SDK, managing all the state, etc. myself. You could probably even re-implement the CloudFormation service with the various underlying APIs... The APIs have varying levels of documentation though. E.g. CloudFormation Java APIs are only mentioned in the raw API reference, not the friendlier Developer Guide.
I find that the difference for me is that the CDK codifies the CloudFormation JSON/YAML. First response, is great ya okay in code but the benefit on the code side of things is you can write unit testing against the code. Therefore you get to build that sense of security or insurance policy against the provisioned services in the CDK.
There are other ways to test CF, however, with a dev background, this feels more comfortable.
I am trying to programmatically recreate a bunch of AWS resources that were created/configured manually via AWS consoles.
The AWS consoles do a lot for you.
For example, you can create a Lambda function with an Api-Gateway trigger in about 10 seconds using the AWS console.
The console is doing a lot of magic under the covers, defining and configuring resources such as policies, stages, permissions, models, etc.
In theory, CloudTrail is supposed to allow me to see what exactly is happening under the covers, but it seems to be silent in this case (i.e. Lambda function with Api-Gateway trigger).
I can play hide and seek and do extensive dumps using the CLI to list stages, policies, export api definitions, etc. etc. and look for the differences but is there an easier way? - like some way to trace the REST calls that the console is creating when it does all its magic?
Note: CloudFormer could have helped but it is only half-written software (Hey Amazon!) and only covers about a third of the resources I have defined. Does embracing Cloudformation imply not using these great time-saving consoles?
CloudFormation and other Infrastructure as code services are there to lessen the clicks you make while using AWS console or any other cloud console to manage your resources.
But these come in handy when you have to spin up resources which will be having almost the same configurations and software stack.
If you use CloudFormation you will be able to define the policies according to your need, which OS image to use, which stack to install etc. etc. it provides you minute control over your resources.
I suggest if you have to deploy these resources multiple times then create a CloudFormation template and use it.
So, I would suggest that rather than finding a way to recreate the code from your current infrastructure, create a CloudFormation template and use it for future needs.
If you need something easier than your current flow, this is it, as you just have to write your required configuration once.
Hashicorp Terraform is also a good alternative to AWS CloudFormation. You can use Terraforming to export the current infrastructure into Terraform readable files.
In an architecture where you have all of your components using aws-specific products, like web servers in EC2 instances, your CDN using CloudFront, microservices in ECS, why would someone choose to use Terraform?
It's usually very individual choice. There are some aspects (not really pros and cons, but just points to which you need to answer):
Are you going to be fully AWS-oriented? If yes, of course you can use both, but if not, you probably anyway (if not now, then in future) will need some additional tools, so in such case Terraform can cover both cases.
Do you already know Terraform? If not, then learning either is probably the same challenge, but if you know Terraform from other projects, you can just start using it. That also apply to your hiring strategy - are you going to look for people who know just AWS (then most likely CloudFormation as well), or in general with some Sys/Dev-Ops experience, then Terraform is a common tool.
There is one another important thing - CloudFormation can't handle existing resources (Terraform usually can, with import, however there are exceptions). That means you can migrate from CloudFormation to Terraform (done it few times, not a big deal in fact), but not the oposite.
I don't want to recommend a tool on itself, but more as an approach: start with CloudFormation and then see if it fits you or not. As mentioned in last point, you can always move to Terraform, but if you start witout CloudFormation, then the doors are closed for you.
Also if you are going to be fully AWS-oriented, CloudFormation can play nicely with more of the services: CodePipeline integrates with it, in case of CI/CD pipeline CloudFormation is a fully managed service (it keeps the states, exposes export values etc.), while in case of Terraform you need to provide own infrastructure for states storage and exposing outputs (you can use for example S3 bucket in AWS for that, but still you need to configure Terraform to use it - CloudFormation is a zero-config in such case).