I would like to use CloudFormer in an automated way to generate templates of existing resources in my VPC. I can create the cloudformer instance using Ruby SDK CloudFormation::Client.create_stack() and tear it down using CloudFormation::Client.delete_stack() - but while it's running, I can't figure out how to talk to it to ask it to generate cloudformation templates for resources. I envision doing something like:
for every SecurityGroup in VPC:
ask CloudFormer to generate a cloudformation template of the SecurityGroup
store the template in S3
scan the template for compliance with business rules etc.
end
How do I talk to CloudFormer once it's running (in a non-interactive way)? Has anyone done this? I hope I don't have to use Selenium etc. to talk to it's webUI.
CloudFormer is still in beta phase and does not have command-line-interface tools in the same way that other first-class AWS services do, so, unfortunately, there is no way to interact with the running application programmatically.
Related
I could not find a definite 'yes' or 'no' anywhere, so I thought maybe I ask here. Is it possible to run a custom script which would automatically create a new environment on AWS with all the settings like (Network, Capacity, Security etc.)? I need to create a lot of new environments as I am switching from individual load balancers to shared ones, and all the settings are the same (apart from the environment and application name), so it involves a lot of manual work.
From What is AWS CloudFormation? - AWS CloudFormation:
AWS CloudFormation is a service that helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; CloudFormation handles that.
If you want to create the CloudFormation template programmatically, you can use AWS CDK - AWS Cloud Development Kit (CDK):
The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.
The AWS CDK supports TypeScript, JavaScript, Python, Java, C#/.Net, and Go. Developers can use one of these supported programming languages to define reusable cloud components known as Constructs. You compose these together into Stacks and Apps.
Or, you can simply write your own script in a programming language that calls an AWS SDK to individually create resources in AWS. Everything in AWS can be done via API calls.
I have created an ec2 instance and configured it as a target behind my load balancer. I want to convert this entire environment as something that can be deployed repeatedly in automation. I have looked at cloudformation but don't know if it can help me with converting this environment to a cloudformation template. Is there any other way to achieve this. If there is an approach outside of Cloudformation, that's fine too.
PS: I am new to AWS and it's capabilities
Thanks
Have a look at Former2.
Former2 allows you to generate Infrastructure-as-Code outputs from your existing resources within your AWS account.
You need an IaC (Infrastructure as Code) tool. Cloudformation is one of them, but there are plenty others. Terraform, Pulumi or even the AWS CDK.
Look at Infrastructure as Code try AWS CDK, Terraform, you should also look at methods for replacing existing infrastructure. Spinning a fresh set of infrastructure along side the existing one and swapping out in DNS is the most common of approach.
I have AWS Lambda service running which I had developed and deployed using the AWS SAM CLI. There are some other services as like ElasticCache, RDS, etc running in my AWS Account.
Now my requirement is to generate the CloudFormation YAML template file out of existing running service which I can use to provision later and keep as a backup. For this, I used CloudFormer template however problem is that CloudFormer does not consider or generate YAML template for AWS Lambda services which I had noticed. And also I believe CloudFormer has become obsolete as well.
Please assist me here to let me know if there any latest service from AWS that can fulfil my requirement here.
Thank you
CloudFormation supports the creation of stacks from existing resources.
Take a further look: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-new-stack.html
Unfortunately, there is no such functionality, apart from CloudFormer which is not maintained anymore by AWS nor reliable (in beta for years).
To use the importing resources feature into CFN, you have to manually write its template. This means you prepare your templates for RDS or EC, which match exactly your existing resources. If you have written the yaml or json template for the imported resources, you then can attempt to import them. In other words, the feature does not generate yaml code for you.
Maybe there are some non-AWS third party tools for that, but I'm not aware of any worth recommending unfortunately.
There is an opensource replacement for Cloudformer named former2 at https://github.com/iann0036/former2. I believe it can work as both a chrome/firefox addon and a cli tool.
I recently started working with AWS and IaC, I'm using Cloudformation to provision my AWS resources, but I discovered that AWS provide both a SDK and a CDK to enable you to provision resources programmatically instead of plain json/yaml.
But based on the documentation I did not really understand how they differ, can someone explain me how they differ and for what use case you should use what?
CDK: Is a framework to model and provision your infrastructure or stack. Stack can consist of a database for ex: DynamoDB, S3 Bucket, Lambda, API Gateway etc. It provides a facility to write code to create an infrastructure in AWS. Also called Infrastructure as code.
Check here
SDK: These are the code libraries provided by Amazon in various languages, like Java, Python, PHP, Javascript, Typescript etc. These libraries help interact with AWS services (like creating data in DynamoDB) which you either create through CDK or console. SDKs simplify using AWS services in your application with an API.
Check here
AWS SDK is a library primarily to ease the access to the AWS services by handling for you the data (de)serialization, credentials management, failure handling, etc. Perhaps, for specific scenarios, you could use the AWS SDK as the infrastructure as a code tool, however it could be cumbersome as it is not the intended usage of the library.
Based on the https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html, dedicated tools for the IaC are AWS CloudFormation and AWS CDK.
AWS CDK is an abstraction on top of CloudFormation. CDK scripts are in fact transformed to the CloudFormation definitions when scripts are synthesized.
The difference can be best described on an example: Imagine that for each lambda function in your stack you want to create an error CloudWatch alarm and connect to the SNS topic.
With CloudFormation you will either a) need to write a pretty much similar bunch of yaml/json definitions for each lambda function to ensure the monitoring, b) use the nested stack templates, c) use CloudFormation modules.
With CDK you can write a generic code construct - class or method, which can create the alarm for the given lambda function and create the SNS alarm action for given topic.
In other words, CDK helps you generalize and re-use your IaC in a very familiar way to how you develop your business code. The code is shorter and more readable than the CF definitions.
The difference is even more remarkable when you need to set up similar resources in different AWS regions and when you have different AWS account per environment. You can manage all AWS accounts and regions with a single CDK codebase.
Some background first: CloudFormation is Amazon's solution for an “Infrastructure as Code” approach to managing the definition, provisioning and deployment of a bunch of resources across accounts/regions. This is done by using their declarative yaml/json-based template language to define it all, and then executing the templates through various means (console, cli, APIs...). More info:
white paper: https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html
faq: https://aws.amazon.com/cloudformation/faqs/
There are other popular IaC solutions or tools to help achieve it more easily out there, such as Terraform and Kubernetes (container orchestration that also uses declarative templates to define desired states).
Potential benefits of IaC: At a high level, you can better track & audit your infra, reuse definitions/processes, make all your changes in a more consistent manner, faster thanks to all the automation and assurances you can get with an infra-as-code approach. You may be familiar with these as mentioned in previous answers and more, such as:
version controlling your infrastructure definitions,
more efficient and logically complex ways of constructing templates,
ability to write tests against them,
do diffs (see "change sets") before making real infra changes with the templates,
detect when live infra differs from your definitions,
automate rollbacks,
and lots of other state management assistance through a framework like CF that might be needed when performing regular ops duties.
CDK:
This is for helping to automate CloudFormation as part of an IaC approach to provisioning and deploying resources. It lets you use various popular programming languages to help with the creation, testing, and management of your CF setup. Some of AWS’s motivations: “YAML is an excellent format for describing the desired state of your cluster, but it is does not have primitives for expressing logic and reusable abstractions.“ “AWS CDK uses the familiarity and expressive power of programming languages for modeling your applications.”
More info: https://docs.aws.amazon.com/cdk/v2/guide/home.html
However, Amazon knows about other solutions, and happily points them out on the main CDK page now, downplaying its original connection to CF. You don't need to use CloudFormation if you don't want to; specifically, they mention you can use the same CDK constructs with the help of:
cdktf for Terraform maintained by its creators, Hashicorp
cdk8s for Kubernetes by AWS. re: “We realized this was exactly the same problem our customers had faced when defining their applications through CloudFormation templates, a problem solved by the AWS Cloud Development Kit (AWS CDK), and that we could apply the same design concepts from the AWS CDK to help all Kubernetes users.”
SDK:
AWS has an API for all of their services, and the various SDKs give you access to them. For example, I can use AWS’s Java SDK to manage an API Gateway. If I wanted to script some custom deployment process, I could do so with the SDK, managing all the state, etc. myself. You could probably even re-implement the CloudFormation service with the various underlying APIs... The APIs have varying levels of documentation though. E.g. CloudFormation Java APIs are only mentioned in the raw API reference, not the friendlier Developer Guide.
I find that the difference for me is that the CDK codifies the CloudFormation JSON/YAML. First response, is great ya okay in code but the benefit on the code side of things is you can write unit testing against the code. Therefore you get to build that sense of security or insurance policy against the provisioned services in the CDK.
There are other ways to test CF, however, with a dev background, this feels more comfortable.
I´m using AWS CLI and CloudFormation, and I could not find any reference in the documentation.
Does anybody know if it´s possible to create a CloudFormation template from a current configuration.
Let´s say that I want to get a CloudFormation template from my current security group configuration.
Any idea if it´s possible to export that configuration as a template using CLI?
Based on our experience we found 3 possible ways to translate existing manually deployed (from Web Console UI) AWS infra to Cloudformation (CF).
Using a new CloudFormation native introduced feature (since Nov 2019) that allows you to Import existing resources into a CloudFormation stack
Using aws cli execute $aws service_name_here describe for each element that make up your stack eg for RDS Database Stack:
RDS Instance -> Type: AWS::RDS::DBInstance,
RDS (EC2) SG -> Type: AWS::EC2::SecurityGroup,
RDS Subnet Group -> Type: AWS::RDS::DBSubnetGroup and
RDS DB Param Group -> Type: AWS::RDS::DBParameterGroup
And manually translate to CF based on the outputs obtained from the aws cli for each of the components. This approach usually requires more experience in both AWS and CF but the templates that you are creating can be structured and designed under good practices, fully parameterized (Sub, Ref, Join, Fn::GetAtt:, Fn::ImportValue), modular, applying conditions and in a 1st iteration the result would probably be close to the final state of the templates (interesting reference examples: https://github.com/widdix/aws-cf-templates/).
Extra points! :)
Some other new alternatives to export your current deployed AWS infra to Cloudformation / Terraform code:
https://former2.com
https://modules.tf
https://www.brainboard.co/
Related Article: https://medium.com/#exequiel.barrirero/aws-export-configuration-as-code-cloudformation-terraform-b1bca8949bca
It's not possible using the AWS CLI but you can use the CloudFormer [1] tool to create a CloudFormation template from existing resources. I've had decent success with it. The templates aren't as "pretty" as hand-made templates but they provide a good starting point.
[1] http://aws.amazon.com/developertools/6460180344805680
In addition to CloudFormer, you might want to take a look at Bellerophon: https://github.com/arminhammer/bellerophon.
I had some problems getting the tradidtional tools - mentioned above - working in our environment; we have a complicated API Gateway. Former2 didnt' find it at all (although seemed ideal for other resources)
I found another tool, "Terraformer" which extracts AWS into Terraform, which can then be turned into CloudFormation -or used directly as IaC.
https://github.com/GoogleCloudPlatform/terraformer#installation
Maybe that will work for others if the above tools don't.