I have joined a project where I have written AWS Lambda Functions in C#. I'm new to working with AWS and Lambda Functions. Does anyone know of a way of automating releases into AWS? Can scripting be applied to releasing without a pipeline? The only option I have found so far is to upload the code in a zip file. Is there a better method of deployment?
Deployment Methods are plentiful.
You can use Terraform, it works well enough using an SDK in your language of chocie.
In-house to AWS, you have CloudFormation Templates which can be applied through CodeDeploy as part of a CodePipeline (since you mention c# you are using a docker deployment of lambdas, you may need a codebuild step.)
The next level up from that is SAM Templates, which I believe will handle a docker deployment of a C# lambda much easier than CloudFormation by itself would.
Finally you have the CDK (Cloud Development Kit) which puts infrastructre as a language, and allows you to program your infrastructure in most languages (I Do not know if C# is one of them - Typescript, Python are the two more popular ones to do CDK in) and I believe that can handle Docker deployment of lambdas as well.
Related
I am not quite clear on the best practices related to using CDK to deploy private Github repos to AWS. I understand that a pipeline should be created by CDK and the pipeline should invoke CodeDeploy to deploy the assets, but beyond that the details are murky.
I also want to understand if for this use case it would make sense to have a separate CDK repo which is responsible for the infrastructure for the entire backend of my project, or if it would make more sense to have CDK code included in each individual component repo. As I will be utilizing a microservice/cell based approach for building out components, the overhead required in adding CDK configuration for each component might be substantial.
You can think of CDK as a compiler that takes a given language and 'compiles' it down to CloudFormation templates. Those templates are then uploaded to Cloudformation by the CDK framework, and run for you when you execute the deploy command.
So. To answer your question more directly - if you can figure out how to do it in cloudformation, you can do it. That may involve spinning up a code build and running a script that executes some api calls or prepares a package for an ec2 server that then uses that package in the next step or any number of things.
But remember that CDK synths its cloudformation template all at once, and is only creating the template. It does not run any scripts that may be part of your code builds, and it does not 'wait' for certain things to be complete - because it isn't doing anything like that. If you have a sequence of events that need to occur, you want to use CodePipeline to orchestrate those events for you - but you can set up your CodePipeline with CDK for certain!
As for Overhead, maybe at first. But trust me when I say it becomes very quick and easy to generate a CDK stack for a given microservice with experience, and its super handy. Being able to spin up an ad-hoc on demand testing environment is super useful. Being able to deploy individual stacks on demand and make quick changes with just a line of code is handy as all get out. Having a single source of code for both your prod and development environments that you make a change in one and on next deployment in each is automatically reflected is super handy.
CDK is a very powerful tool - but it is a very low level one. It creates the template that will create your resources. Thats it. If your resources need to do something after being created for something else to happen you have to make use of other services to orchestrate that (CodePipeline, StepFunctions, Cloudwatch Events, ect)
I have AWS Lambda service running which I had developed and deployed using the AWS SAM CLI. There are some other services as like ElasticCache, RDS, etc running in my AWS Account.
Now my requirement is to generate the CloudFormation YAML template file out of existing running service which I can use to provision later and keep as a backup. For this, I used CloudFormer template however problem is that CloudFormer does not consider or generate YAML template for AWS Lambda services which I had noticed. And also I believe CloudFormer has become obsolete as well.
Please assist me here to let me know if there any latest service from AWS that can fulfil my requirement here.
Thank you
CloudFormation supports the creation of stacks from existing resources.
Take a further look: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-new-stack.html
Unfortunately, there is no such functionality, apart from CloudFormer which is not maintained anymore by AWS nor reliable (in beta for years).
To use the importing resources feature into CFN, you have to manually write its template. This means you prepare your templates for RDS or EC, which match exactly your existing resources. If you have written the yaml or json template for the imported resources, you then can attempt to import them. In other words, the feature does not generate yaml code for you.
Maybe there are some non-AWS third party tools for that, but I'm not aware of any worth recommending unfortunately.
There is an opensource replacement for Cloudformer named former2 at https://github.com/iann0036/former2. I believe it can work as both a chrome/firefox addon and a cli tool.
Why would anyone use Serverless framework CLI to write the lambda functions or deploy them when we have AWS console GUI? Are there any specific benefits out of it?
Normally you don't develop and deploy a lambda function in isolation, instead it is one part of your cloud infrastructure. That can include other lambdas, S3 buckets, databases, API Gateways, IAM roles, environment variables and much more.
Serverless framework is allows you to write your infrastructure as code. For AWS services, it translates serverless.yaml config files into AWS cloudformation files, and from there deploys any new or updated services you define. You lambda function is just one part of that.
A major benefit of writing and deploying this way is that you can use your favourite editor locally, and can check your code into version control (i.e. git). This is not just for your lambda code, but also your infrastructure config i.e. serverless.yaml and associated files.
The Serverless Framework is more than just a replacement for the AWS Console (GUI). You can definitely set everything up via the AWS console for a Serverless application but how do you share that with your team? What if you wish to deploy that repeatedly into multiple applications? The Serverless Framework gives you a configuration file, usually called serverless.yml, where you define all the services within AWS (and other vendors, there is support for more than just AWS) and then you use the CLI to perform functions on this configuration file such as deploy, invoke and lot more.
Then there are the Serverless Framework plugins designed by the community around the project to make other tasks even easier such as unit testing, configuration of S3 buckets, CloudFront and domains to make frontend deployment easier and a lot, lot more.
Lastly, but most importantly, there is a professional product provided in addition to the open source framework that you can use to add on monitoring, deployment management, troubleshooting, optimisation, CI/CD and too many other benefits to list here.
Definitely, if you are doing a big project the Serverless framework has a lot of benefits, imagine you developing an MVC c# project with notepad. How do you feel about that?
The framework are done to make our life ( for developers ) very much easier.
I recently started working with AWS and IaC, I'm using Cloudformation to provision my AWS resources, but I discovered that AWS provide both a SDK and a CDK to enable you to provision resources programmatically instead of plain json/yaml.
But based on the documentation I did not really understand how they differ, can someone explain me how they differ and for what use case you should use what?
CDK: Is a framework to model and provision your infrastructure or stack. Stack can consist of a database for ex: DynamoDB, S3 Bucket, Lambda, API Gateway etc. It provides a facility to write code to create an infrastructure in AWS. Also called Infrastructure as code.
Check here
SDK: These are the code libraries provided by Amazon in various languages, like Java, Python, PHP, Javascript, Typescript etc. These libraries help interact with AWS services (like creating data in DynamoDB) which you either create through CDK or console. SDKs simplify using AWS services in your application with an API.
Check here
AWS SDK is a library primarily to ease the access to the AWS services by handling for you the data (de)serialization, credentials management, failure handling, etc. Perhaps, for specific scenarios, you could use the AWS SDK as the infrastructure as a code tool, however it could be cumbersome as it is not the intended usage of the library.
Based on the https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html, dedicated tools for the IaC are AWS CloudFormation and AWS CDK.
AWS CDK is an abstraction on top of CloudFormation. CDK scripts are in fact transformed to the CloudFormation definitions when scripts are synthesized.
The difference can be best described on an example: Imagine that for each lambda function in your stack you want to create an error CloudWatch alarm and connect to the SNS topic.
With CloudFormation you will either a) need to write a pretty much similar bunch of yaml/json definitions for each lambda function to ensure the monitoring, b) use the nested stack templates, c) use CloudFormation modules.
With CDK you can write a generic code construct - class or method, which can create the alarm for the given lambda function and create the SNS alarm action for given topic.
In other words, CDK helps you generalize and re-use your IaC in a very familiar way to how you develop your business code. The code is shorter and more readable than the CF definitions.
The difference is even more remarkable when you need to set up similar resources in different AWS regions and when you have different AWS account per environment. You can manage all AWS accounts and regions with a single CDK codebase.
Some background first: CloudFormation is Amazon's solution for an “Infrastructure as Code” approach to managing the definition, provisioning and deployment of a bunch of resources across accounts/regions. This is done by using their declarative yaml/json-based template language to define it all, and then executing the templates through various means (console, cli, APIs...). More info:
white paper: https://docs.aws.amazon.com/whitepapers/latest/develop-deploy-dotnet-apps-on-aws/infrastructure-as-code.html
faq: https://aws.amazon.com/cloudformation/faqs/
There are other popular IaC solutions or tools to help achieve it more easily out there, such as Terraform and Kubernetes (container orchestration that also uses declarative templates to define desired states).
Potential benefits of IaC: At a high level, you can better track & audit your infra, reuse definitions/processes, make all your changes in a more consistent manner, faster thanks to all the automation and assurances you can get with an infra-as-code approach. You may be familiar with these as mentioned in previous answers and more, such as:
version controlling your infrastructure definitions,
more efficient and logically complex ways of constructing templates,
ability to write tests against them,
do diffs (see "change sets") before making real infra changes with the templates,
detect when live infra differs from your definitions,
automate rollbacks,
and lots of other state management assistance through a framework like CF that might be needed when performing regular ops duties.
CDK:
This is for helping to automate CloudFormation as part of an IaC approach to provisioning and deploying resources. It lets you use various popular programming languages to help with the creation, testing, and management of your CF setup. Some of AWS’s motivations: “YAML is an excellent format for describing the desired state of your cluster, but it is does not have primitives for expressing logic and reusable abstractions.“ “AWS CDK uses the familiarity and expressive power of programming languages for modeling your applications.”
More info: https://docs.aws.amazon.com/cdk/v2/guide/home.html
However, Amazon knows about other solutions, and happily points them out on the main CDK page now, downplaying its original connection to CF. You don't need to use CloudFormation if you don't want to; specifically, they mention you can use the same CDK constructs with the help of:
cdktf for Terraform maintained by its creators, Hashicorp
cdk8s for Kubernetes by AWS. re: “We realized this was exactly the same problem our customers had faced when defining their applications through CloudFormation templates, a problem solved by the AWS Cloud Development Kit (AWS CDK), and that we could apply the same design concepts from the AWS CDK to help all Kubernetes users.”
SDK:
AWS has an API for all of their services, and the various SDKs give you access to them. For example, I can use AWS’s Java SDK to manage an API Gateway. If I wanted to script some custom deployment process, I could do so with the SDK, managing all the state, etc. myself. You could probably even re-implement the CloudFormation service with the various underlying APIs... The APIs have varying levels of documentation though. E.g. CloudFormation Java APIs are only mentioned in the raw API reference, not the friendlier Developer Guide.
I find that the difference for me is that the CDK codifies the CloudFormation JSON/YAML. First response, is great ya okay in code but the benefit on the code side of things is you can write unit testing against the code. Therefore you get to build that sense of security or insurance policy against the provisioned services in the CDK.
There are other ways to test CF, however, with a dev background, this feels more comfortable.
It appears that SAM is implemented in the Serverelss framework; however, I am unable to find any documentation on this based on the lack of documentation from server less framework compared to AWS. AWS goes in to depth with SAM (as I am studying for the test) but I use serverless framework for my deployments and I'm trying to figure out the difference between the configuration file between the two.
They are related but are not quite the same thing. SAM and Serverless Framework are both layers on top of CloudFormation (when deploying to AWS; Serverless works with other providers too). They abstract away some of the things that make defining serverless applications in CloudFormation hard, but code written for SAM won't work with Serverless Framework, and vice versa. SAM templates can also have CloudFormation templates embedded in them. In that regard they are an extension or a superset.
Another difference is that SAM provides some tools for running functions locally and debugging them.