After doing some brief research, I'm receiving conflicting answers regarding best practices for the AWS lambda service. I'm writing a few microservices for my company that will automate the steps for adding clients to our various services: creating api keys, uploading documents to a repo, sending an email, etc.
I have copied and pasted my code for 3 lambdas now (only changing around a few variable values), but, before I start doing this for all of them, I wanted to request if anyone had an easier method. I do know about the ProxyIntegration, where I could use the same lambda for similar requests and differentiate them by their resource paths; however, is there an easier way I could "map" the lambdas to shared code?
I was thinking about using an S3 Object to hold the code, then change the variables by environment variables (which could very well work), but does anyone have any other recommendations or obvious solutions I'm not realizing?
Thanks!
There is a very recent feature called Lambda Layers that specifically allows you to share code between AWS Lambda functions.
You would build the common code as a library and deploy it as a Layer. Then each individual Lambda function would include that Layer.
Related
I have a few things to get clear, specifically regarding modeling architecture for a serverless application using AWS CDK.
I’m currently working on a serverless application developed using AWS CDK in TypeScript. Also as a convention, we follow the below rules too.
A stack should only have one table (dynamo)
A stack should only have one REST API (api-gateway)
A stack should not depend on any other stack (no cross-references), unless its the Event-Stack (a stack dedicated to managing EventBridge operations)
The reason for that is so that each stack can be deployed independently without any interferences of other stacks. In a way, our stacks are equivalent to micro-services in a micro-service architecture.
At the moment all the REST APIs are public and now we have decided to make them private by attaching custom Lambda authorizers to each API Gateway resource. Now, in this custom Lambda authorizer, we have to do certain operations (apart from token validation) in order to allow the user's request to proceed further. Those operations are,
Get the user’s role from DB using the user ID in the token
Get the user’s subscription plan (paid, free, etc.) from DB using the user ID in the token.
Get the user’s current payment status (due, no due, fully paid, etc.) from DB using the user ID in the token.
Get scopes allowed for this user based on 1. 2. And 3.
Check whether the user can access this scope (the resource user currently requesting) based on 4.
This authorizer Lambda function needs to be used by all the other Stacks to make their APIs private. But the problem is roles, scopes, subscriptions, payments & user data are in different stacks in their dedicated DynamoDB tables. Because of the rules, I have explained before (especially rule number 3.) we cannot depend on the resources defined in other stacks. Hence we are unable to create the Authoriser we want.
Solutions we could think of and their problems:
Since EventBridge isn't bi-directional we cannot use it to fetch data from a different stack resource.
We can invoke a Lambda in a different stack using its ARN and get the required data from its' response but, AWS has discouraged this as a CDK Anti Pattern
We cannot use technology like gRPC because it requires a continuously running server, which is out of the scope of the server-less architecture.
There was also a proposal to re-design the CDK layout of our application. The main feature of this layout is going from non-crossed-references to adopting a fully-crossed-references pattern. (Inspired by layered architecture as described in this AWS best practice)
Based on that article, we came up with a layout like this.
Presentation Layer
Stack for deploying the consumer web app
Stack for deploying admin portal web app
Application Layer
Stack for REST API definitions using API Gateway
Stack for Lambda functions running business-specific operations (Ex: CRUDs)
Stack for Lambda functions runs on event triggers
Stack for Authorisation (Custom Lambda authorizer(s))
Stack for Authentication implementation (Cognito user pool and client)
Stack for Events (EvenBuses)
Stack for storage (S3)
Data Layer
Stack containing all the database definitions
There could be another stack for reporting, data engineering, etc.
As you can see, now stacks are going to have multiple dependencies with other stacks' resources (But no circular dependencies, as shown in the attached image). While this pattern unblocks us from writing an effective custom Lambda authorizer we are not sure whether this pattern won't be a problem in the long run, when the application's scope increases.
I highly appreciate the help any one of you could give us to resolve this problem. Thanks!
Multiple options:
Use Parameter Store rather than CloudFormation exports
Split stacks into a layered architecture like you described in your
answer and import things between Stacks using SSM parameter store like the other answer describes. This is the most obvious choice for breaking inter-stack dependencies. I use it all the time.
Use fixed resource names, easily referencable and importable
Stack A creates S3 bucket "myapp-users", Stack B imports S3 bucket by fixed name using Bucket.fromBucketName(this, 'Users', 'myapp-users'). Fixed resource names have their own downsides, so this should be used only for resources that are indeed shared between stacks. They prevent easy replacement of the resource, for example. Also, you need to enforce the correct Stack deployment order, CDK will not help you with that anymore since there are no cross-stack dependencies to enforce it.
Combine the app into a single stack
This sounds extreme
and counter intuitive, but I found that most real life teams don't
actually have a pressing need for multi-stack deployment. If your only concern is
separating code-owners of different parts of the application - you
can get away by splitting the stack into multiple Constructs,
composed into a single stack, where each team takes care of their
Construct and its children. Think of it as combining multiple Git repos into a Monorepo. A lot of projects are doing that.
A strategy I use to avoid hard cross-references involves storing shared resource values in AWS Systems Manager.
In the exporting stack, we can save the name of an S3 Bucket for instance:
ssm.StringParameter(
scope=self,
id="/example_stack/example_bucket_name",
string_value=self.example_bucket.bucket_name,
parameter_name="/example_stack/example_bucket_name",
)
and then in the importing stack, retrieve the name and create an IBucket by using a .from_ method.
example_bucket_name = ssm.StringParameter.value_for_string_parameter(
scope=self,
parameter_name="/example_stack/example_bucket_name",
)
example_bucket = s3.Bucket.from_bucket_name(
scope=self,
id="example_bucket_from_ssm",
bucket_name=example_bucket_name,
)
You'll have to figure out the right order to deploy your stacks but otherwise, I've found this to be a good strategy to avoid the issues encountered with stack dependencies.
I have been checking some resources on internet and all the examples of lambda in AWS are very basic but I am not sure how we will modularize an application with multiples dependencies, for example in java we usually have some structure like this
packages
repository
controllers
..
..
And we place the code related to each logic inside the package, but now in AWS seems that is more like scripting that will glue the pieces than OOP that I am used to, so my question is how we handle (if apply) this relationships, because I have seen code that all the logic is in one lambda and that not seems the best way to go, for example if we had some functionality that fist authenticate, authorize, transform, call an external api, get the response and then do a call to a final rest endpoint, how we can split this, for example will be the same lambda with packages(directories) inside and we call to each other? or we have multiples lambdas each one with one purpose? and this will generate cold start for each lambda?
I was thinking in using layers, but seems very new and not sure if this is production ready feature and seems that is more related to reuse code that is common across all the environment that the way to modularize our code
Generally when you're developing Lambda functions, the function should have a single purpose (which will keep the function relatively small).
If you have multiple actions, by having each Lambda as its own function it will improve the development and deployment experience. Having a single developer working on the function reduces the risk of breaking unrelated functionality, whilst also allowing them to deploy only the function that they've worked on.
To orchestrate between Lambdas for APIs people tend to use API Gateway (be that for your clients communicating to the Lambdas, or between the Lambdas themselves).
Regarding any shared dependencies/libraries Lambda Layers as you mentioned is the correct way to go. It will allow you to centralise the dependencies that your applications share rather than the need to package the Lambda with a version of the dependencies each time.
There's an article on Best Practices for Developing on AWS Lambda that should offer additional guidance.
In traditional application development using Spring-boot / nodeJS, we have a controller/router in which we create different methods to handle appropriate HTTP request
Reservation Controller / Router
GET getReservation(id)
POST createReservation()
PUT updateReservation()
GET getAllReservation()
Controller/router calls the service classes to get the job done. Assume that you have multiple controller/service classes like this.
Now my quesiton is, If I need to create similar application using AWS lambda, I have to create multiple lambda functions separately which do not seem to be organized under a controller. (I understand that API Gateway is the controller here - please correct me if it is not). How to organize lambda functions / what is best practise you follow for your serverless architecture?
There is no strict architecture to develop what you want. It depends on your need for isolation and maintenance. You can do it either way with Lambda.
If your code is small enough for all methods. You can perform ANY integration with API Gateway, that will get all the methods under control of single Lambda.
If you want to separate the code to own lambda's, you can create independent lambda and deploy them separately. If you have dependent libraries across all of your methods, you can share them with Lambda Layers.
Both of the above approaches discussed here
Hope it helps.
So I have a class and some constants defined that 10 or more lambda functions will need. Currently, I have packaged the common code into each lambda function. Unfortunately, if I change the common code I have to repackage all 10 lambda functions and upload the changes.
Ideas that I had considered:
Lambda return a class with defs and constants – not feasible, lambda returns JSON
Try to magically load the common code from S3 – (not sure how and do not
really like that there are multiple steps to update a lambda
function)
Packaged the common code into each lambda function – (current design)
What is the best method for referencing common python code for lambda?
The first thing I did was an automatic deployment script in python for my lambdas + API gateway (+ intern usage of S3 etc). You can deploy your lambda without changing your API gateway endpoint, and be doing so, all your lambdas can be updated in one click without modification of the interaction with outside of your AWS box.
Inside, you can manage the bounds with S3 or dynamoDB or anything else automatically. It's an investment at the beginning, but it's definitely worth it, even more in your case with many lambdas.
Your solution of a constant provider could be a short term good strategy, but you'll need to be sure that your old lambdas will work with your news constants, so either you are limited in your provider's evolutions, either you'll have to manage many versions of your provider. Lambdas are meant to be easily deployed and replaced.
I'm trying to deploy an API suite by using Api Gateway and implementing code in Java using lambda. Is it ok to have many ( related, of course ) lambdas in a single jar ( what I'm supposing to do ) or it is better to create a single jar for each lambda I want to deploy? ( this will became a mess very easily)
This is really a matter of taste but there are a few things you have to consider.
First of all there are limitations to how big a single Lambda upload can be (50MB at time of writing).
Second, there is also a limit to the total size of all all code that you upload (currently 1.5GB).
These limitations may not be a problem for your use case but are good to be aware of.
The next thing you have to consider is where you want your overhead.
Let's say you deploy a CRUD interface to a single Lambda and you pass an "action" parameter from API Gateway so that you know which operation you want to perform when you execute the Lambda function.
This adds a slight overhead to your execution as you have to route the action to the appropriate operation. This is likely a very fast routing but nevertheless, it adds CPU cycles to your function execution.
On the other hand, deploying the same jar over several Lambda function will quickly get you closer to the limits I mentioned earlier and it also adds administrative overhead in managing your Lambda functions as that number grows. They can of course be managed via CloudFormation or cli scripts but it will still add an administrative overhead.
I wouldn't say there is a right and a wrong way to do this. Look at what you are trying to do, think about what you would need to manage the deployment and take it from there. If you get it wrong you can always start over with another approach.
Personally I like the very small service Lambdas that do internal routing and handles more than just a single operation but they are still very small and focused on a specific type of task be it a CRUD for a database table or managing a selected few very closely related operations.
There's some nice advice on serverless.com
As polythene say's, the answer is "it depends". But they've listed the pros and cons for 4 ways of going about it:
Microservices Pattern
Services Pattern
Monolithic Pattern
Graph Pattern
https://serverless.com/blog/serverless-architecture-code-patterns/