Experts,
I am using AWS Api gateways as a proxy over AWS Lambda for my client calls. API Gateway has a timeout of 29000 milliseconds which is becoming an issue for one of our Lambda call requires around 2 minutes to complete.
Memory is not an issue - the operation is a bit time consuming.
One of the dirty but quickest ways to solve this issue to call the lambda directly from the client app skipping the AWS API gateway however I have to then hardcode the access key and secret in the client app.
As we are struggling with this issue on production the quickest and dirty solution seems to be the way forward for now and then later on replaced with WebSockets.
Still, to avoid major security issue I was thinking about whether can we restrict the access key and secret to only 1 lambda function
Related
I am contemplating if I should invoke lambda directly from another lambda or should I expose api through api-gateway in front of lambda. I am looking for pros and cons for both.
Approach #1 Using API Gateway
API Gateway and Lambda have one of the best integrations for serverless applications. It is very widely used and offers a ton of features - proxy integration, mapping templates, custom domain names and different types of authentication.
However, with these pros comes the cons due to some limits with using API Gateway. API Gateway has a default integration time out (a hard limit) of 29 seconds - which means the Lambda function needs to send back a response to API within this time frame or API fails with a 504 response. You may review other limits related to API Gateway here.
Approach #2 Lambda invoking Lambda
I'm not a big fan of this approach and have multiple reasons for it. I'll start with the additional code you have to write - same task with better features can be done by API Gateway with simple configurations on the AWS console.
A container calling another container(Lambda) can result it container-related problems - networking, container reuse and even managing IAM permissions properly.
Also, a Lambda function can be invoked by only three options - SDK, CLI or an entity that has "Invoke" permission. So basically, you need to have some kind of resource in front of your first Lambda to invoke it which will then invoke the second. In my opinion, API Gateway is the best front-end you can have for Lambda which is exactly AWS had in mind building these two services.
One of the pros I can think of this approach is the time out value - Lambda can run for up to max of 15 mins. Unless your client does not require a response back pretty quickly, you can run these two Lambda functions for a longer time to execute code.
Summary
All the above information was pretty general for anyone looking to use API Gateway and Lambda. I'll say it again that using API Gateway is a more convenient and useful approach however it may depend on your use-case. Hope this helps!
Here is my use case:
Static react frontend hosted on s3
Python backend on lambda conduction long running data analysis
Postgres database on rds
Backend and frontend communicate exclusively with JSON
Occasionally backend creates and stores powerpoint files in s3 bucket and then serves them up by sending s3 link to frontend
Convince me that it is worthwhile going through all the headaches of setting up API gateway to connect the frontend and backend rather than invoking lambda directly from the frontend!
Especially given the 29s timeout which is not long enough for my app meaning I need to implement asynchronous processing and add a whole other layer of aws architecture (messaging, queuing and polling with SNS and SQS) which increases cost, time and potential for problems. I understand there are some security concerns, is there no way to securely invoke a lambda function?
You are talking about invoking a lambda directly from JavaScript running on a client machine.
I believe the only way to do that would be embedding the AWS SDK for JavaScript in your react frontend. See:
https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/browser-invoke-lambda-function-example.html
There are several security concerns with this, only some of which can be mitigated.
First off, you will need to hardcode AWS credentials in to your frontend for the world to see. The access those credentials have can be limited in scope, but be very careful to get this right, or otherwise you'll be paying for someone's cryptomining operation.
Assuming you only want certain people to upload files to a storage service you are paying for, you will need some form of authentication and authorisation. API Gateway doesn't really do authentication, but it can do authorisation, albeit by connecting to other AWS services like Cognito or Lambda (custom authorizers). You'll have to build this into your backend Lambda yourself. Absolutely doable and probably not much more effort than using a custom authorizer from the API Gateway.
The main issue with connecting to Lambda direct is that Lambda has the ability to scale rapidly, which can be an issue if someone tries to hit you with a denial of service attack. Lambda is cheap, but running 1000 concurrent instances 24 hours a day is going to add up.
API Gateway allows you rate limit per second/minute/hour/etc., Lambda only allows you to limit the number of concurrent instances at any given time. So if you were to set that limit at 1, an attacker could cause that 1 instance to run for 24 hours a day.
I'm considering about moving my service from a VPS to AWS Lambda + DynamoDB to use it as a FaaS, because it's basically 2 API GET calls that fetch info from the database and serve it, and the normal use of those API calls are really rare (about 50 times a week)
But it makes me wonder... As I can't setup a limit on how many calls I want to serve each month, some attacker could theoretically flood my service by calling it a couple thousands times a day and make my AWS bill extremely expensive. Setting up a limit per month wouldn't be a nice idea either, because the attacker could flood the first day and I won't have more requests to serve. The ideal thing would be to set up a limit on request rate per client.
Anyone knows how could I protect it? I've seen that AWS also offers a Firewall, but that's for CloudFront. Isn't there any way to make it work with Lambda directly?
You can put AWS CloudFront in front API Gateway and Lambda so that, the traffic will be served to outside through CloudFront.
In addition by configuring AWS WAF with rate base blocking, it is possible to block high frequencies of access by attackers.
However when configuring AWS CloudFront in front of API Gateway and Lambda, you also need to restrict direct access to API Gateway (Since API Gateway will be publicly accessible by default). This can be achieved in following ways.
Enable API Keys for API Gateway and use the API Key in AWS CloudFront Headers in the Origin.
Use a Token Header and Verify it using a Custom Authorizer Lambda function.
Two options spring to mind:
place API Gateway in front of Lambda so that API requests
have to be authenticated. API Gateway also has built-in throttles and other useful features.
invoke the Lambda directly, which will require the client
invoking the Lambda to have the relevant IAM credentials.
I've recently gotten into AWS Serverless Architecture with .NET Core 1.0. In my application we use Elasticsearch on its own machine in order to maintain it. What I am trying to do is use AWS Elasticsearch Service from AWS API Gateway which is being proxied by AWS Lambda. (I believe I have typed this correctly)
When ever my code accesses my Elasticsearch domain I receive a timeout error. As of right now, my Elasticsearch domain is left wide open so anyone can access the information. I would like to lock it down for only the API Gateway and Lamda function.
I've tried messing with the policies and roles to no success. Has anyone tried to do what I am trying to do, if so, how were they able to connect it? Or is there a better way?
The simple solution is to put all of your services out of the VPC they are in right now (I believe they are not in the same one, as your IO calls get timed out).
My answer here would give you a nice background on AWS Lambda with VPC and why external IO calls time out.
AWS lambda invoke not calling another lambda function - Node.js
note: the answer is not related to NodeJS.
Is it possible to make some kind of HTTP request that will trigger Lambda and allow it to build a response for the request?
Is it possible for Lambda to access CloudFront cache directly or somehow get the data it needs. I guess it can be done making HTTP requests to CloudFront, but maybe there is more direct way to do that, no?
Or all this stuff I'm asking here is a peace of **** and I better go and buy a new server or optimize my code (actually, i would like to, but manager wants CloudFront + Lambda, so I'm trying to figure out if that is possible, but the docs don't give me an answer. Am I blind maybe?)
You can expose your lambda function via an API gateway. Then your lambda function can just run code that will access other services/resources (CloundFront, SNS, SQS, etc). Use the AWS SDK to access these services.
See Amazon API Gateway documentation: http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html