In SNS documentation I can see some information about delivery policies and receive rate in particular:
http://docs.aws.amazon.com/sns/latest/dg/DeliveryPolicies.html#delivery-policy-maximum-receive-rate
But is this configuration applicable only for HTTP endpoints or Lambda functions as well?
Is it somehow possible to control lambda concurrent execution with SNS?
As #mark-b pointed in the comments, this feature just got announced during AWS ReInvent 2017
Set Concurrency Limits on Individual AWS Lambda Functions:
https://aws.amazon.com/about-aws/whats-new/2017/11/set-concurrency-limits-on-individual-aws-lambda-functions/
To set the limit, navigate to your Lambda function in the Console
Scroll down, and at the very end you can set the limit.
As noted in comments, this does not appear to be implemented, currently, but there does appear to be a reasonably straightforward workaround:
SNS → API Gateway (an HTTPS endpoint) → Lambda
Arguably it's an unnecessary additional complexity and expense, but if you want a native solution with no external wizardry required, this might be the way to go.
You'd also have to handle validation of the SNS message signature, as well as the subscription confirmation, which a direct Lambda integration doesn't require... but this seems like a way to accomplish what you're looking for. Note that your optimal configuration here would be a regional endpoint for API Gateway, rather than an edge-optimized endpoint.
Related
I am contemplating if I should invoke lambda directly from another lambda or should I expose api through api-gateway in front of lambda. I am looking for pros and cons for both.
Approach #1 Using API Gateway
API Gateway and Lambda have one of the best integrations for serverless applications. It is very widely used and offers a ton of features - proxy integration, mapping templates, custom domain names and different types of authentication.
However, with these pros comes the cons due to some limits with using API Gateway. API Gateway has a default integration time out (a hard limit) of 29 seconds - which means the Lambda function needs to send back a response to API within this time frame or API fails with a 504 response. You may review other limits related to API Gateway here.
Approach #2 Lambda invoking Lambda
I'm not a big fan of this approach and have multiple reasons for it. I'll start with the additional code you have to write - same task with better features can be done by API Gateway with simple configurations on the AWS console.
A container calling another container(Lambda) can result it container-related problems - networking, container reuse and even managing IAM permissions properly.
Also, a Lambda function can be invoked by only three options - SDK, CLI or an entity that has "Invoke" permission. So basically, you need to have some kind of resource in front of your first Lambda to invoke it which will then invoke the second. In my opinion, API Gateway is the best front-end you can have for Lambda which is exactly AWS had in mind building these two services.
One of the pros I can think of this approach is the time out value - Lambda can run for up to max of 15 mins. Unless your client does not require a response back pretty quickly, you can run these two Lambda functions for a longer time to execute code.
Summary
All the above information was pretty general for anyone looking to use API Gateway and Lambda. I'll say it again that using API Gateway is a more convenient and useful approach however it may depend on your use-case. Hope this helps!
I'm trying to trigger a Lambda function when I click on deploy in the API-Gateway console to deploy API on a stage.
I already tried with cloudwatch rule, but there is no event patterns for API-Gateway deployment.
My questions are:
Is it possible to trigger a lambda function when I click on the deploy button on API-Gateway console?
If yes, how can I do that?
Thank you
Unfortunately, there is no straight forward way for achieving this.
CloudWatch rule will not help as there is no logging generated on API deployment.
The only thing left behind a deploy action is a CloudTrail event.
The best solution I could think for this involves Amazon EventBridge which is an event bus managed service provided by AWS.
In EventBridge you can create rules that collect specific events from various AWS services within (and beyond) your AWS account.
API Gateway is not one of these services, but CloudTrail is! (For reference here is a list of the EventBridge supported services)
An API deployment in API Gateway emits an event to CloudTrail which has CreateDeployment as event name and apigateway.amazonaws.com as event source. The event payload also includes data such as the restApiId, the stage, the IAM identity details of the deploying agent and more.
Note, that there is not much documentation around CloudTrail event schemas, but the event would look something like the one listed here
Now, we need to create an EventBridge rule that captures such CloudTrail events.
This is a very good, step by step, guide on how to do this.
For your use case, you need to choose API Gateway as the service name and add CreateDeployment as a Specific Operation as shown in the screenshot below:
Once the EventBridge rule is set up then you can directly attach it as a trigger in any Lambda function. See relevant tutorial.
Downsides
The above solution cannot be applied on the individual API level. The EventBridge rule will capture the deployments of all APIs of any stage in a specific region. Additional filtering has to be implemented within the lambda logic.
This will lead to unnecessary lambda executions if the solution is scoped for anything less than all the APIs of a region. However as we're talking about API deployments, the extra lambda execution cost will be negligible.
There is instructions on how to integrate between API Gateway and SNS here but this is a bit of a toy example.
I want to know how I can subscribe to a topic via API Gateway -> SNS Integration.
And to this purpose what I'm looking is general documentation on doing this as I assume it is possible. If you can ListTopics (which the example indicates) surely you can do other things...
Edit: So I now know that when I do an integration into SNS it sends the following to SNS: https://sns.eu-west-1.amazonaws.com/?Action=CreateTopic
So that's a good start as that is how to call SNS to create a topic.
So now the question is how to I parameterise this?
I have also figured out that I can do from post to the SNS endpoint and thus include my parameters. However, the I get a signature not present error...
Is it possible to make some kind of HTTP request that will trigger Lambda and allow it to build a response for the request?
Is it possible for Lambda to access CloudFront cache directly or somehow get the data it needs. I guess it can be done making HTTP requests to CloudFront, but maybe there is more direct way to do that, no?
Or all this stuff I'm asking here is a peace of **** and I better go and buy a new server or optimize my code (actually, i would like to, but manager wants CloudFront + Lambda, so I'm trying to figure out if that is possible, but the docs don't give me an answer. Am I blind maybe?)
You can expose your lambda function via an API gateway. Then your lambda function can just run code that will access other services/resources (CloundFront, SNS, SQS, etc). Use the AWS SDK to access these services.
See Amazon API Gateway documentation: http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html
I have a lambda function i'd like to invoke from the client-side. I was going to use the API Gateway, but it occurred to me that the queuing SNS affords might be handy.
After researching, it appears the only way to publish to SNS thru the Javsacript SDK is auth thru google/facebook or AWS Cognito. I'd like users (more specifically, events) to be able to push w/o auth'ng, so that's not an option.
The last option is hard-coding an AWS key. This is pretty explicitly discouraged in the docs, but after looking into it, it looks like I can create security provisions for a specific key and limit it to publishing only to one topic.
In other words, it'd ostensibly mimic a REST API, wouldn't it?
The only drawback I can think of is malicious spamming of the SNS. I know AWS API allows for rate-throttling, but couldn't find something similar on SNS.
So, 2 related question:
Is there a way to prevent malicious spam to an SNS topic?
are there other drawbacks to using SNS instead of an AWS API for invoking lambdas?
What queueing are you wanting to get from an SNS topic? I think you may be confusing SNS with SQS.
I see no advantage to using SNS->Lambda in this instance versus API->Lambda. I do however see several drawbacks to using SNS in this instance as it adds an unnecessary complication, as well as opens up unnecessary security risks.
You literally get no advantage to using SNS here, while you get several advantages to using API Gateway such as rate limiting and API key support. Not to mention that API Gateway endpoints are much easier to access from the browser than SNS topics. This is API Gateway's intended use, why try to hack together some method using SNS and hard coded AWS keys?