I am working in AWS with the API gateway together with a lambda function. I read about how to pass parameters over to lambda function, that is fine. But I want to pass the whole path over to lambda. Does someone know how that would be done? Especially I want to pass the stage of the API gateway. The lambda function should connect to either the test server or the prod based on the stage. In the following example it would be test:
https://skjdfsdj.execute-api.us-east-1.amazonaws.com/test/name/name2
In next example it would be prod:
https://skjdfsdj.execute-api.us-east-1.amazonaws.com/prod/name/name2
Any information how that would work?
Thanks,
Benni
We can configure/deploy the API Gateway with respect to the stages and the HTTP Methods that are required Docs.
There may be two cases :
You may either have two different AWS lambda functions implemented, in this scenario its pretty simple as you can just create another stage and map the lambda function and the respective methods accordingly.
If you have to access the same lambda function and take action corresponding to the stage, You can add, remove, and edit stage variables and their values. You can use stage variables in your API configuration to parametrize the integration of a request. Stage variables are also available in the $context object of the mapping templates, and once we have mapped the particular stage variable in the incoming request you can use it and configure which server to call accordingly. Do check this out API Gateway context/stage variables
Related
I have an architectural question about the design and organisation of AWS Serverless resources using CloudFormation.
Currently I have multiple stack organised by the domain specific purpose and this works well. Most of the stack that contain Lambdas have to transformed using Serverless (using SAM for all). The async communication is facilitated using a combination of EventBridge and S3+Events and works well. The issue I have is with synchronous communication.
I don't want to reference Lambdas from other stacks using their exported names from other stacks and invoke them directly as this causes issues with updating and versions (if output exports are referenced in other stacks, I cannot change the resource unless the reference is removed first, not ideal for CI/CD and keeping the concerns separate).
I have been using API Gateway as an abstraction but that feels rather heavy handed. It is nice to have that separation but having to have domain and DNS resolving + having the API GW exposed externally doesn't feel right. Maybe there is a better way to configure API GW to be internal only. If you had success with this, could you please point me in the direction?
Is there a better way to abstract invocation of Lambda functions from different stacks in a synchronous way? (Common template patterns for CF or something along those lines?)
I see two questions:
Alternatives for Synchronous Lambda Functions with API Gateway .
Api Gateway is one easy way, with IAM Authentication to make it secure. HTTP Api is much simplified and cheaper option compared to REST APIs. We can choose Private Api rather than a Regional/Edge, which is not exposed outside VPC to make it even move secure.
we can have a private ALB with target as Lambda functions, for a simple use case that doesn't need any API gateway features.(this will cost some amount every month)
We can always call lambdas directly with AWS SDK invoke.
Alternatives to share resources between templates.
Exporting and Importing will be bit of problem if we need to delete and recreate the resource, shouldn't be a problem if we are just updating it though.
We can always store the Arn of the Lambda function in an SSM parameter in source template and resolve the value of the Arn from SSM parameter in destination template. This is completely decoupled. This is better than simply hard coding the value of Arn.
I'm writing to you because I'm quite a novice with AWS... I only worked before with EC2 instances for simple tasks...
I am currently looking for an AWS service for reciving data using REST API calls (to external AWS services).
So far I have used EC2 where I deployed my library (python) that made calls and stored data in S3.
What more efficient ways does AWS offer for this? some SaaS?
I know that they are still more details to know in order to choose a good services but I would like to know from where I can start looking.
Many thanks in advance :)
I make API requests using AWS Lambda. Specifically, I leave code that makes requests, writes the response to a file and pushes the response object (file) to AWS S3.
You'll need a relative/absolute path to push the files to wherever you want to ingest. By default lambda servers current working directory is: /var/task but you may want to write your files to /tmp/ instead.
You can automate the ingestion process by setting a CloudWatch rule to trigger your function. Sometimes I chain lambda functions when I need to loop requests with changing parameters instead of packing all requests within a single function,
i.e.
I leave the base request (parameterized) in one function and expose the function through an API Gateway endpoint.
I create a second function to call the base function once for each value I need by using the Event object (which is the JSON body of a regular request). This data will replace parameters within the base function.
I automate the second function.
Tip:
Lambda sometimes will run your requests inside the same server. So if you're continuously running these for testing the server may have files from past calls that you don't want, so I usually have a clean-up step at the beginning of my functions that iterates through my filesystem to make sure there are no files before making the requests.
Using python 3.8 as a runtime I use the requests module to send the request, I write the file and use boto3 to push the response object to an aws S3 bucket.
To invoke an external service you need some "compute resources" to run your client. Under compute resources in aws we understand ec2, ecs (docker container) or lambda (serverless - my favorite)
You had your code already running on EC2 so you should already know you need VPC with a public subnet and ip address to make an outbound call regardless the compute resource you choose
Given a REST API, outside of my AWS environment, which can be queried for json data:
https://someExternalApi.com/?date=20190814
How can I setup a serverless job in AWS to hit the external endpoint on a periodic basis and store the results in S3?
I know that I can instantiate an EC2 instance and just setup a cron. But I am looking for a serverless solution, which seems to be more idiomatic.
Thank you in advance for your consideration and response.
Yes, you absolutely can do this, and probably in several different ways!
The pieces I would use would be:
CloudWatch Event using a cron-like schedule, which then triggers...
A lambda function (with the right IAM permissions) that calls the API using eg python requests or equivalent http library and then uses the AWS SDK to write the results to an S3 bucket of your choice:
An S3 bucket ready to receive!
This should be all you need to achieve what you want.
I'm going to skip the implementation details, as it is largely outside the scope of your question. As such, I'm going to assume your function already is written and targets nodeJS.
AWS can do this on its own, but to make it simpler, I'd recommend using Serverless. We're going to assume you're using this.
Assuming you're entirely new to serverless, the first thing you'll need to do is to create a handler:
serverless create --template "aws-nodejs" --path my-service
This creates a service based on the aws-nodejs template on the provided path. In there, you will find serverless.yml (the configuration for your function) and handler.js (the code itself).
Assuming your function is exported as crawlSomeExternalApi on the handler export (module.exports.crawlSomeExternalApi = () => {...}), the functions entry on your serverless file would look like this if you wanted to invoke it every 3 hours:
functions:
crawl:
handler: handler.crawlSomeExternalApi
events:
- schedule: rate(3 hours)
That's it! All you need now is to deploy it through serverless deploy -v
Below the hood, what this does is create a CloudWatch schedule entry on your function. An example of it can be found over on the documentation
First thing you need is a Lambda function. Implement your logic, of hitting the API and writing data to S3 or whatever, inside the Lambda function. Next thing, you need a schedule to periodically trigger your lambda function. Schedule expression can be used to trigger an event periodically either using a cron expression or a rate expression. The lambda function you created earlier should be configured as the target for this CloudWatch rule.
The resulting flow will be, CloudWatch invokes the lambda function whenever there's a trigger (depending on your CloudWatch rule). Lambda then performs your logic.
When running functions in AWS lambda, it's common to use environment variables to control settings. However, when invoking Lambda via API gateway you have 'stage variables' to contend with.
My question is this: is an AWS Lambda instance scoped to a particular API gateway stage when invoked from API gateway, such that I can rely on the stage not changing between calls. In effect, does each API 'stage' get it's own pool of instances to work with, which are recycled in accordance with stage variables?
Examples of where I might want to depend on this behaviour:
Creating connections to tables - the table name will be different per-stage, so if I create the connection on first usage I'd end up using the first callers stage context. What happens when I make a call on a different API gateway stage?
Varying JWT keys for environments.
My gut feeling on this is that if API gateway has two versions/stages of the deployment referencing the exact same function verison, the lambda-managed function instances can recieve calls from two stages interchangably, and I shoulnd't cache the context and request derrived information (stage-variables) variables in process.
There's a lot of AWS API Gateway / Lambda stuff out there, but couldn't find a clear answer to this issue.
You're right, a single Lambda function version will have a pool of instances that are totally independent. Different API Gateway stages and even different APIs can call the same function and this has no impact on the instance pool in Lambda.
So any in-function caching you're doing should not use the assumption that only a specific API and/or stage will access the cached data.
I created a resource in Amazon's ApiGateway. It is pointing to a Lambda function. This is being hit by a native mobile application (android and ios) which is already in the wild.
I now want to modify the Lambda function, but I see no way to change my ApiGateway resource to point to an alias of the lambda. This is my first time playing with any of these technologies and I see no easy mechanism to manage this in the aws console.
How can I modify my ApiGateway resource to point to my lambda alias so I can edit trunk without affecting existing clients?
Under Integration Type -> Lambda Function you need to add a reference to the stage variable MyLambdaFuntionName:${stageVariables.lambdaAlias} and then for each stage set the lambdaAlias in the Stage Variables tab accordingly(lambdaAlias=dev, lambdaAlias=prod, etc.)
There is an example with screenshots here: https://aws.amazon.com/blogs/compute/using-api-gateway-stage-variables-to-manage-lambda-functions/
Its kind of hidden towards the very bottom of the page starting with "Alternatively, you can mix and match static names"
For the later googler, be careful to add permissions WITH the correct alias like yourfunc:prod not only yourfunc. That means if you'r planning to use 3 alias to invoke the lambda functions, you have to add 3 of them.
On the Api Gateway console, use ARN instead of a lambda function name.
For my case, I would add directly the arn of the lambda function to replace the Lambda name. This works for me without have to to add the {} notation.