I want to implement a simple sequence of tasks on AWS Step Function. Something like next:
I can't fire and forget External API, because I need a response from it. So it is a bad idea to wrap it in a lambda function.
I can't implement the External API task on Lambda Function, because work exceeds lambda limitations.
The best way that I see is the implementation of a call to External API from the task of Step Function. If I understand correctly it is possible to do with Activities and Worker.
I see some Ruby example, but it isn't clear for me.
Could anybody suggest me a good tutorial with clear examples of similar implementation?
PS: External API I could wrap in anything on EC2.
Unfortunately you can't call an external API from the Step Function.
You have to wrap the call in a Lambda.
AWS documentation:
Step Functions supports the ability to call HTTP endpoints through API Gateway, but does not currently support the ability to call generic HTTP endpoints.
What Lambda limitation prevents you from wrapping the external API? If Lambda indeed cannot be used, depending on your usage, you may choose:
If the traffic is not stable/continuous, take a look at the ECS task which can be called by Step Functions (https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ecs.html), because it can save you the cost of paying idle.
Otherwise, using EC2/ECS and Activity is the way to go.
Related
The official guide for AWS API gateway introduces the way to use a lambda function to make responses to the API calls from the gateway. But it can only deal with one function, not for the condition of several functions call one by one.
For a solution, here are two to the best of my mind:
Use AWS Step Function services to bundle the function workflow.
Use one main thread function for orchestration.
Obviously, method 1 will bring extra fees, while method 2 needs a redundant function to run for long.
Could you please give me any help？
If they must be completed one after another before you can return the result, use AWS Step Functions (like you said) to orchestrate this. Synchronous invocation of the other Lambda is also an option (though it's commonly an architectural smell and indicates other issues with your architecture - why are the Lambdas not combined if they need each other etc.)
For asynchronous invocation of the other function (fire and forget), either invoke the Lambda using invocation type Event or even better would be to have an SQS queue in between the Lambdas for improved resiliency.
Seems a little inefficient the way it currently is:
response.body = {
user: await userService(userID) // calls a user service to get info on user
friends: await friendsService(userID) // calls a friends service to get info on friends for
}
Let's say the userService and friendsService are configured on different API Gateway endpoints.
Then wouldn't that make the network request take longer than if I were to just package my entire backend into one zip file that's uploaded to AWS Lambda.
Seems like this is very inefficient.
Is there a way to call other lambdas without having to make a network request? I understand putting the lambdas/gateway in the same VPC as the main Gateway endpoint exposed to the internet, but this is expensive?
Anyway to do this more efficiently?
You can call a Lambda function by using the AWS SDK (a LambdaClient object). So, for example assume you wrote two Lambda functions - funA and funB.
Next assume:
you want to call funB from funA
you wrote your Lambda function by using the Lambda Java runtime
You can use the Lambda Java API to invoke funB. There is no need to wrap either one in Restful call using API Gateway. You can use the AWS SDK. Here is the Java API example that shows you how to invoke a Lambda function:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/lambda/src/main/java/com/example/lambda/LambdaInvoke.java
#smac2020 writes in the answer about using SDK, that is of course a network call too. It just skips api gateway and calls directly AWS api.
I think the key point about Lambda is if it scales well. Let you try to thing about the algorithms in a different way. For example you can create a pipeline where in each step your "state object" is enriched with additional data. You can use step function or SQS and send the requests in between the steps or you can make the client responsible to manage the data. You should try to avoid one function waiting for another function. You are then paying two lambdas running - the caller and the called one.
If you are thinking "but microservices..." - look at the design of the AWS API itself. You do not need output of one service as an input to another one. It needs some time to adapt and to look at problems from different perspectives. In your case I would consider if the user list can for example live in the user object and the calls can be merged (look at some no-sql database design principles).
In traditional application development using Spring-boot / nodeJS, we have a controller/router in which we create different methods to handle appropriate HTTP request
Reservation Controller / Router
GET getReservation(id)
POST createReservation()
PUT updateReservation()
GET getAllReservation()
Controller/router calls the service classes to get the job done. Assume that you have multiple controller/service classes like this.
Now my quesiton is, If I need to create similar application using AWS lambda, I have to create multiple lambda functions separately which do not seem to be organized under a controller. (I understand that API Gateway is the controller here - please correct me if it is not). How to organize lambda functions / what is best practise you follow for your serverless architecture?
There is no strict architecture to develop what you want. It depends on your need for isolation and maintenance. You can do it either way with Lambda.
If your code is small enough for all methods. You can perform ANY integration with API Gateway, that will get all the methods under control of single Lambda.
If you want to separate the code to own lambda's, you can create independent lambda and deploy them separately. If you have dependent libraries across all of your methods, you can share them with Lambda Layers.
Both of the above approaches discussed here
Hope it helps.
Context:
I have a usecase where my backend service should compute 1 or more features, where each feature is a simple peace of computation (can be as simple as adding two numbers) and each feature takes input and return an output value, which can be boolean or a number. Client can actually request features (1 or 10 etc), also each feature can have multiple versions.
Design:
Lambda function seems like a good choice, since it supports versioning and takes care of scaling. In my design, one Lambda will receive the request and then call further lambda functions in parallel (Say user asked for 12 features, Lambda function L1 will invoke 12 Lambda functions in parallel) synchronously, and return all computed feature values as one response (HTTP). This way, all features can be versioned in their own Lambda functions.
Questions:
Is it ok to call a lambda function directly from another Lambda function? Is it a good usecase for using Lambda functions?
Thanks
I think Lambda would work just fine for your use case. For versioning, you could use the API versioning provided by API Gateway but I think that is a bit much for your case. Just create different functions.
Check out serverless.com. It is a solid framework and easy to get started with. It will take a lot of the work out of setting it up, plus you'll have your infrastructure as code.
Yes, it is okay to call lambdas from other Lambdas. There is not a 'clean' way to do that though. On the other hand, 'Step functions' may be what you need. Lambda support's chaining functions in a workflow. The previous lambda is not 'calling' the next function as much as proceeding to the next step in the workflow. The Serverless framework also supports using the method and can be configured in the serverless.yml config file
I don't find a way to use a ktor application inside an AWS lambda...
That is, instead of starting an embedded server or using an external server as described in http://ktor.io/servers/engine.html, I just need to "execute" the pipeline.
I suppose this is more or less like the TestEngine but I am not so familiar with the ktor framework to be sure
Note :
I have already found examples to run one kotlin function per lambda (the best tutorial IMHO being https://aws.amazon.com/fr/blogs/machine-learning/use-amazon-rekognition-to-build-an-end-to-end-serverless-photo-recognition-system/).
The problem is I dont want to manage one lambda per function (I want one microservice per lambda, the microservice being responsible for multiple tightly coupled operations)
After digging a lot more into AWS lambda and the serverless world in general, I've found that using ktor is not what lambda (or more generally function as a service) is useful for.
That is, I wanted to use ktor to group multiple functions in a logical service and to do the "routing" inside this group.
To achieve that in the FaaS world, you must declare one HTTP endpoint for each function.
As this is very tedious to maintain manually, you can use the serverless framework with a proper serverlesss.yml file.
I had this revealation when reading https://github.com/ajurasz/ascii-less-gallery which is a perfect follow of the article I mention in my intial question