I am making an HTTP API in AWS. Most of my logic is handled in lambda functions. I use terraform to describe and deploy my infrastructure. So far this is going well.
I have a single project that has lambdas for CRUD operations
GET /journal
POST /journal
GET /journal/{id}
PUT /journal/{id}
DELETE /journal/{id}
The code and infrastructure are all currently inside a monorepo.
Its now time to add endpoints for nested resources
for example...
/journals/{id}/sentence/{id}
and
/journals/{id}/sentence/{id}/correction
I really want the code and terraform for these sub-resources to be in a separate project because it's going to get very, very big.
I am really struggling to figure out how to make sub-resources for API gateway that exist in multiple separately deployed projects.
I want to avoid exporting the ARNs for the api-gateway resources and using them in other projects as I don't think it's a great idea to create dependencies between separate deployments.
I would really appreciate hearing any advice on how this could be managed as I am sure many people have faced this issue.
Is it possible to use route53 to route all API calls on a domain to lots of separate API gateways? If this is possible then this make life much easier. I am really struggling to find documentation or literature that explains anything beyond creating an API with single resource endpoints.
Edit: I had one other idea that maybe all my lambda projects could be completely ignorant of the api-gateway and just have their ARNs exported as outputs. Then I could have a completely separate project which defines the whole api-gateway and creates lambda integrations which simply use the function ARNs exported by the function in the other projects. This would prevent each lambda project from needing to reference the api_gateway_resource of the corresponding parent resource.
I feel like this may not be a good idea though.
I want to avoid exporting the ARNs for the api-gateway resources and
using them in other projects as I don't think it's a great idea to
create dependencies between separate deployments.
I am not sure about that statement. Independently of the way you go, you will have dependencies anyway between your project that creates the API GW and the lambda sub project.
So far I understood the question is in which direction you should create this dependency:
export the apigw and reuse it in the lambda project
export the lambda and reuse it in the apigw project
I think it makes sense for the lambda to be considered as independent piece of infrastructure in your terraform project and not create any kind of apigw related resource in it. First of all because in such case it already creates a strong implicit dependency and constraint on the usage of your lambda. And secondly you can think this way: what if your project grow more and more and you need to add more and more lambdas to your API. In such scenario you probably don't want to create each time new methods/resources and it's probably more convenient to use something like for_each in terraform and write your code one single time and automatically creates new integrations when you add new lambda.
Hence I would avoid the first choice and go for the second option which is way cleaner from an architectural stand point. Everything that deals with API GW stays in the same project. Your edits point it out to the right direction. You could have one repo for the "Infrastructure" (call it whatever you want) and one for "Lambda". You could output the lambda ARNs as a list (or a custom map with other key parameters) and use remote state from the apigw project to loop through your lambda and create the needed resource in one single place with for_each.
For example, use the remote state from apigw project:
data "terraform_remote_state" "lambda" {
backend = "s3"
config = {
bucket = "my-bucket"
key = "my/key/state/for/lambda"
region = "region"
}
}
And use the outputs like this:
resource "aws_api_gateway_integration" "lambda" {
for_each = data.terraform_remote_state.lambda.outputs.lambdas
...
uri = each.value
}
Is it possible to use route53 to route all API calls on a domain to
lots of separate API gateways? If this is possible then this make life
much easier. .
Sorry I am not sure to understand that point but I will still try to detail a bit the topic. The "link" API gateway provides you to call your api is just a DNS. When you create an API, behind the scenes AWS creates a CloudFront distribution for you in us-east-1 region. You don't have access to it through the console because it's managed by AWS. When you map an API to a domain name, you actually map the domain to the CloudFront distribution of your API. When you add methods or resources to your API (this is what you do with your lambda), you actually don't create new APIs each time.
I am developing a "micro-services" application using AWS API Gateway with either Lambda or ECS for compute. The issue now is communication between services are via API calls through the API gateway. This feels inefficient and less secure than it can be. Is there a way to make my microservices talk to each other in a more performant and secure manner? Like somehow talk directly within the private network?
One way I thought of is multiple levels of API gateway.
1 public API gateway
1 private API gateway per microservice. And each microservice can call another microservice "directly" inside the private network
But in this way, I need to "duplicate" my routes in 2 levels of API ... this does not seem ideal. I was thinking maybe use {proxy+}. So anything /payment/{proxy+} goes to payment API gateway and so on - theres still 2 levels of API gateway ... but this seem to be the best I can go?
Maybe there is a better way?
There are going to be many ways to build micro-services. I would start by familiarizing yourself with the whitepaper AWS published: Microservices on AWS, Whitepaper - PDF version.
In your question you stated: "The issue now is communication between services are via API calls through the API gateway. This feels inefficient and less secure than it can be. Is there a way to make my microservices talk to each other in a more performant and secure manner?"
Yes - In fact, the AWS Whitepaper, and API Gateway FAQ reference the API Gateway as a "front door" to your application. The intent of API Gateway is to be used for external services communicating to your AWS services.. not AWS services communicating with each other.
There are several ways AWS resources can communicate with each other to call micro-services. A few are outlined in the whitepaper, and this is another resource I have used: Better Together: Amazon ECS and AWS Lambda. The services you use will be based on the requirements you have.
By breaking monolithic applications into small microservices, the communication overhead increases because microservices have to talk to each other. In many implementations, REST over HTTP is used as a communication protocol. It is a light-weight protocol, but high volumes can cause issues. In some cases, it might make sense to think about consolidating services that send a lot of messages back and forth. If you find yourself in a situation where you consolidate more and more of your services just to reduce chattiness, you should review your problem domains and your domain model.
To my understanding, the root of your problem is routing of requests to micro-services. To maintain the "Characteristics of Microservices" you should choose a single solution to manage routing.
API Gateway
You mentioned using API Gateway as a routing solution. API Gateway can be used for routing... however, if you choose to use API Gateway for routing, you should define your routes explicitly in one level. Why?
Using {proxy+} increases attack surface because it requires routing to be properly handled in another micro-service.
One of the advantages of defining routes in API Gateway is that your API is self documenting. If you have multiple API gateways it will become colluded.
The downside of this is that it will take time, and you may have to change existing API's that have already been defined. But, you may already be making changes to existing code base to follow micro-services best practices.
Lambda or other compute resource
Despite the reasons listed above to use API Gateway for routing, if configured properly another resource can properly handle routing. You can have API Gateway proxy to a Lambda function that has all micro-service routes defined or another resource within your VPC with routes defined.
Result
What you do depends on your requirements and time. If you already have an API defined somewhere and simply want API Gateway to be used to throttle, monitor, secure, and log requests, then you will have API Gateway as a proxy. If you want to fully benefit from API Gateway, explicitly define each route within it. Both approaches can follow micro-service best practices, however, it is my opinion that defining each public API in API Gateway is the best way to align with micro-service architecture. The other answers also do a great job explaining the trade-offs with each approach.
I'm going to assume Lambdas for the solution but they could just as well be ECS instances or ELB's.
Current problem
One important concept to understand about lambdas before jumping into the solution is the decoupling of your application code and an event_source.
An event source is a different way to invoke your application code. You mentioned API Gateway, that is only one method of invoking your lambda (an HTTP REQUEST). Other interesting event sources relevant for your solution are:
Api Gateway (As noticed, not effective for inter service communication)
Direct invocation (via AWS Sdk, can be sync or async)
SNS (pub/sub, eventbus)
There are over 20+ different ways of invoking a lambda. documentation
Use case #1 Sync
So, if your HTTP_RESPONSE depends on one lambda calling another and on that 2nd lambdas result. A direct invoke might be a good enough solution to use, this way you can invoke the lambda in a synchronous way. It also means, that lambda should be subscribed to an API Gateway as an event source and have code to normalize the 2 different types of events. (This is why lambda documentation usually has event as one of the parameters)
Use case #2 Async
If your HTTP response doesn't depend on the other micro services (lambdas) execution. I would highly recommend SNS for this use case, as your original lambda publishes a single event and you can have more than 1 lambda subscribed to that event execute in parallel.
More complicated use cases
For more complicated use cases:
Batch processing, fan-out pattern example #1 example #2
Concurrent execution (one lambda calls next, calls next ...etc) AWS Step functions
There are multiple ways and approaches for doing this besides being bound to your current setup and infrastructure without excluding the flexibility to implement/modify the existing code base.
When trying to communicate between services behind the API Gateway is something that needs to be carefully implemented to avoid loops, exposing your data or even worst, blocking your self, see the "generic" image to get a better understanding:
While using HTTP for communicating between the services it is often common to see traffic going out the current infrastructure and then going back through the same API Gateway, something that could be avoided by just going directly the other service in place instead.
In the previous image for example, when service B needs to communicate with service A it is advisable to do it via the internal (ELB) endpoint instead of going out and going back through the API gateway.
Another approach is to use "only" HTTP in the API Gateway and use other protocols to communicate within your services, for example, gRPC. (not the best alternative in some cases since depends on your architecture and flexibility to modify/adapt existing code)
There are cases in where your infrastructure is more complex and you may not communicate on demand within your containers or the endpoints are just unreachable, in this cases, you could try to implement an event-driven architecture (SQS and AWS Lambda)
I like going asynchronous by using events/queues when possible, from my perspective "scales" better and must of the services become just consumers/workers besides no need to listen for incoming request (no HTTP needed), here is an article, explaining how to use rabbitmq for this purpose communicating microservices within docker
These are just some ideas that hope could help you to find your own "best" way since is something that varies too much and every scenario is unique.
I don't think your question is strictly related to AWS but more like a general way of communication between the services.
API Gateway is used as an edge service which is a service at your backend boundary and accessible by external parties. For communication behind the API Gateway, between your microservices, you don't necessary have to go through the API Gateway again.
There are 2 ways of communication which I'd mention for your case:
HTTP
Messaging
HTTP is the most simplistic way of communication as it's naturally easier to understand and there are tons of libraries which makes it easy to use.
Despite the fact of the advantages, there are a couple of things to look out for.
Failure handling
Circuit breaking in case a service is unavailable to respond
Consistency
Retries
Using service discovery (e.g. Eureka) to make the system more flexible when calling another service
On the messaging side, you have to deal with asynchronous processing, infrastructure problems like setting up the message broker and maintaining it, it's not as easy to use as pure HTTP, but you can solve consistency problems with just being eventually consistent.
Overall, there are tons of things which you have to consider and everything is about trade-offs. If you are just starting with microservices, I think it's best to start with using HTTP for communication and then slowly going to the messaging alternative.
For example in the Java + Spring Cloud Netflix world, you can have Eureka with Feign and with that it's really easy to use logical address to the services which is translated by Eureka to actual IP and ports. Also, if you wanna use Swagger for your REST APIs, you can even generate Feign client stubs from it.
I've had the same question on my mind for a while now and still cannot find a good generic solutions... For what it's worth...
If the communication is one way and the "caller" does not need to wait for a result, I find Kinesis streams very powerful - just post a "task" onto the stream and have the stream trigger a lambda to process it. But obviously, this works in very limited cases...
For the response-reply world, I call the API Gateway endpoints just like an end user would (with the added overhead of marshaling and unmarshaling data to "fit" in the HTTP world, and unnecessary multiple authentications).
In rare cases, I may have a single backend lambda function which gets invoked by both the Gateway API lambda and other microservices directly. This adds an extra "hop" for "end users" (instead of [UI -> Gateway API -> GatewayAPI lambda], now I have [UI -> Gateway API -> GatewayAPI lambda -> Backend lambda]), but makes microservice originated calls faster (since the call and all associated data no longer need to be "tunneled" through an HTTP request). Plus, this makes the architecture more complicated (I no longer have a single official API, but now have a "back channel" direct dependencies).
To reduce the cost on instances, we were looking for options.
AWS lambda seems to be a good option for us.
Its still in the preliminary stage of searching for available alternatives.
My concern is if we switch some of our applications to lambda, we will be confined to use AWS environments only , and in future it might become a boundation for a scenario , which we cant predict at the moment.
So my question is, is there a way that we can still use lambda in an environment which is not an AWS environment.
Thanks!
AWS Lambda functions are basically containers, where its lifecycle is managed by Amazon.
When you use Lambda, there are several best practices you can follow, to avoid full locking. One of the recommended practice is to separate the business logic from Lambda handler. When you separate the Lambda handler, it only works as the controller which points to the executing code.
/handler.js
/lib
/create-items
/list-items
For example, if you design a web application API this way with NodeJS in Lambda, you can later move the business logic to an ExpressJS server by moving the handler code to ExpressJS Routes.
As you can see, you will still require putting additional effort to move an application from Lambda to another environment. By properly designing, you can only reduce the efforts.
As per my knowledge,
Its AWS lambda function, so it is suppose to be deployed on AWS instances only, because they support the needed environment.
From AWS site there are couple of options ...
https://docs.aws.amazon.com/lambda/latest/dg/deploying-lambda-apps.html
When running functions in AWS lambda, it's common to use environment variables to control settings. However, when invoking Lambda via API gateway you have 'stage variables' to contend with.
My question is this: is an AWS Lambda instance scoped to a particular API gateway stage when invoked from API gateway, such that I can rely on the stage not changing between calls. In effect, does each API 'stage' get it's own pool of instances to work with, which are recycled in accordance with stage variables?
Examples of where I might want to depend on this behaviour:
Creating connections to tables - the table name will be different per-stage, so if I create the connection on first usage I'd end up using the first callers stage context. What happens when I make a call on a different API gateway stage?
Varying JWT keys for environments.
My gut feeling on this is that if API gateway has two versions/stages of the deployment referencing the exact same function verison, the lambda-managed function instances can recieve calls from two stages interchangably, and I shoulnd't cache the context and request derrived information (stage-variables) variables in process.
There's a lot of AWS API Gateway / Lambda stuff out there, but couldn't find a clear answer to this issue.
You're right, a single Lambda function version will have a pool of instances that are totally independent. Different API Gateway stages and even different APIs can call the same function and this has no impact on the instance pool in Lambda.
So any in-function caching you're doing should not use the assumption that only a specific API and/or stage will access the cached data.
I am using AWS to build an API, and deploy this to multiple stages.
When a call is made to a specific environment, I need to get a stage variable in Lambda and then data is recorded in a DynamoDB table such as "environment-Table".
Is this the best way to work with environments (like development, production etc) using AWS API Gateway, Lambda and DynamoDB?
It difficult to say what the best approach is for your specific situation, given the limited data in your post. Managing multiple environments such as development and production was one of the intended uses of stage and stage variables. I don't see any obvious problems with what your are proposing.
Depending on your use case, you can call a Lambda function to record data in DynamoDB, or you may be able to skip the Lambda function and record the data in DynamoDB directly using the AWS proxy integration type.