We're configuring an AWS API Gateway proxy in front of Elasticsearch deployed on Elastic Cloud (for throttling, usage plans, and various other reasons). In order to authenticate between the Gateway and ES, one idea is to configure an integration request on the API Gateway resource to add an Authorization header with creds created in ES. Is this the best strategy? It seems inferior to IAM roles, but that option isn't available as they're not accessible for the ES instance (Elastic Cloud hosts our deployment on AWS, but it's not a resource under our control). The API Gateway itself will require an API key.
I am not an expert at Elasticsearch, but it sounds like you want to securely forward a request from API gateway to another REST web service. Because Elasticsearch is an external REST web service to AWS, you will not have access to IAM roles. I had a similar integration to another cloud rest service (not elasticsearch) will do my best to review the tools in AWS that are available to complete the request.
One idea is to configure an integration request on the API Gateway resource to add an Authorization header with creds created in ES. Is this the best strategy?
This is the most straightforward strategy. In API Gateway, you can map custom headers in the Integration Request. This is where you will map your Authorization header for Elastic Search.
Similarly you can map your Authorization Header as a "Stage Variable" which will make it easier to maintain if the Authorization Header will change across different Elasticsearch environments.
In both strategies, you are storing your Authorization Header in API Gateway. Since the request to Elasticsearch should be HTTPS, the data will be secure in transit. This thread has more information about storing credentials in API Gateway.
From MikeD#AWS: There are currently no known issues with using stage variables to manage credentials; however, stage variables were not explicitly designed to be a secure mechanism for credentials management. Like all API Gateway configuration information, stage variables are protected using standard AWS permissions and policies and they are encrypted when transmitted over the wire. Internally, stage variables are treated as confidential customer information.
I think this applies to your question. You can store the Authorization Header in the API Gateway Proxy, however you have to acknowledge that API Gateway Configuration information was not explicitly designed for sensitive information. That being said, there are no known issues with doing so. This approach is the most straightforward to configure if you are willing to assume that risk.
What is a more "AWS" Approach?
An "AWS" approach would be to use the services designed for the function. For example, using the Key Management Service to store your Elasticsearch Authorization Header.
Similarly to the tutorial referenced in the comments, you will want to forward your request from API Gateway to Lambda. You will be responsible for creating the HTTPS request to Elasticsearch in the language of your choice. There are several tutorials on this but this is the official AWS documentation. AWS provides blueprints as a template to start a Lambda Function. The Blueprint https-request will work.
Once the request is being forwarded from API Gateway to Lambda, configure the authorization header for the Lambda request as an Environment Variable and implement Environment Variable Encryption. This is a secure recommended way to store sensitive data, such as the Elasticsearch authorization header.
This approach will require more configuration but uses AWS services for intended purposes.
My Opinion: I initially used the first approach (Authorization Headers in API Gateway) to authenticate with a dev instance because it was quick and easy, but as I learned more I decided the second approach was more aligned with the AWS Well Architected Framework
Related
The Task:
Imagine a large-ish company or product that has many microservices run by separate teams. Each microservice exists in a separate repo. You want to build a single unified Apollo GraphQL API which collates all the subgraphs from the separate microservice APIs. And you want to build it using Serverless technologies. The Unified API should be authenticated using Cognito but the underlying subgraph APIs shouldn't be exposed to the public internet.
Ideal Scenario: AWS-AppSync would support federation natively.
In reality Scenario: Since this can't be done easily we have to run the Apollo Federation Server in a lambda which is fronted by API Gateway. The setup of the Federation server knows where the endpoints are for the subgraph.
Question: How to we construct the subgraphs using AppSync or further Lambda servers for each microservice, or something else? What techniques have people used to deploy Apollo Federation within AWS?
Design Considerations:
Resources Based Policies:
What I would prefer is the API-Gateway to be authenticated with Cognito but the App Sycn subgraphs to be given full access to the Lambda. However because App Sync doesn't support Resource Based Policies this isn't possible.
API-Key:
I can use API-Key auth on the subgraphs but since AppSync has publically accessible endpoints this feels like a security risk.
Cognito:
A possibility - would need to pass through Cognito auth from API-Gateway, to Lambda, then to subgraphs. Feels icky.
Lambda Authorization:
Add Lambda auth for subgraphs and use request context(?) to determine the request was coming internally. A hack for resource-based policies.
Out to Internet and Back Subgraph:
AppSync provides a pubilc url for the endpoint and composing the federated graph pulls the schemas to build the supegraph schema. This feels like internal services going out to the internet and then back in. The best solution would be some internal ip addresses / urls and hosting all the subgraphs within a private VPC.
Conclusion:
Building a secure federated graph feels hacky with serverless technologies. It feels like I should avoid AppSync all together and use a subgraphs (Private API Gateways in private VPC - powered by Lambdas) feeding info to a Public API Gateway authenticated by Cognito.
Interested in thoughts.
I have a private S3 bucket with lots of small files. I'd like to expose the contents of the bucket (only read-only access) using AWS API Gateway as a proxy. Both S3 bucket and AWS API Gateway belong to the same AWS account and are in the same VPC and Availability Zone.
AWS API Gateway comes in two types: HTTP API, REST API. The configuration options of REST API are more advanced, additionally, REST API supports much more AWS services integrations than the HTTP API. In fact, the use case I described above is fully covered in one of the documentation tabs of REST API. However, REST API has one huge disadvantage - it's about 70% more expensive than the HTTP API, the price comes with more configuration options but as for now, I need only one - integration with the S3 service that's why I believe this type of service is not well suited for my use case. I started searching if HTTP API can be integrated with S3, and so far I haven't found any way to achieve it.
I tried creating/editing service-linked roles associated with the HTTP API Gateway instance, but those roles can't be edited (only read-only access). As for now, I don't have any idea where I should search next, or if my goal is even achievable using HTTP API.
I am a fan of AWSs HTTP APIs.
I work daily with an API that serves a very similar purpose. The way I have done it is by using AWS Lambda functions integrated with the APIs paths.
What works for me is this:
Define your API paths, and integrate them with AWS Lambda functions.
Have your integrated Lambda function return a signed URL for any objects you want to provide access to through API calls.
There are several different ways to pass the name of the object(s) you want to the Lambda function servicing the API call.
This is the short answer. I plan to give a longer answer at a later time. But this has worked for me.
I have a task to replace current CA layer 7 with new API gateway.
New API gateway should be able to handle
1. Rate limiting
2. Authentication
3. Version handling etc.,
After researching i found we could use AWS api gateway or Kong api gateway or AWS ALB with Cognito for authentication support.
This is so overwhelming to understand the basic differences, could you please give some insight on basic concept in simple words and some pointers or link that i should refer to start with.
API Gateway keep track of every deploy you make in the Deployment History tab. There you will find all versions of your API and you can change to any of them whenever you want.
You can also create your api gateway from a Swagger file.
For every method that you create for a resource you need to configure the Method Request, the Integration Request, the Integration Response and the Method Response.
The Integration Request is where everything happens. You will set there how you are going to handle your requests, if you are going to integrate with any aws service like firehose or if you are going for a lambda integration or with an existing HTTP endpoint.
Mapping Templates uses Apache Velocity Template Language (VTL). http://velocity.apache.org/engine/1.7/vtl-reference.html
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
Getting started with REST apis:
https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html
API GATEWAY INTEGRATION TYPES:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-api-integration-types.html
How to import a rest api:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html
Limits and known issues:
https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html
Deploying:
https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-deploy-api.html
Publish:
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-publish-your-apis.html
AWS API Gateways supports lambda authoriser for authentication which is integrated with any identity provider - Azure AD, Cognito pool etc. It supports both Client Credentials (service to service) authentication and Authentication code(user based authentication) but AWS ALB don't support client credentials authentication flow.
AWS API Gateway also provides caching, request & response mapping, customise handling for each response type, request validation, throttling where AWS ALB is yet to be improved for all these feature.
Kong api gateway also provide similar feature as AWS API Gateway with added features
If all the backend services are deployed in AWS and you don't need
complex API gateway then go for AWS API Gateway. It is pay per use service and you don't need to pay for extra support for API gateway assuming your services are already deployed in AWS.
If you need api gateway solution with complex requirement and extra features then Kong API gateway can be considered. But you will need to either pay for Kong API gateway support or need extra effort in coding when used open source.
AWS ALB can be used only for specific scenarios and it is getting matured day by day.
As my project is going to be deployed on AWS, we started thinking about AWS API Gateway as a way to have one main entry point for all of our microservices(frankly speaking, we also would like to use by some other reasons like security). I was playing with API Gateway REST API and I had feeling that it it a bit incovinient if we have to register there every REST service we have.
I found very good option of using AWS API Gateway and lambda function as a proxy. It is described here:
https://medium.com/wolox-driving-innovation/https-medium-com-wolox-driving-innovation-building-microservices-api-aws-e9a455cc3456
https://aws.amazon.com/blogs/compute/using-api-gateway-with-vpc-endpoints-via-aws-lambda
I would like to know your opinion about this approach. May be you could also share some other approaches that can simplify API Gateway configuration for REST API?
There are few considerations when you proxy your existing services through API Gateway.
If your backend is not publicly then you need to setup a VPC and a site to site VPN connection from the VPC to your backend Network and use Lambda's to proxy your services.
If you need do any data transformations or aggregations, you need to use Lambda's(Inside VPC is optional unless VPN connection is needed).
If you have complex integrations behind the API gateway for your services, you can look into having ESB or Messaging Middleware running in your on-premise or AWS then proxy to API Gateway.
You can move data model schema validations to API Gateway.
You can move service authentication to API Gateway by writing a Custom Authorizer Lambda.
If you happen to move your User pool and identity service to AWS, you can migrate to AWS Cognito Manage Service and use AWS Cognito Authorizer in API Gateway to authenticate.
For usecases when you adopt dumb pipes (as described on martinfowler.com) AWS API Gateway is a reasonable option.
For AWS API Gateway I'd suggest to describe/design your API first with RAML or OpenAPI/Swagger and then import into AWS using AWS API Importer.
As soon as you plan to move logic in there, such as dynamic routing, detailed monitoring, alerting, etc, I'd suggest considering other approaches, such as:
Apigee
Mulesoft
WSO2
You can also host them on an EC2 within your VPC or opt-in for the hosted version. (which does have a significant pricetag in some cases)
For describing APIs you can use RAML (for Mulesoft) or OpenAPI (ex-Swagger, for Apigee and WSO2). You can also convert between them using APIMATIC which enables you to migrate your specification across various API Gateways (even AWS).
I have a serverless backend that operates with APIGateway and Lambda. Here is my architecture:
Currently, anyone with my APIGateway's URL can query or mutate the data. How do I protect the URL, so that only the client(react app) can access it. So, here is my concern, anyone can open the network tab in chrome console and get my APIGateway's URL and can use it using curl or postman. I want to prevent that.
Solutions I had in my mind:
Set up a CORS, so that only the origin can access it. But, I have a different lambda that invokes this URL. So, CORS wont work out.
I am sure there are some methods with the APIGateway itself. I am not getting right search term to get it from AWS documentation. I would also like to know what are the best practices to prevent accessing the backend URL apart from the Client(React App)
Update after #Ashan answer:
Thank you #Ashan for the answer. In my case, I use Auth0, so custom authoriser should work for me. I just came across this https://www.youtube.com/watch?v=n4hsWVXCuVI, which pretty much explains all the authorization and authentication possible with APIGateway. I am aware that authentication is possible either by Cognito/Auth0, but I have some simple websites, that has form, whose backend is handled by APIGateway. I can prevent the abuse from scraping bots using captcha, but once the attacker has got the URL, header and request parameters, he can invoke that million times. One thing, we can do is having an API-Key, but it is a static string with no expiration. Once the headers are with him, he can abuse it. So, any idea, how to prevent this in APIGateway. If not any other service apart from AWS that I can look for? Would be glad, If I get an answer for this.
Currently API Gateway does not support private urls, so it will be publicly available.
To restrict access you need to use a authorizer to authenticate and authorize the request using IAM policies. There are two options available at the moment.
IAM authorizer
Custom authorizer
If your authentication flow can directly (AWS STS, IAM user access keys or roles) or indirectly(Using AWS Cognito Userpools or any other SSO provider) can get temporary security credentials, then you can use IAM authorizer. From API Gateway side no code involved and its a matter of selecting the IAM check box for each API Gateway resource. You can use the API Gateway SDKs to invoke API Gateway requests where the SDK will handle the heavy liftings in setting up authentication headers.
If you use your own authentication mechanism, then you can write a seperate Lambda function to validate the tokens. This Lambda function name can be specified at API Gateway with the http hearder name to access the custom token to verify the requests.
To control API usage by authorized consumers, using API Key is the only way native to AWS at the moment.
Since you are using S3 for the react app hosting, you can further reduce the attack surface by using AWS WAF and CloudFront infront your application stack. The API Key can be added to CloudFront headers to forward to your APIGateway origin and since CloudFront and APIGateway communication happens using SSL, its nearly impossible for someone to find the API key. Using AWS WAF you can limit malicious access for common attacks. This includes rate based blocking to limit someone from repeatedly invoking the API.