I have built an EC2 reverse proxy (Nginx) that communicates with an external API endpoint over the internet. I have a Route53 DNS with an A record linking to my EC2. There are a few endpoints (Nginx locations) and depending on which url you hit, you are redirected to a specific proxy location, and forwarded to the right endpoint on the external API. It all works great.
Now i want to create some type of job that will test this process periodically to ensure that it's running and notify me if it's not. AWS has so many tools and i think i need to use Lambda and API Gateway.
I'd like to hit my url(Route53 DNS) go thru the EC2 and receive a response from the endpoint server. My site does this, postman can, but i can't figure out how to accomplish this in an automated way and alert me based on the response values.
how can i test my full pathway (www.example.com/option -> nginxEC2 path('/option') -> www.endpoint.com/option) and be notified based on the results.
EDIT: i need to be able to send a body with this. if i send it without body the server returns 404, if i can send with a body/payload, i'll get a response.
EDIT: basically looking for a way to hit my DNS, which thru A record, routes to my reverse proxy, to an endpoint. i just need to do an HTTP request to the Domain, and get and answer back and know the status code.
Mark B's solution is the closest as the free site he sent me has an option to pay for this service. gonna leave it open a few more days.
You definitely don't need API Gateway for this. That wouldn't help you test this at all. API Gateway would just give you an entirely new API that you would need to test.
You could use Lambda for this as you mentioned. You would write a Lambda function that hits the URLs you want to test, checks the results, and sends you a message over SES or SNS or some other means when it fails. The Lambda function could be configured to automatically run on a schedule.
However, AWS already has a service that does exactly what you are looking for: Route53 Health Checks.
What you are describing is called an HTTP health check or HTTP uptime monitor. There are tons of services that provide this feature, some of them free.
It looks like the word that you're looking for is trace -- you want to trace requests along your application. AWS offer for that is X-Ray. As you see in their official documentation, you need to use their SDK to instrument your application, which talks to a deamon in your EC2 instance. You can then integrate with CloudWatch and SNS to be notified upon errors (e.g. 4xx codes): https://aws.amazon.com/blogs/devops/using-amazon-cloudwatch-and-amazon-sns-to-notify-when-aws-x-ray-detects-elevated-levels-of-latency-errors-and-faults-in-your-application/
Hope it helps!
Related
I'm looking after solution where AWS Api Gateway changes method endpoint Url dynamically.
I am familiar with stage variables and in Integration request I can change endpoint per method like (https://${stageVariables.Url}/api/DoSomething).
What I need is that information how parse endpoint is included in requests.
https://${RequestData.Url}/api/DoSomething
I have same Api in different locations and to implement centralized Api keys and logging services I try to forward all traffic through this one Api Gateway.
After first request client gets its endpoint information, but I don't know how to solve that clients next requests to Gateway should forward to that endpoint which client get earlier.
I got an answer from AWS support. They told that I have to make a lambda function to process all requests or just use Stage variables.
I've got a Django app setup to use django-allauth to connect to Twitter. The flow is all working locally and I've followed the same setup steps on Lambda to add my tokens, site, etc.
When I try to access the login url (/accounts/twitter/login/) the request eventually times out with this message from AWS Lambda:
{"message": "Endpoint request timed out"}
The last message from zappa tail before the timeout event is:
[1619019159940] [DEBUG] 2021-04-21T15:32:39.939Z 7f66a0e3-58de-4612-82c0-54590d69676f Starting new HTTPS connection (1): api.twitter.com:443
I've seen that the gateways have a 30 second timeout but I don't think it should be taking this long anyway. Locally, it's taking a couple of seconds.
Does anyone with knowledge of these platforms have an idea where the bottleneck might be and where the issue could be? Or have any pointed questions to help debug?
Things I've already checked and (tentatively) ruled out:
The database backend is AWS Aurora Serverless, and I did worry that the double-serverless setup might be causing the slow speeds. However, a simple call of the Django management command (zappa manage dev migrate returns takes less than a second so I've ruled that out for now. Plus the Admin dashboard loads fine which is also accessing the DB.
I've got both the dev and live URLs added into Twitter's dashboard as valid OAuth callback URLs.
Leaving this answer to help future searches, although it's not the route I'll take.
Thanks to #Jens in the comments for pointing towards the VPC issue. You need to add a NAT Gateway service to the Lambda to add public internet access to a private VPC.
"To grant internet access to your function, its associated VPC must have a NAT gateway (or NAT instance) in a public subnet."
Source: https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
This is a per-hour billed extra so might defeat the point of using serverless (if you're using it for small fees and simplicity rather than scale like I was).
Currently, AWS ELB, both classic and application, do not log the body of POST requests. Neither does Cloudfront. Is it possible to get the full body of HTTP requests at this level above the instance?
You can't get access to full POST body in ELB logs, it's not really what the ELB logs are for. They are more around request counts and timings etc. Also, the body of a POST could contain all sorts of sensitive information - email, password, date of birth, address etc.
It's also fairly unusual that logs would contain this kind of information, it's generally captured by the request handling application.
If recording the request in your application isn't an option you could look at Traffic Mirroring - one of the use cases is content inspection. This will allow you to copy all traffic from the ELB ENI and send it to another application, e.g. a service running on an EC2 instance that writes the body to and S3 bucket.
You can inspect HTTP request contents via WAF, and block based on certain conditions. This occurs before the request reaches the application. See https://aws.amazon.com/about-aws/whats-new/2016/01/aws-waf-now-inspects-http-request-body-and-adds-size-constraint-condition/
I have an API Gateway endpoint that proxies a Lambda function. When I test the endpoint in the console (a GET request), it works fine and returns the correct JSON.
When I try the same endpoint in the browser or Postman, it doesn't work. The error is:
server IP address could not be found
I have logs set up in CloudWatch for the API and nothing comes up, so it looks like the API is not even getting called.
I'm using the serverless framework with NodeJS. I've tried deleting my whole CloudFormation stack and redeploying from scratch but it didn't make a difference. The API was working fine until today.
I'm having the exact same issue. Specificially, Starbucks wifi doesn't like AWS API Gateway endpoints but quite a few other public wifi too. The solution is to use custom domain for the API Gateway endpoints, I'm afraid
I am fairly new to AWS Lambda but sure can see the benefits of it and stumbled upon the superb framework Serverless to help me built solutions on Lambda.
I started out building solutions using AWS API Gateway but really need "internal" VPC API's and not public Internet facing API's like API GW creates.
I found that Servless indeed can expose a HTTP endpoint but I can't figure out how this is done and how the URL is created.
When I deploy the Lambda from Serverless it gives me the URL, e.g.:
https://uxezd6ry8z.execute-api.eu-west-1.amazonaws.com/dev/ping
I would like to be able to find (or create) this same http listener for already existing Lambdas so my question is how is the URL created and where is teh actual HTTP listener deployed?
You might be looking for the invoke url,
1. go to https://console.aws.amazon.com/apigateway
2. select api link (which you have deployed on aws lambda).
3. select stages in left side panel and
see the invoke url.
Adding a http listener can be done by going to your lambda function, selecting the 'triggers' tab and 'add trigger', finally selecting API Gateway - but as others mentioned this does create a public facing url.
Duh, I was in the wrong AWS logon previously so the API GW was not showing any matching Serverless API and that was why I couldn't understand how they did it...
Once I logged into the AWS account that hosts the Serverless structure I can see the API GW GET API's for the Serverless HTTP listener.