I am using ALB with Lambda Integration. My ALB gets a request from a server, forwards it to the Lambda and forwards the lambda response back to the server.
However, the server only accepts an xml body and not the whole request with headers that are added by ALB. (Please see responses below).
Is there a way for ALB to only send the body back?
The server expects something like this:
[<?xml version="1.0" encoding="UTF-8"?>
<EventMessage xmlns:hb="Heartbeat" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Message.xsd">
...
</EventMessage>
]
ALB sends back this:
[HTTP/1.1 200 OK
Server: awselb/2.0
Date: Mon, 19 Dec 2022 14:52:01 GMT
Content-Type: application/octet-stream
Content-Length: 28
Connection: keep-alive
<EventMessage xmlns:hb="Heartbeat" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Message.xsd">
...
</EventMessage>
]
I have no control over what the server accepts, so somehow I need to remove the headers and send back the eventMessage which can be accepted.
Here is my current return (callback) statement in the lambda:
const resFromRequestsLambda = {
"isBase64Encoded": false,
"statusCode": 200,
"body": "<EventMessage ...>...</EventMessage>,
}
callback(null, resFromRequestsLambda);
Thanks for any help!
ps. I realise this may be a very secific case, and possibly the answer is you 'can't/shouldn't'. Any way around this would be great if i can't directly change the way alb responds to the server.
These headers are part of the http protocol. The load balancer speaks http, so it will always include the headers. There is no way around it.
It’s exceedingly odd that the server won’t accept the headers. Perhaps you can use a custom TCP server on an EC2 instance instead of an ALB with a Lambda target. This way you can respond to requests with total control over the payload.
Related
I have configured the following redirect rule to my AWS Application Load Balancer to redirect all HTTP traffic to HTTPS:
The issue is that when I now curl (or visit the domain in the browser), I will get this ugly and redundant Location response (domain changed to example.com):
~ $ curl -I http://www.example.com
HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Mon, 14 Sep 2020 18:28:48 GMT
Content-Type: text/html
Content-Length: 150
Connection: keep-alive
Location: https://www.example.com:443/
I know that https://www.example.com:443/ is in practice just fine, and I know that it will not be shown in the end user's browser's URL field. But still, it will be shown in the browser's network tab's 'Response headers', and to me it just looks unprofessional compared to a redirect without the port, e.g.:
~ $ curl -I http://www.apple.com
HTTP/1.1 301 Moved Permanently
Server: AkamaiGHost
Content-Length: 0
Location: https://www.apple.com/
Cache-Control: max-age=0
Expires: Mon, 14 Sep 2020 18:33:23 GMT
Date: Mon, 14 Sep 2020 18:33:23 GMT
Connection: keep-alive
strict-transport-security: max-age=31536000
Set-Cookie: geo=FI; path=/; domain=.apple.com
Set-Cookie: ccl=izHQtSPVGso4jrdTGqyAkA==; path=/; domain=.apple.com
It would seem like a logical thing to just drop the port from the URL, but unfortunately it's a required field:
Also the 'Switch to full URL' option doesn't seem to really help, even though the port can be cleared there:
it still appears back after saving:
Is there any way to make this work?
Edit:
My domain is managed through AWS Route 53.
UPDATE: maybe it's worth mentioning this is nice as an exercise and show capabilities. However between a Lambda redirection and just a pure Load Balancer redirection, I'd still suggest you go with the ALB-native. It's more performatic (no cold-start, no extra hop), less error-prone (no coding required), for the cost of just aesthetics on a network trace or developer tools. End users would never notice that (you probably browse in lot of these sites everyday and you don't even know). So if you ask me, I'd suggest against going to production with something like this.
If you want something very simple as a redirect, you could just create a Target Group pointing to a Lambda function that returns a redirect. It doesn't require the overhead of managing a CloudFront distribution or trying to make it work behind a load balancer, as that would require changes in your DNS as well, since you cannot just place CloudFront behind a load balancer (not that I know of, at least).
For your case, I created a Lambda function (Python 3.8) with the following code:
def lambda_handler(event, context):
response = {}
if event.get('headers', {}).get('x-forwarded-proto', '') == 'http':
print(event)
response["statusCode"]=302
response["headers"] = {"Location": f"https://{event['headers']['host']}{event['path']}"}
response["body"]=""
return response
Then I created a new Target Group with that Lambda function as backend:
Finally, I configured my Listener on port 80 to the redirect Target Group:
With this implementation, your 302 redirect will show up as you expected:
This would still display in your browser's URL field without the port if this is what you're worrying about.
By default interpreters assume that if the protocol is HTTPS then the port is 443 so you do not need to specify anything, when the redirect occurs it does not add a port number to the end unless you specify a non-standard port such as 4430.
You have to specify the protocol (HTTPS) and port (443) simply because the configuration requires these items, this will not display for the user. This is the same as other AWS configurations such as during the configuration for an ELB.
You could put CloudFront in front of your server and have that perform https redirect. If you are only using ELB for SSL termination, you could remove it entirely and have CloudFront terminate your SSL. This would save you some money as ELB is pretty expensive if your only using it for one origin server.
https://aws.amazon.com/cloudfront/
You can use CloudFront and set the the origin as the load balancer. Then force HTTPS redirection on the cloud-front with Viewer Protocol Policy set to Redirect HTTP to HTTPS.
In this approach, all the ELB rules will still be applied.
We are catching a BigCommerce webhook event in our Google Cloud Run application. The request looks like:
Headers
host: abc-123-ue.a.run.app
AccountId: ABC
Content-Type: application/json
Password: Goodbye
Platform: BC
User-Agent: akka-http/10.1.10
Username: Hello
Content-Length: 197
Connection: keep-alive
Body
{"created_at":1594914374,"store_id":"1001005173","producer":"stores/gy68868uk5","scope":"store/product/created","hash":"139fab64ded23b3e1b8473ba24ab21bedd3f535b","data":{"type":"product","id":132}}
For some reason, this causes a 400 response from Google Cloud Run. Our application doesn't even seem to be passed the request. All other endpoints work (including other post requests).
Any ideas?
Edit
In the original post, I had the path in the host header. This was a mistake made in creating this post and not the actual value passed to us. We can only inspect the request via Requestbin (I can't find the request values anywhere in Google logs) so I'm speculating on the host value and made a mistake writing it out here.
Research so far...
So upon further testing, it seems that BigCommerce Webhooks also fail to send to any Google Cloud Function we set up. As a workaround, I'm having Pipedream catch the webhook and send the payload to our application. No problems there. This endpoint also works with mirror payloads from local and Zapier which seems to eliminate authentication errors.
We are running FastAPI on Google Run and the simplest function on Google Cloud Functions. This seems to be an error with how Google Serverless and BigCommerce Webhook Events communicate with each other. I'm just not sure how...
Here are the headers we managed to capture on one of the only times a BigCommerce Webhook Event came through to our Google Cloud Function:
Content-Length: 197
Content-Type: application/json
Host: us-central1-abc-123.cloudfunctions.net
User-Agent: akka-http/10.1.10
Forwarded: for="0.0.0.0";proto=https
Function-Execution-Id: unes7v34vzyo
X-Appengine-Country: ZZ
X-Appengine-Default-Version-Hostname: f696ddc1d56c3fd66p-tp.appspot.com
X-Appengine-Https: on
X-Appengine-Request-Log-Id: 5f10e15c00ff082ecbb02ee3a70001737e6636393664646331643536633366643636702d7470000165653637393633633164376565323033383131366437343031613365613263303a36000100
X-Appengine-Timeout-Ms: 599999
X-Appengine-User-Ip: 0.0.0.0
X-Cloud-Trace-Context: a62207698d141465d0f38488492d088b/9870406606828581415
X-Forwarded-For: 0.0.0.0
X-Forwarded-Proto: https
Accept-Encoding: gzip
Connection: close
> host: abc-123-ue.a.run.app/bigcommerce/webhooks/
This is most likely the issue. Host headers must contain only the hostname, not the request /paths.
You can clearly see this will fail:
$ curl -IvH 'Host: pdf-2wvlk7vg3a-uc.a.run.app/foo' https://pdf-2wvlk7vg3a-uc.a.run.app
...
HTTP/2 400
However if you don't craft the Host header yourself, it will work.
An AWS Lamba function (NodeJS) returning 3 HTTP headers: aaa, Access-Control-Allow-Origin and bbb was created:
exports.handler = async (event) => {
const response = {
statusCode: 200,
headers: { "aaa":"aaa", "Access-Control-Allow-Origin":"*", "bbb":"bbb" },
body: JSON.stringify('Hello from Lambda!'),
};
return response;
};
The function is integrated into a HTTP API (not REST API). In the HTTP API Gateway Configuration, Section "Configure CORS", the HTTP header "Access-Control-Allow-Origin" was set to "*". Please see the screenshot:
Gateway Config
The command "curl -i https://xxxxxxxxxx.execute-api.eu-central-1.amazonaws.com" proves that the HTTP Header Access-Control-Allow-Origin is explicitly removed, because only HTTP headers aaa and bbb are returned:
HTTP/2 200
date: Tue, 14 Apr 2020 11:01:58 GMT
content-type: text/plain; charset=utf-8
content-length: 20
aaa: aaa
bbb: bbb
apigw-requestid: K-S2EjVWliAEJKw=
Why on earth is this header still not present, even after "Configure CORS" was done?
(I'm googling now for more than two days in order to find a solution and it makes me go nuts)
As per Configuring CORS for an HTTP API -
If you configure CORS for an API, API Gateway ignores CORS headers
returned from your backend integration.
That's why the CORS headers from your Lambda (integration) are being ignored. This is one of the differences between the new HTTP APIs from the original REST APIs. In case of these APIs -
For a CORS request, API Gateway adds the configured CORS headers to
the response from an integration.
When you do a simple curl, that is not actual doing a cross-origin request. Hence, you don't see the CORS headers that would be set by the HTTP API. To verify if a CORS request works, I passed an Origin header in the below request and I can see the CORS headers along with my custom headers from Lambda -
$ curl -v -X GET https://$API_ID.execute-api.$AWS_REGION.amazonaws.com -H "Origin: https://www.example.com"
< HTTP/2 200
< date: Tue, 14 Apr 2020 18:02:26 GMT
< content-type: text/plain; charset=utf-8
< content-length: 18
< aaa: aaa
< bbb: bbb
< access-control-allow-origin: https://www.example.com
< access-control-expose-headers: date, x-api-id
Below is a snippet of my CORS configuration on the API. I added Access-Control-Allow-Origin value as https://www.example.com and passed this as a part of the Origin header in my curl request. Such a request would qualify as CORS.
For POST/PUT requests, you'll need to white list the content-type header.
Putting the wildcard doesn't do the trick for some reason, you need to explicitly whitelist it.
For Googlers:
If your OPTIONS preflights succeed but no Access-Control- headers present, and if you are testing using CURL, take extra attention to spell required headers for a preflight:
Access-Control-Request-Method: GET
Access-Control-Request-Headers: authorization
Origin: https://theaws.blog
If wrongly spelt, OPTIONS succeeds with 204 No Content, but no Access-Control- headers thus render the preflight result invalid.
Also check you have enough scope for the parameters, as long as your requested method and headers are a subset of the parameter, you will get Access-Control- headers. If not, you will get 204 No Content instead, which I would say it's not very informative!
I am trying to hit this URL https://subdomain.example.com in JMeter and recorded using the Blazemeter Chrome extension has all the necessary config elements but get an error:
HTTP/1.1 429 Too Many Requests
Content-Type: text/html; charset=utf-8
Content-Length: 1031
Connection: keep-alive
Cache-Control: private, no-cache, no-store, must-revalidate
Date: Tue, 20 Aug 2019 01:21:35 GMT
Expires: 0
p3p: CP="This site does not specify a policy in the P3P header"
I have tried coping the Header Cookies from Browser Header Response which works for sometime but then start throwing an error
As per HTTP Status Code 429 Too Many Requests description:
The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting").
A Retry-After header might be included to this response indicating how long to wait before making a new request.
So there are following options:
Your server is overloaded, in this case there is nothing you can do here apart from reporting the error as the bottleneck
Your script doesn't have proper correlation implemented, i.e. you're sending recorded hard-coded values instead of getting dynamic parameters
Your server doesn't allow such amount of requests from a single IP address within the given timeframe, you could try implementing IP Spoofing so your server would "think" that the requests are coming from the different machines.
Thanks for your reply. In the end I figured out that no limitation for number of calls implemented.
Now come to answer this is how I managed to work this:
Opened the page in chrome and from the header section copied all the header elements into the header manager hard coded.
First time it fails and returns p3p: CP="This site does not specify a policy in the P3P header" but also return the update variable value needed for next request which I extract and used in the next and subsequent Requests. The way I was able to find out which variable is changing by using the string comparison of 2 Response Headers
This was a difficult one but somehow worked with very minor change I also added the Header Manager to each request for safer side.
I'm writing a program in C++ that needs to download JSON data from an HTTPS URL. The program is based on wxWidgets. That URL is for the translation service at Glosbe
So I've tried multiple different libraries including:
libcurl
Boost.Asio
the http functionality included in wxWidgets
wxCurl
Urdl
However, it always throws an error saying it can't connect, or I get a reply that says "Moved Permanently".
When i copy and paste the URL I am testing it with into a browser, it returns the JSON data perfectly.
Does anyone know the correct way to do this?
Any help would be great!
301 Moved Permanently is what the server responds when you try to access the page with HTTP instead of HTTPS. Here's a complete response I just received from the server:
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Thu, 16 Jul 2015 20:25:01 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://en.glosbe.com/a-api
It means exactly that: "The content you are looking for is really at https://en.glosbe.com/a-api." Your browser simply adheres to the HTTP protocol by following the server's hint and automatically proceeding to request https://en.glosbe.com/a-api when you try to access http://en.glosbe.com/a-api. It works seamlessly for you as a user.
You will have to read more documentation to create HTTPS requests yourself. Each of the libraries you mentioned will have a different way of supporting HTTPS (or not support it at all). For example, have a look at http://www.boost.org/doc/libs/1_58_0/doc/html/boost_asio/overview/ssl.html, especially the "Notes" section where it says that "OpenSSL is required to make use of Boost.Asio's SSL support."