POST url encoded form to Amazon API Gateway - amazon-web-services

I'm creating a webhook to receive notifications from a 3rd-party service, they sent the data in the body of a POST with content type application/x-www-form-urlencoded.
But it generates the same error:
{"message": "Could not parse request body into json: Unrecognized token \'name\': was expecting \'null\', \'true\', \'false\' or NaN\n at [Source: [B#456fe137; line: 1, column: 6]"}
I could reproduce the error with the following curl call:
% curl -v -X POST -d 'name=Ignacio&city=Tehuixtla' https://rl9b6lh8gk.execute-api.us-east-1.amazonaws.com/prod/mandrillListener
* Trying 54.230.227.63...
* Connected to rl9b6lh8gk.execute-api.us-east-1.amazonaws.com (54.230.227.63) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-east-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> POST /prod/mandrillListener HTTP/1.1
> Host: rl9b6lh8gk.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 27 out of 27 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Content-Length: 180
< Connection: keep-alive
< Date: Thu, 28 Jan 2016 12:29:40 GMT
< x-amzn-RequestId: cd4d9232-c5ba-11e5-a158-b9b39f0b0599
< X-Cache: Error from cloudfront
< Via: 1.1 1915b8b49d2fbff532431a79650103eb.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: cxU2_b5DzIw4M_n3hJBFXTu9AVRBL3GpbQqUId9IxgS004DfLYqYmg==
<
* Connection #0 to host rl9b6lh8gk.execute-api.us-east-1.amazonaws.com left intact
{"message": "Could not parse request body into json: Unrecognized token \'name\': was expecting \'null\', \'true\', \'false\' or NaN\n at [Source: [B#d92973b; line: 1, column: 6]"}
If I wrap the body with double-quotes it works fine:
% curl -v -X POST -d '"name=Ignacio&city=Tehuixtla"' https://rl9b6lh8gk.execute-api.us-east-1.amazonaws.com/prod/mandrillListener
* Trying 54.230.227.19...
* Connected to rl9b6lh8gk.execute-api.us-east-1.amazonaws.com (54.230.227.19) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-east-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> POST /prod/mandrillListener HTTP/1.1
> Host: rl9b6lh8gk.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 6
< Connection: keep-alive
< Date: Thu, 28 Jan 2016 12:33:20 GMT
< x-amzn-RequestId: 50610606-c5bb-11e5-b140-5d837ffe26ed
< X-Cache: Miss from cloudfront
< Via: 1.1 a670cda0e28541e40881b95b60c672b7.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: mCLKL4eOnpUMd15IXQZw0RStJHw9Vdf3ivdCl37dcmno2JFOfxw0Vg==
<
* Connection #0 to host rl9b6lh8gk.execute-api.us-east-1.amazonaws.com left intact
"true"%
The lamba has only one line:
context.succeed('true');
How can I make the api gateway do not treat the body as json?
I tried the documentation about template mapping with no success, I even tried to convert it to a static template, with no variables at all! In all cases the error happens before getting to my code.

Try to set mapping template as following:
{
"body" : $input.json('$')
}
This would convert you string into json and pass to lambda.
From amazon docs:
$input.json(x) function evaluates a JSONPath expression and returns the results as a JSON string.

This is not entirely related, but if you are new to Amazon API Gateway, one additional step I did not know was required was to (re) deploy your API after adding the mapping template as others have suggested (in the case you had previously deployed your API). This cost me a bunch of debugging time as I did not understand why I was continuing to get this error even after making the suggestions posted here.
If using the AWS Console,
navigate to any pane within your API
Select Actions menu at the top
Select Deploy API from the menu, choose the relevant stage and confirm

The mapping template to make form data work is pretty complicated. Here is a gist: https://gist.github.com/ryanray/668022ad2432e38493df
Also, you can see this post I wrote that has an example of how to integrate with Slack(their hooks send a POST as form data to API Gateway): http://www.ryanray.me/serverless-slack-integrations

In the API Gateway, select the POST method for your resource, select Integration Request and create a new Mapping Template for application/x-www-form-urlencoded:
#set($body = $input.path('$'))
#set($jsonString = $util.urlencode($body))
#set($json = $util.parsejson($jsonString))
{
"body" : $json,
}
Alternatively, you can simply pass the url encoded string:
#set($body = $input.path('$'))
{
"body" : "$body",
}
and url decode and parse the JSON in your lambda.

Related

Cognito authorization endpoint (without client secret) returning Invalid client error message

I have a problem with Cognito and api clients like Postman or Insomnia.
There is a mobile app that makes calls to the backend.
There is an AWS Cognito instance, with one user pool and one API client, configured for using Authorization Code, with Cognito User Pool set as an Identity Provider
At first, the API client was configured to use client secret. I was able to make API calls from Postman or Insomnia using Oauth2 authentication, but for some unknown reason I wasn't able to authenticate using the mobile app
Then there was a change in the infrastructure - the old API Client entry in Cognito was recreated, but configured NOT to use client secret. We immediately removed client secret data from the code. After that, I was able to log in from the mobile application and send requests to the backend, but now I cannot authenticate with Postman/Insomnia. The browser window is opening, I can see the credentials form, I can properly login, but after that when Postman is calling the token endpoint, I get a browser window with one message in it:
An error was encountered with the requested page.
And I do not receive my tokens. Postman says:
Authentication failed
Couldn’t complete authentication. Check the Postman Console for more details.
Insomnia:
[oauth2] Failed to fetch token url=https://my-app-address.amazoncognito.com/oauth2/token status=400
And finally, here's the Insomnia's response timeline:
* Preparing request to https://my-app-name.auth.eu-west-1.amazoncognito.com/oauth2/token
* Current time is 2023-01-04T10:56:38.314Z
* Enable automatic URL encoding
* Using default HTTP version
* Enable timeout of 30000ms
* Enable SSL validation
* Enable cookie sending with jar of 2 cookies
* Found bundle for host my-app-name.auth.eu-west-1.amazoncognito.com: 0x12f2ee990 [can multiplex]
* Re-using existing connection! (#13) with host my-app-name.auth.eu-west-1.amazoncognito.com
* Connected to my-app-name.auth.eu-west-1.amazoncognito.com (x.x.x.x) port 443 (#13)
* Using Stream ID: 9 (easy handle 0x14e9f8400)
> POST /oauth2/token HTTP/2
> Host: my-app-name.auth.eu-west-1.amazoncognito.com
> user-agent: insomnia/2022.7.0
> cookie: XSRF-TOKEN=f1264cb4-b688-41a3-9126-cf021df2fa30
> content-type: application/x-www-form-urlencoded
> accept: application/x-www-form-urlencoded, application/json
> authorization: Basic MzJkN2JvnzNhNUTzNGNIN3UzdjY5b3Zkb246ZXI=
> content-length: 166
| grant_type=authorization_code&code=9e3e6ae8-30c7-6c5c-9aee-1930131a6624&redirect_uri=myapp%3A%2F%2Ffrontpage&code_verifier=dhQD1EtvMm_yP6eGorgQU7budSloaspeuGUM_OzS34k
* We are completely uploaded and fine
< HTTP/2 400
< date: Wed, 04 Jan 2023 10:56:38 GMT
< content-type: application/json;charset=UTF-8
< x-amz-cognito-request-id: 71156586-7bd3-4485-944c-07b5b930ce15
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< strict-transport-security: max-age=31536000 ; includeSubDomains
< x-frame-options: DENY
< server: Server
* Received 26 B chunk
* Connection #13 to host my-app-name.auth.eu-west-1.amazoncognito.com left intact
| {"error":"invalid_client"}
And the cognito user pool config:
resource "aws_cognito_user_pool_client" "my_app_client" {
name = "my-app-dev"
user_pool_id = aws_cognito_user_pool.default.id
allowed_oauth_flows = ["code"]
allowed_oauth_flows_user_pool_client = true
allowed_oauth_scopes = ["email", "openid"]
callback_urls = ["myapp://frontpage"]
logout_urls = ["myapp://signout"]
supported_identity_providers = ["COGNITO"]
}
What could be the reason why I am unable to authenticate using Postman/Insomnia and receive such errors? Could this be something related to AWS configuration? Or I'm doing something wrong?
As you have not configured the client secret in the App client, authorization header should not be added in the token request.
As mentioned in the document, authorization header needs to be provided only if the client was issued a secret.

AWS Api Gateway cannot rewrite path - 400 Bad Request error

I'm trying to set up an AWS API Gateway for something which was previously handled by an nginx reverse proxy. My endpoints are EC2 instances inside a VPC. I've already set it up so the gateway can access these instances.
The previous nginx setup looked like this:
http {
server {
listen 80;
location /host1/ {
proxy_pass http://host1:8000/;
}
location /host2/ {
proxy_pass http://host2:8070/;
}
...
}
}
The Problem arises when I try to rewrite the request path. I've set up a test route in the Gateway: ANY /test/{proxy+}, which I passed to the corresponding EC2 instance. I've verified, that requests pass through, but they contain the complete paths of the requests:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 404
< date: Sat, 18 Dec 2021 09:21:42 GMT
< content-type: text/html;charset=utf-8
< content-length: 469
< server: SimpleHTTP/0.6 Python/3.7.10
< apigw-requestid: Kic2FiLIFiAEN_g=
<
--- response ---
# server:
192.168.9.6 - - [18/Dec/2021 09:15:05] "GET /test/hello_world/test/a HTTP/1.1" 404 -
(the 404 is expected, the important part is the request hitting the server)
I then tried to rewrite the request path to remove the leading /test using a parameter mapping: I specified "all incoming requests", Parameter to modify: path, Modification type: overwrite, Value: $request.path.proxy (the catch-all field defined in the route).
Now I get a 400 error, and the requests don't hit my server anymore:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 400
< date: Sat, 18 Dec 2021 09:19:53 GMT
< content-type: text/html
< content-length: 122
< server: awselb/2.0
< apigw-requestid: KiclDhxXFiAEMhg=
<
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
# server:
-nothing-
When I map the $request.path.proxy to querystring.path, instead of path the requests hit the server:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 404
< date: Sat, 18 Dec 2021 09:21:42 GMT
< content-type: text/html;charset=utf-8
< content-length: 469
< server: SimpleHTTP/0.6 Python/3.7.10
< apigw-requestid: Kic2FiLIFiAEN_g=
<
--- response ---
# server:
192.168.9.6 - - [18/Dec/2021 09:21:42] "GET /test/hello_world/test/a?path=hello_world%2Ftest%2Fa HTTP/1.1" 404 -
notice the value of the path query parameter is exactly the correct value which I would have wanted to replace the original requests path.
Is this a bug with AWS, or am I just missing some documentation, stating that you cannot rewrite path that way? Notably, when the {proxy+} path parameter is empty, requests get routed through correctly...
The problem was with the value of the path rewrite: It should have been /$request.path.proxy instead of $request.path.proxy.

django : don't return cookie for a particular endpoint

I need to return a response from Django without returning a cookie.
I'm trying to implement a webhook client API that requires:
the use of https
response within 5 seconds
no body in the response
no cookies in the response headers
a 401 unauthorised status code for invalid hmac signatures
I'm working on Django 1.10 (soon to be upgraded to 2.x) where the rest of the app is protected by user validation via sessions.
Part of the endpoint view is as follows:
response200 = HttpResponse(status=200)
response401 = HttpResponse(status=401)
response401.close() # attempt not to set cookie
signature = request.META.get('HTTP_WEBHOOK_SIGNATURE')
if not request.method == 'POST':
return response401
if not signature:
return response401
and so on.
However my attempt to avoid setting the session using response401.close() doesn't work. I've also tried del response401['Set-Cookie']see Django docs
The cookie LocalTest... is still set in this curl session:
$ curl -d "param1=value1&param2=value2" \
-H "webhook-signature: $SIGVAL" \
-H "Content-Type: application/x-www-form-urlencoded" \
-X POST http://127.0.0.1:8000/invoices/webhookendpoint \
-w "\n" -v
...
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> POST /invoices/webhookendpoint HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.52.1
> Accept: */*
> x-xero-signature: ZSlYlcsLbYmas53uHNrBFiVL0bLbIKetQI6x8JausfA=n
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 27
>
* upload completely sent off: 27 out of 27 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 401 Unauthorized
< Date: Thu, 11 Apr 2019 08:32:50 GMT
< Server: WSGIServer/0.1 Python/2.7.13
< Vary: Cookie
< Content-Type: text/html; charset=utf-8
< Set-Cookie: LocalTest=gwx7jhsshy2qvtct1rmzv86h7xshe6ot; httponly; Path=/
<
* Curl_http_done: called premature == 0
* Closing connection 0
It appears that this works:
# ensure no cookie header is set
del request.session
response200 = HttpResponse(status=200)
response401 = HttpResponse(status=401)
...
as shown in the curl response:
< HTTP/1.0 200 OK
< Date: Thu, 11 Apr 2019 08:49:28 GMT
< Server: WSGIServer/0.1 Python/2.7.13
< Content-Type: text/html; charset=utf-8
<
Naturally, if you go to this endpoint as a logged in user, you will have to log in again.

Betfair API'ing Certificate login using C++ boost SSL Sockets

Firstly, thanks for coming here. I'm trying to login with betfair using a certificate login using boost's ssl sockets however, once I send my http login POST, I receive the message CERT_AUTH_REQUIRED. On the betfair website it says this means "Certificate required or certificate present but could not authenticate with it".
I am able to connect, handshake and send/receive data. However, I just can't seem to login with my code. I've tested the exact certificates using curl without any problems.
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=IE; ST=Leinster; L=Dublin; O=Paddy Power Betfair Public Limi
ted Company; OU=IT Networks; CN=betfair.com
* start date: Sep 11 05:50:38 2018 GMT
* expire date: Sep 11 05:59:00 2020 GMT
* issuer: C=US; O=HydrantID (Avalanche Cloud Corporation); CN=HydrantID S
SL ICA G2
* SSL certificate verify result: self signed certificate in certificate c
hain (19), continuing anyway.
> POST /api/certlogin HTTP/1.1
> Host: identitysso-cert.betfair.com
> User-Agent: curl/7.46.0
> Accept: */*
> X-Application: AOxcQMZwVN3jOsLZ4
> Content-Length: 41
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 41 out of 41 bytes
< HTTP/1.1 200 OK
< Content-Type: text/plain;charset=ISO-8859-1
< Content-Length: 87
< Date: Wed, 06 Mar 2019 11:09:35 GMT
<
{"sessionToken":"ZFbyo3HeAh07UFTHzhhGjOyQFeX2MKdHHHHtAm2S7FXw=","loginStatus":"SU
CCESS"}* Connection #0 to host identitysso-cert.betfair.com left intact
I have further tested these certificates with python code that also works.
My C++ code is below. I've tried sending incorrect passwords which result in the status from server INVALID_USERNAME_OR_PASSWORD instead.
boost::asio::ssl::context ctx(boost::asio::ssl::context::tlsv12);
// load certificates
ctx.load_verify_file(cert_filename.c_str());
// ctx.use_private_key_file(private_filename.c_str(), boost::asio::ssl::context::pem);
ctx.use_rsa_private_key_file(private_filename.c_str(), boost::asio::ssl::context::pem);
mSocket.reset(new boost::asio::ssl::stream<tcp::socket>(mIoService, ctx));
mSocket->set_verify_mode(boost::asio::ssl::verify_peer);
mSocket->set_verify_callback(
boost::bind(&BetfairSession::VerifyCertificate, this, _1, _2));
tcp::resolver resolver(mIoService);
tcp::resolver::query query("identitysso-cert.betfair.com", port);
tcp::resolver::iterator endpointIter = resolver.resolve(query);
Many thanks in advance :)
I think you are confusing the client certificate and the CA List.
This:
ctx.load_verify_file(cert_filename.c_str());
Is loading a list of CA certificates to verify the server certificate against.
You can find a example of this list here:
http://curl.haxx.se/ca/cacert.pem
You also need to setup the certificate to use for the SSL connection, you do this with the "use_certificate_chain_file" method.
e.g.
ctx.use_certificate_chain_file(cert_filename.c_str());

webserver based on node.js couldn't parse the right data which sent by method 'POST' via CURL

I implemented a simple webserver based on node.js, which handle the data
sent by method 'POST' via libCURL.
Client: c++ code based on libCURL
Server: Node.js
Result:
Client
I implemented a simple webserver based on node.js, which handle the data
sent by method 'POST' via libCURL.
Client: c++ code based on libCURL
Server: Node.js
###Client####
>>>
* Found bundle for host 109.123.121.146: 0x1b13f00 [can pipeline]
* Re-using existing connection! (#0) with host 109.123.121.146
* Connected to 109.123.121.146 (109.123.121.146) port 8124 (#0)
> POST / HTTP/1.1
Host: 109.123.121.146:8124
Accept: */*
Content-Type: text/xml
Content-Length: 11
* upload completely sent off: 11 out of 11 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: text/plain;charset=utf-8
Content-Type: text/plain;charset=utf-8
< Date: Thu, 05 Jan 2017 21:14:05 GMT
Date: Thu, 05 Jan 2017 21:14:05 GMT
< Connection: keep-alive
Connection: keep-alive
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
<
* Connection #0 to host 109.123.121.146 left intact
###Server####
####### 'POST' ######
Partial body: B=~� size:11
Body: B=~� size:11 ---->> why the data is invalid char, size is right.
Source Code
Client.cpp
string strUrl = "http://109.123.121.146:8124/";
string strLog = "Hello,CURL!";
curl_easy_setopt(g_curl_handle, CURLOPT_URL, strUrl.c_str());
curl_easy_setopt(g_curl_handle, CURLOPT_POST, 1);
curl_easy_setopt(g_curl_handle, CURLOPT_POSTFIELDS, strLog.c_str());
Server.js
req.addListener('data', function (data) {
body += data;
console.log("Partial body: " + data + " size:"+data.toString().length);
});
req.addListener('end', function () {
console.log("Body: " + body + " size:"+body.toString().length);
});
I wonder why the request data is invalid char:"B=~�" , or not the right string "Hello,CURL!",
Anyone konws, help to explain to me, thanks ~~~