I'm trying to use libcurl in C++ to send requests to a url. When I set up the request in the command line with curl, it seems to work fine:
curl -vvv -X POST -H "Authorization: <api key here>" -H "Content-Type:application/json" "<host>" --data-binary '<json data here>'
The response starts something like this:
> POST <host> HTTP/1.1
> Host: <host>
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: <api_key>
> Content-Type:application/json
> Content-Length: 80
So i can see the authorization is being sent properly.
When I try to do a similar thing in C++, using the libcurl C library, however, I don't notice the ">" in front of the request headers:
Code:
struct curl_slist *chunk = NULL;
chunk = curl_slist_append(chunk, "Authorization: <api_key>");
chunk = curl_slist_append(chunk, "Content-Type:application/json");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
curl_easy_setopt(curl, CURLOPT_URL, "<host>");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "<json>");
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
Response:
> POST <host> HTTP/1.1
Host: <host>
Accept: */*
Authentication: <api_key>
Content-Type:application/json
Content-Length: 97
So I'm not even sure if the headers are even properly being processed or received by the host.
Any ideas?
I get the following response:
{
"message": "No authorization header given",
"code": 401
}
Incorrect Header. Should be Authorization, not Authentication.
In your command line verbose output the header is named "Authorization: ". In your libcurl vebose output its "Authentication: ". Authorization != Authentication?
Verbose output:
Its only a different output format for verbose between command line and libcurl. The headers are sent. Same output format is used for example by php curl. Only first line has ">" and then all following headers has no ">". But they are all submited.
PHP curl verbose example output:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 80 (#0)
> GET /XXX/api.php HTTP/1.1
Host: localhost
Accept: */*
Authorization: XXX
Content-Type: application/json
< HTTP/1.1 200 OK
< Date: Mon, 31 Dec 2018 20:12:51 GMT
< Server: Apache/2.4.34 (Win32) OpenSSL/1.1.0i PHP/7.2.10
< X-Powered-By: PHP/7.2.10
< Content-Length: 2390
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host localhost left intact
Related
vuejs is running inside a docker container served by:
CMD [ "http-server", "dist" ]
when using axios inside Vue.js mounted() to do a GET request against a flask api it shows "blocked" in the network tab, accessing other REST-API's works fine.
testing with curl (localhost:6000 being the flask server):
- curl is running from my real host and conneting to the container
curl -H "Origin: http://localhost:5000"
-H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested With"
-X OPTIONS --verbose http://localhost:6000/todo/api/v1.0/wheel/40
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 6000 (#0)
> OPTIONS /todo/api/v1.0/wheel/40 HTTP/1.1
> Host: localhost:6000
> User-Agent: curl/7.58.0
> Accept: */*
> Origin: http://localhost:5000
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Allow: OPTIONS, GET, HEAD
< Access-Control-Allow-Origin: http://localhost:5000
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT
< Vary: Origin
< Content-Length: 0
< Server: Werkzeug/1.0.0 Python/3.8.2
< Date: Sun, 15 Mar 2020 15:43:44 GMT
<
* Closing connection 0
from what ive read so far, for example here: 1, for a unauthorized GET request the headers look ok.
This one gets the real data:
curl -H "Origin: http://l:5000" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested-With" -v http://localhost:6000/todo/api/v1.0/wheel/40
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 6000 (#0)
> GET /todo/api/v1.0/wheel/40 HTTP/1.1
> Host: localhost:6000
> User-Agent: curl/7.58.0
> Accept: */*
> Origin: http://l:5000
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: application/json
< Content-Length: 39
< Access-Control-Allow-Origin: http://l:5000
< Vary: Origin
< Server: Werkzeug/1.0.0 Python/3.8.2
< Date: Sun, 15 Mar 2020 15:56:43 GMT
<
{"result":{"model 1":0,"model 2":150}}
* Closing connection 0
Manipulating -H "Origin..." to this:
-H "Origin: http://l:5000"
also shows a normal reply. Isn't that a good test?
As it turns out mozilla allows certain ports for certain protocols as shown here:
https://developer.mozilla.org/en-US/docs/Mozilla/Mozilla_Port_Blocking
6000 is the "x11" port and on that list - As port for x11 and not to be used for xhr. So every port not on that list should do the trick.
reason:
Cert issued a Vulnerability Note VU#476267 for a "Cross-Protocol" scripting attack, known as the HTML Form Protocol Attack
In your app's __init__.py file add these lines and you'll be good to go.
$ pip install flask-cors
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
Read more about CORS here-- MDN CORS
There are many questions on StackOverflow of users making a CORS request which doesn't properly set cookies. I'm having the same problem, but I'm wondering if Chrome or another browser provides a method to diagnose and understand why the cookies aren't set.
I'm finding all kinds of things that can cause this: missing CORS headers, missing withCredentails=true, situations across https secure boundaries, certain flags on the cookies... I want to understand what Chrome is doing instead of just guessing why it doesn't work.
Here's my specific problem, but really I'm after a process to understand not just the fix.
Domain Creating Request:
Note withCredentails = true.
var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
if (xmlhttp.readyState === XMLHttpRequest.DONE) {
callback(xmlhttp);
}
};
xmlhttp.open('POST', url, true);
xmlhttp.withCredentials = true;
xmlhttp.setRequestHeader('Content-Type', 'application/json');
xmlhttp.send(payload);
Request:
Notice that Chrome doesn't show any Set-Cookie headers for the request.
Curl of same request:
But if I right-click > copy > cUrl and execute here are the raw headers include Set-Cookie:
curl 'http://satellite:6988/medic/login' -H 'Referer: http://upstream:5988/medic/login?redirect=http%3A%2F%2Fupstream%3A5988%2Fmedic%2F_design%2Fmedic%2F_rewrite%2F%23%2Fmessages%2Fcontact%3A13863168-be61-418a-aea1-1b19b9a5af39' -H 'Origin: http://upstream:5988' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36' -H 'Content-Type: text/plain;charset=UTF-8' --data-binary '{"user":"worker","password":"dSNH1sv2jzdB"}' --compressed -v
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to satellite (127.0.0.1) port 6988 (#0)
> POST /medic/login HTTP/1.1
> Host: satellite:6988
> Accept: */*
> Accept-Encoding: deflate, gzip
> Referer: http://upstream:5988/medic/login?redirect=http%3A%2F%2Fupstream%3A5988%2Fmedic%2F_design%2Fmedic%2F_rewrite%2F%23%2Fmessages%2Fcontact%3A13863168-be61-418a-aea1-1b19b9a5af39
> Origin: http://upstream:5988
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36
> Content-Type: text/plain;charset=UTF-8
> Content-Length: 43
>
* upload completely sent off: 43 out of 43 bytes
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: http://upstream:5988
< Vary: Origin, Accept-Encoding
< Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, HEAD, DELETE
< Access-Control-Allow-Headers: accept, authorization, content-type, origin, referer, x-csrf-token
< Access-Control-Allow-Credentials: true
< Set-Cookie: AuthSession=d29ya2VyOjVDMDdEMjgwOj06hJxoYrYGtUxNa3ImYd1AZ7Vw; Path=/
< Set-Cookie: userCtx=%7B%22name%22%3A%22worker%22%2C%22roles%22%3A%5B%22chv%22%5D%7D; Max-Age=31536000; Path=/; Expires=Thu, 05 Dec 2019 13:28:32 GMT
< Set-Cookie: satelliteServer=; Path=/; Expires=Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 16
< ETag: W/"10-oV4hJxRVSENxc/wX8+mA4/Pe4tA"
< Date: Wed, 05 Dec 2018 13:28:32 GMT
< Connection: keep-alive
<
* Connection #0 to host satellite left intact
{"success":true}
The Chrome console is helpful for understanding problems with CORS headers. Since it writes clear descriptive headers when a request is fully blocked.
For my particular scenario, I used chrome://net-internals/#events which contains cookie related events to understand that the Set-Cookie headers are omiited from developer tools but were being served, received, and properly set.
My issue was that I was setting options.credentials=true when using fetch on requests after the cookies were set instead of the required options.credentials='include'.
I have an ALB with a target group and ECS cluster running PHP API.
I am trying to query the API for a CSV response but I am getting truncated results if the Request is coming through the ALB.
When I SSH into the EC2 instance running the cluster and try to run curl manually (going through the load balancer) the response gets truncated:
curl -sSL -D - 'https://my.domain.com/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' -o /dev/null
I am getting these headers:
HTTP/2 200
date: Wed, 21 Nov 2018 20:25:27 GMT
content-type: text/csv; charset=utf-8
content-length: 173019
server: nginx
content-transfer-encoding: binary
content-description: File Transfer
content-disposition: attachment;filename=export.csv
cache-control: private, must-revalidate
etag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"
strict-transport-security: max-age=2592000; includeSubDomains; preload
content-security-policy-report-only: default-src 'self';
x-frame-options: DENY
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: origin
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
If I try to run the same curl against the container (running locally - not through ALB)
curl -sSL -D - 'http://localhost:32776/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' -o /dev/null
Response:
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/csv; charset=utf-8
Content-Length: 173019
Connection: keep-alive
Content-Transfer-Encoding: binary
Content-Description: File Transfer
content-disposition: attachment;filename=export.csv
Cache-Control: private, must-revalidate
Date: Wed, 21 Nov 2018 20:36:55 GMT
ETag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"
Strict-Transport-Security: max-age=2592000; includeSubDomains; preload
Content-Security-Policy-Report-Only: default-src 'self;
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: origin
When I compare them, there is a difference in the HTTP version. I tried switching to HTTP1 in ALB but still getting the same (or similar) issue: curl: (18) transfer closed with 130451 bytes remaining to read.
Another difference is the Keep-Alive option. I am not sure if this is an attribute I can enable on the ALB.
When I try to return a different response (complex web page/really long) the response goes through ALB without a problem (not truncated). According to the error message when ALB has HTTP/1.1 enabled the Response is truncated every time after 42568 bytes.
Any ideas?
UPDATE
If I leave out the Content-Type header in the response, it doesn't get truncated.
return new Response($content, Response::HTTP_OK, [
# Works without this:
# 'Content-Type' => 'text/csv; charset=utf-8',
'Content-Transfer-Encoding' => 'binary',
'Content-Description' => 'File Transfer',
'Content-Disposition' => "attachment;filename=export.csv",
'Content-Length' => strlen($content),
]);
UPDATE 2
Changing the response Content-Type to be text/html returns the response properly.
So after some joyful debugging, I found this in the Nginx logs from the container:
nginx stderr | 2018/11/22 01:03:59 [warn] 39#39: *65 an upstream response is
buffered to a temporary file /var/tmp/nginx/fastcgi/4/01/0000000014 while reading
upstream, client: 10.1.1.163, server: _, request: "GET /api/export?
token=foobar&start_date=01-01-2015&end_date=01-01-2019 HTTP/1.1", upstream:
"fastcgi://unix:/var/run/php-fpm.sock:", host: "my.domain.com"
Which can basically be solved by baking in these two lines into my nginx config:
client_body_temp_path /tmp 1 2;
fastcgi_temp_path /tmp 1 2;
The question why was this happening only for csv output will remain a mystery.
Thanks for the help!
You should enable keep-alive on your EC2 instances.
You can enable HTTP keep-alive in the web server settings for your EC2
instances.
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
Also double check that the Content-Length header is accurate. An incorrect size here will result in the error you are seeing.
I've a simple API Gateway, that sends the data to an HTTP endpoint (Express/Node).
For testing, I'm using curl, which is great. Sending the curl request without CORS works like a charm, however if I try to mimic CORS in curl, I get a HTTP 500 and have no idea why. These are both requests:
curl -v -H "X-Api-Key: myapikey" -H "Origin: example.com" "https://apigatewayid.execute-api.us-west-2.amazonaws.com/dev/path/prettyParam?anotherParam=1"
* Trying x.x.x.x...
* TCP_NODELAY set
* Connected to apigatewayid.execute-api.us-west-2.amazonaws.com (x.x.x.x) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-west-2.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev/path/prettyParam?anotherParam=1 HTTP/1.1
> Host: apigatewayid.execute-api.us-west-2.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> X-Api-Key: myapikey
> Origin: example.com
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 64
< Connection: keep-alive
< Date: Fri, 21 Jul 2017 00:28:50 GMT
< x-amzn-RequestId: numbers-6dab-11e7-b411-b7f8fd6c0cc3
< Access-Control-Allow-Origin: *
< X-Amzn-Trace-Id: Root=1-5morenumbersletters3e8be5c86a2c72781a0b356
< X-Cache: Miss from cloudfront
< Via: 1.1 numbersletters7a8621aabe6b30d2f5a48.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: numberslettersUk3Bs9dL4KJR4QccPmILA4tJUjO0X_h7cQc9DxA==
<
* Curl_http_done: called premature == 0
* Connection #0 to host apigatewayid.execute-api.us-west-2.amazonaws.com left intact
{"resultDataFromServer":"dataReceived!"}
curl -H "Origin: example.com" -H "X-Api-Key: myapikey" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested-With" -X OPTIONS --verbose "https://apigatewayid.execute-api.us-west-2.amazonaws.com/dev/path/prettyParam?anotherParam=1"
* Trying x.x.x.x...
* TCP_NODELAY set
* Connected to apigatewayid.execute-api.us-west-2.amazonaws.com (x.x.x.x) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-west-2.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> OPTIONS /dev/path/prettyParam?anotherParam=1 HTTP/1.1
> Host: apigatewayid.execute-api.us-west-2.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> X-Api-Key: myapikey
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json
< Content-Length: 36
< Connection: keep-alive
< Date: Fri, 21 Jul 2017 00:29:07 GMT
< x-amzn-RequestId: numbers-6dab-11e7-b411-b7f8fd6c0cc3
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token
< Access-Control-Allow-Methods: GET,OPTIONS
< X-Cache: Miss from cloudfront
< Via: 1.1 numbersletters7a8621aabe6b30d2f5a48.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: numberslettersUk3Bs9dL4KJR4QccPmILA4tJUjO0X_h7cQc9DxA==
<
* Curl_http_done: called premature == 0
* Connection #0 to host apigatewayid.execute-api.us-west-2.amazonaws.com left intact
{"message": "Internal server error"}
I really don't understand what I'm doing wrong. I enabled CORS in the API Gateway, and in Express CORS is enabled also, so not sure what is going on.
#Raul, did you test your API method via API Gateway? Try deploying your API again and test it from the APIGateway itself by providing the URL param. If you get the same {"message": "Internal server error"} there is a problem with the code. Sometimes it might look like a CORS issue, but actually it could be a lambda logic error.
I have created a cURL post with headers in a c++ application:
CURL *curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &bufferdata);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writerResponse);
struct curl_slist *headers=NULL;
char outputmessage[]="";
headers = curl_slist_append(headers, "Accept: application/json");
headers = curl_slist_append(headers, "Content-Type: application/json");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers );
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, cJSONstring.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, cJSONstring.length());
curl_easy_setopt(curl, CURLOPT_VERBOSE,1);
curl_easy_setopt(curl, CURLOPT_HEADER,1);
curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, my_trace);
curl_multi_add_handle(m_telemetryCurlm, curl);
I have created a thread to call multi perform:
void ContentManager::processTelemetry()
{
m_telemetryThreadRunning = true;
while(m_telemetryThreadRunning)
{
static_cast<ADCAnalytics*>(m_adcAnalytics)->performTelemerty();
usleep(5000);
}
}
void ADCAnalytics::performTelemerty()
{
// number of active downloads
int counter = 0;
// preform downloading
curl_multi_perform(m_telemetryCurlm, &counter);
// handle CURL actions (connection errors, download finish, etc.)
checkCURLMessages();
}
but when I check the curl messages I get:
12-10 13:23:00.845: I/ADC(17248): buffer = HTTP/1.1 400 Bad Request
12-10 13:23:00.845: I/ADC(17248): X-TraceUrl: /appstats/details?time=1355145806563&type=json
12-10 13:23:00.845: I/ADC(17248): Content-Type: text/plain
12-10 13:23:00.845: I/ADC(17248): Vary: Accept-Encoding
12-10 13:23:00.845: I/ADC(17248): Date: Mon, 10 Dec 2012 13:23:26 GMT
12-10 13:23:00.845: I/ADC(17248): Server: Google Frontend
12-10 13:23:00.845: I/ADC(17248): Cache-Control: private
12-10 13:23:00.845: I/ADC(17248): Transfer-Encoding: chunked
It seems that the content-type is changed between curl_multi_add_handle and curl_multi_perform. Does anybody has any idea what is wrong here? What can I do to avoid content-type to change?
When adding the following code
do {
while(curl_multi_perform(m_telemetryCurlm, &still_running) ==CURLM_CALL_MULTI_PERFORM);
} while (still_running);
immediately after setting the cURL object. The request is sent to the server and I get
12-10 13:11:48.105: I/ADC(16845): buffer = HTTP/1.1 200 OK
12-10 13:11:48.105: I/ADC(16845): X-TraceUrl: /appstats/details?time=1355145133776&type=json
12-10 13:11:48.105: I/ADC(16845): Content-Type: application/json
12-10 13:11:48.105: I/ADC(16845): Vary: Accept-Encoding
12-10 13:11:48.105: I/ADC(16845): Date: Mon, 10 Dec 2012 13:12:13 GMT
12-10 13:11:48.105: I/ADC(16845): Server: Google Frontend
12-10 13:11:48.105: I/ADC(16845): Cache-Control: private
12-10 13:11:48.105: I/ADC(16845): Transfer-Encoding: chunked
So, I don't believe is something wrong on the server side.
Thanks
Aren't you comparing the Content-Type you SEND with the one you RECEIVE?