my working curl is as follows
curl -ki 'https://host/api/getData' -H 'authorization: APP-AUTH "<token>"' -H 'content-type: application/json'
but doesn't work as follows:
curl -XGET 'https://host/api/getData' -H 'authorization: APP-AUTH "<token>"' -H 'content-type: application/json'
I cannot replicate it with python requests library, any help would be appreciated.
Related
recently i had this error message in my sentry
Invalid HTTP_HOST header: 'attacker.web'. You may need to add 'attacker.web' to ALLOWED_HOSTS.
and i saw the request like this
curl \
-H "Accept: */*" \
-H "Content-Length: " \
-H "Content-Type: " \
-H "Forwarded: for=\"attacker.web:8888\";by=\"attacker.web:9999\"" \
-H "Host: attacker.web" \
-H "User-Agent: Report Runner" \
-H "X-Forwarded-For: " \
-H "X-Forwarded-Host: mysite.com" \
-H "X-Forwarded-Proto: https" \
-H "X-Real-Ip: " \
"https://attacker.web/subpage/"
how do i prevent this kind of request ? and what's the name of the attack ?
i've been config my nginx to
drop curl request
return 444 when the host name doesn't the same with server_name
how to deal with this kind of request ?
I want to download the following pdf via Postman.
I entered the URL to postman and made a GET request:
curl --location --request GET 'https://innovationisrael.org.il/sites/default/files/2020%20High-Tech%20Human%20Capital%20Report%20-%20Hebrew%20Version.pdf'
I reviewed 403 Forbidden response.
I also tried to import the request from the network tab and then to make the request in postman, then I receive a 304 status code with empty body.
The imported request:
curl --location --request GET 'https://innovationisrael.org.il/sites/default/files/2020%20High-Tech%20Human%20Capital%20Report%20-%20Hebrew%20Version.pdf' \
--header 'authority: innovationisrael.org.il' \
--header 'cache-control: max-age=0' \
--header 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'upgrade-insecure-requests: 1' \
--header 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36' \
--header 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
--header 'sec-fetch-site: none' \
--header 'sec-fetch-mode: navigate' \
--header 'sec-fetch-user: ?1' \
--header 'sec-fetch-dest: document' \
--header 'accept-language: he-IL,he;q=0.9,en-US;q=0.8,en;q=0.7' \
--header 'cookie: SSESSf011814d55d1325f79eb4fc5bf454a5c=UMRu4tTqN_hBG13cGNW8ZvUrpWCThW1HvjFrfzSVi_I' \
--header 'if-none-match: "38e2fa-5c08b608caec4"' \
--header 'if-modified-since: Thu, 22 Apr 2021 08:22:16 GMT'
I was able to replicate your problem (I can access the file in my browser, but when I do a GET request in Postman, I received a 403 Forbidden response).
It seems that the website is doing some basic parsing of the User-Agent header in order to determine if the traffic is coming from a real browser, or from some other source. I went into my Postman headers, and modified the User-Agent to the same value that Chrome would send, and now I get response 200 OK with PDF in body.
I'm trying to use libcurl in C++ to send requests to a url. When I set up the request in the command line with curl, it seems to work fine:
curl -vvv -X POST -H "Authorization: <api key here>" -H "Content-Type:application/json" "<host>" --data-binary '<json data here>'
The response starts something like this:
> POST <host> HTTP/1.1
> Host: <host>
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: <api_key>
> Content-Type:application/json
> Content-Length: 80
So i can see the authorization is being sent properly.
When I try to do a similar thing in C++, using the libcurl C library, however, I don't notice the ">" in front of the request headers:
Code:
struct curl_slist *chunk = NULL;
chunk = curl_slist_append(chunk, "Authorization: <api_key>");
chunk = curl_slist_append(chunk, "Content-Type:application/json");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
curl_easy_setopt(curl, CURLOPT_URL, "<host>");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "<json>");
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
Response:
> POST <host> HTTP/1.1
Host: <host>
Accept: */*
Authentication: <api_key>
Content-Type:application/json
Content-Length: 97
So I'm not even sure if the headers are even properly being processed or received by the host.
Any ideas?
I get the following response:
{
"message": "No authorization header given",
"code": 401
}
Incorrect Header. Should be Authorization, not Authentication.
In your command line verbose output the header is named "Authorization: ". In your libcurl vebose output its "Authentication: ". Authorization != Authentication?
Verbose output:
Its only a different output format for verbose between command line and libcurl. The headers are sent. Same output format is used for example by php curl. Only first line has ">" and then all following headers has no ">". But they are all submited.
PHP curl verbose example output:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 80 (#0)
> GET /XXX/api.php HTTP/1.1
Host: localhost
Accept: */*
Authorization: XXX
Content-Type: application/json
< HTTP/1.1 200 OK
< Date: Mon, 31 Dec 2018 20:12:51 GMT
< Server: Apache/2.4.34 (Win32) OpenSSL/1.1.0i PHP/7.2.10
< X-Powered-By: PHP/7.2.10
< Content-Length: 2390
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host localhost left intact
I have an ALB with a target group and ECS cluster running PHP API.
I am trying to query the API for a CSV response but I am getting truncated results if the Request is coming through the ALB.
When I SSH into the EC2 instance running the cluster and try to run curl manually (going through the load balancer) the response gets truncated:
curl -sSL -D - 'https://my.domain.com/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' -o /dev/null
I am getting these headers:
HTTP/2 200
date: Wed, 21 Nov 2018 20:25:27 GMT
content-type: text/csv; charset=utf-8
content-length: 173019
server: nginx
content-transfer-encoding: binary
content-description: File Transfer
content-disposition: attachment;filename=export.csv
cache-control: private, must-revalidate
etag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"
strict-transport-security: max-age=2592000; includeSubDomains; preload
content-security-policy-report-only: default-src 'self';
x-frame-options: DENY
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: origin
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
If I try to run the same curl against the container (running locally - not through ALB)
curl -sSL -D - 'http://localhost:32776/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' -o /dev/null
Response:
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/csv; charset=utf-8
Content-Length: 173019
Connection: keep-alive
Content-Transfer-Encoding: binary
Content-Description: File Transfer
content-disposition: attachment;filename=export.csv
Cache-Control: private, must-revalidate
Date: Wed, 21 Nov 2018 20:36:55 GMT
ETag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"
Strict-Transport-Security: max-age=2592000; includeSubDomains; preload
Content-Security-Policy-Report-Only: default-src 'self;
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: origin
When I compare them, there is a difference in the HTTP version. I tried switching to HTTP1 in ALB but still getting the same (or similar) issue: curl: (18) transfer closed with 130451 bytes remaining to read.
Another difference is the Keep-Alive option. I am not sure if this is an attribute I can enable on the ALB.
When I try to return a different response (complex web page/really long) the response goes through ALB without a problem (not truncated). According to the error message when ALB has HTTP/1.1 enabled the Response is truncated every time after 42568 bytes.
Any ideas?
UPDATE
If I leave out the Content-Type header in the response, it doesn't get truncated.
return new Response($content, Response::HTTP_OK, [
# Works without this:
# 'Content-Type' => 'text/csv; charset=utf-8',
'Content-Transfer-Encoding' => 'binary',
'Content-Description' => 'File Transfer',
'Content-Disposition' => "attachment;filename=export.csv",
'Content-Length' => strlen($content),
]);
UPDATE 2
Changing the response Content-Type to be text/html returns the response properly.
So after some joyful debugging, I found this in the Nginx logs from the container:
nginx stderr | 2018/11/22 01:03:59 [warn] 39#39: *65 an upstream response is
buffered to a temporary file /var/tmp/nginx/fastcgi/4/01/0000000014 while reading
upstream, client: 10.1.1.163, server: _, request: "GET /api/export?
token=foobar&start_date=01-01-2015&end_date=01-01-2019 HTTP/1.1", upstream:
"fastcgi://unix:/var/run/php-fpm.sock:", host: "my.domain.com"
Which can basically be solved by baking in these two lines into my nginx config:
client_body_temp_path /tmp 1 2;
fastcgi_temp_path /tmp 1 2;
The question why was this happening only for csv output will remain a mystery.
Thanks for the help!
You should enable keep-alive on your EC2 instances.
You can enable HTTP keep-alive in the web server settings for your EC2
instances.
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
Also double check that the Content-Length header is accurate. An incorrect size here will result in the error you are seeing.
Im trying to assign a role to a user in wso2 using admin service. Below is request stored in setuser.xml. Im using wso2 apim manager 1.10
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:ser="http://service.ws.um.carbon.wso2.org" xmlns:xsd="http://common.mgt.user.carbon.wso2.org/xsd">
<soap:Header/>
<soap:Body>
<ser:setUserClaimValues>
<!--Optional:-->
<ser:userName>test123</ser:userName>
<ser:claims>
<!--Optional:-->
<xsd:claimURI>http://wso2.org/claims/role</xsd:claimURI>
<!--Optional:-->
<xsd:value>Internal/publisher</xsd:value>
</ser:claims>
</ser:setUserClaimValues>
Above is my post request.Im getting 202 status after executing curl, but role is not added. Could you tell me where im doing wrong.
curl -i -k --header "Content-Type: application/soap+xml" -H "Authorization:
Basic xxxxxxxxxx" -X POST
https://xxxxx:9443/services/RemoteUserStoreManagerService -d #setuser.xml
HTTP/1.1 202 Accepted
Set-Cookie: JSESSIONID=83C74EE8FC35CAE7D015FA32803440BE; Path=/; Secure;
HttpOnly
Content-Type: text/xml;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 12 Jul 2017 15:03:08 GMT
Server: WSO2 Carbon Server
You are missing SOAPAction header
curl -k -H "Content-Type: application/soap+xml;charset=UTF-8;" -H "SOAPAction:urn:setUserClaimValues" --basic -u "admin:admin" --data #setuser.xml https://localhost:9443/services/RemoteUserStoreManagerService