I have an ALB with a target group and ECS cluster running PHP API.
I am trying to query the API for a CSV response but I am getting truncated results if the Request is coming through the ALB.
When I SSH into the EC2 instance running the cluster and try to run curl manually (going through the load balancer) the response gets truncated:
curl -sSL -D - 'https://my.domain.com/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' -o /dev/null
I am getting these headers:
HTTP/2 200
date: Wed, 21 Nov 2018 20:25:27 GMT
content-type: text/csv; charset=utf-8
content-length: 173019
server: nginx
content-transfer-encoding: binary
content-description: File Transfer
content-disposition: attachment;filename=export.csv
cache-control: private, must-revalidate
etag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"
strict-transport-security: max-age=2592000; includeSubDomains; preload
content-security-policy-report-only: default-src 'self';
x-frame-options: DENY
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: origin
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
If I try to run the same curl against the container (running locally - not through ALB)
curl -sSL -D - 'http://localhost:32776/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' -o /dev/null
Response:
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/csv; charset=utf-8
Content-Length: 173019
Connection: keep-alive
Content-Transfer-Encoding: binary
Content-Description: File Transfer
content-disposition: attachment;filename=export.csv
Cache-Control: private, must-revalidate
Date: Wed, 21 Nov 2018 20:36:55 GMT
ETag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"
Strict-Transport-Security: max-age=2592000; includeSubDomains; preload
Content-Security-Policy-Report-Only: default-src 'self;
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: origin
When I compare them, there is a difference in the HTTP version. I tried switching to HTTP1 in ALB but still getting the same (or similar) issue: curl: (18) transfer closed with 130451 bytes remaining to read.
Another difference is the Keep-Alive option. I am not sure if this is an attribute I can enable on the ALB.
When I try to return a different response (complex web page/really long) the response goes through ALB without a problem (not truncated). According to the error message when ALB has HTTP/1.1 enabled the Response is truncated every time after 42568 bytes.
Any ideas?
UPDATE
If I leave out the Content-Type header in the response, it doesn't get truncated.
return new Response($content, Response::HTTP_OK, [
# Works without this:
# 'Content-Type' => 'text/csv; charset=utf-8',
'Content-Transfer-Encoding' => 'binary',
'Content-Description' => 'File Transfer',
'Content-Disposition' => "attachment;filename=export.csv",
'Content-Length' => strlen($content),
]);
UPDATE 2
Changing the response Content-Type to be text/html returns the response properly.
So after some joyful debugging, I found this in the Nginx logs from the container:
nginx stderr | 2018/11/22 01:03:59 [warn] 39#39: *65 an upstream response is
buffered to a temporary file /var/tmp/nginx/fastcgi/4/01/0000000014 while reading
upstream, client: 10.1.1.163, server: _, request: "GET /api/export?
token=foobar&start_date=01-01-2015&end_date=01-01-2019 HTTP/1.1", upstream:
"fastcgi://unix:/var/run/php-fpm.sock:", host: "my.domain.com"
Which can basically be solved by baking in these two lines into my nginx config:
client_body_temp_path /tmp 1 2;
fastcgi_temp_path /tmp 1 2;
The question why was this happening only for csv output will remain a mystery.
Thanks for the help!
You should enable keep-alive on your EC2 instances.
You can enable HTTP keep-alive in the web server settings for your EC2
instances.
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
Also double check that the Content-Length header is accurate. An incorrect size here will result in the error you are seeing.
Related
I have one normal Wordpress website on which I am trying to block requests to wp-json. I am aware that such requests go via the core of the Wordpress. My request looks like this:
[root#SV-CentOS-01 ~]# curl -i https://www.website.com/wp-json/wp/v2/users/1
When I add RewriteRule ^wp-json.*$ - [L,R=404] on top of my htaccess I get 404 server response and the API returns me the users of my website. Is there actually a way to achieve what I want via .htaccess or we need to make it the Wordpress way?
Example:
[root#SV-CentOS-01 ~]# curl -i https://www.website.com/wp-json/wp/v2/users/1
HTTP/1.1 404 Not Found
Date: Mon, 20 Sep 2021 14:14:13 GMT
Server: Apache
Vary: Accept-Encoding,Cookie,Origin
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Robots-Tag: noindex
Link: <https://www.website.com/wp-json/>; rel="https://api.w.org/"
X-Content-Type-Options: nosniff
Access-Control-Expose-Headers: X-WP-Total, X-WP-TotalPages, Link
Access-Control-Allow-Headers: Authorization, X-WP-Nonce, Content-Disposition, Content-MD5, Content-Type
Allow: GET
Set-Cookie: PHPSESSID=5c07eaa455457ca0ef4b358d016c3b8d; path=/
Upgrade: h2,h2c
Connection: Upgrade
Transfer-Encoding: chunked
Content-Type: application/json; charset=UTF-8
{"id":1,"name":"User One","url":"","description":"","link":"https:\/\/www.website.com\/author\/admin\/","slug":"admin","meta":[],"_links":{"self":[{"href":"https:\/\/www.website.com\/wp-json\/wp\/v2\/users\/1"}],"collection":[{"href":"https:\/\/www.website.com\/wp-json\/wp\/v2\/users"}]}}[root#SV-CentOS-01 ~]#
vuejs is running inside a docker container served by:
CMD [ "http-server", "dist" ]
when using axios inside Vue.js mounted() to do a GET request against a flask api it shows "blocked" in the network tab, accessing other REST-API's works fine.
testing with curl (localhost:6000 being the flask server):
- curl is running from my real host and conneting to the container
curl -H "Origin: http://localhost:5000"
-H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested With"
-X OPTIONS --verbose http://localhost:6000/todo/api/v1.0/wheel/40
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 6000 (#0)
> OPTIONS /todo/api/v1.0/wheel/40 HTTP/1.1
> Host: localhost:6000
> User-Agent: curl/7.58.0
> Accept: */*
> Origin: http://localhost:5000
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Allow: OPTIONS, GET, HEAD
< Access-Control-Allow-Origin: http://localhost:5000
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT
< Vary: Origin
< Content-Length: 0
< Server: Werkzeug/1.0.0 Python/3.8.2
< Date: Sun, 15 Mar 2020 15:43:44 GMT
<
* Closing connection 0
from what ive read so far, for example here: 1, for a unauthorized GET request the headers look ok.
This one gets the real data:
curl -H "Origin: http://l:5000" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested-With" -v http://localhost:6000/todo/api/v1.0/wheel/40
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 6000 (#0)
> GET /todo/api/v1.0/wheel/40 HTTP/1.1
> Host: localhost:6000
> User-Agent: curl/7.58.0
> Accept: */*
> Origin: http://l:5000
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: application/json
< Content-Length: 39
< Access-Control-Allow-Origin: http://l:5000
< Vary: Origin
< Server: Werkzeug/1.0.0 Python/3.8.2
< Date: Sun, 15 Mar 2020 15:56:43 GMT
<
{"result":{"model 1":0,"model 2":150}}
* Closing connection 0
Manipulating -H "Origin..." to this:
-H "Origin: http://l:5000"
also shows a normal reply. Isn't that a good test?
As it turns out mozilla allows certain ports for certain protocols as shown here:
https://developer.mozilla.org/en-US/docs/Mozilla/Mozilla_Port_Blocking
6000 is the "x11" port and on that list - As port for x11 and not to be used for xhr. So every port not on that list should do the trick.
reason:
Cert issued a Vulnerability Note VU#476267 for a "Cross-Protocol" scripting attack, known as the HTML Form Protocol Attack
In your app's __init__.py file add these lines and you'll be good to go.
$ pip install flask-cors
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
Read more about CORS here-- MDN CORS
I'm using WhiteNoise to serve static files from a Django app running under gunicorn. For some reason, the Cache-Control and Access-Control-Allow-Origin headers returned by the gunicorn backend are not being passed back to the client through the nginx proxy.
Here's what the response looks like for a sample request to the gunicorn backend:
% curl -I -H "host: www.myhost.com" -H "X-Forwarded-Proto: https" http://localhost:8000/static/img/sample-image.1bca02e3206a.jpg
HTTP/1.1 200 OK
Server: gunicorn/19.8.1
Date: Mon, 02 Jul 2018 14:20:42 GMT
Connection: close
Content-Length: 76640
Last-Modified: Mon, 18 Jun 2018 09:04:15 GMT
Access-Control-Allow-Origin: *
Cache-Control: max-age=315360000, public, immutable
Content-Type: image/jpeg
When I make a request for the same file via the nginx server, the two headers are missing.
% curl -I -H "Host: www.myhost.com" -k https://my.server.com/static/img/sample-image.1bca02e3206a.jpg
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 02 Jul 2018 14:09:25 GMT
Content-Type: image/jpeg
Content-Length: 76640
Last-Modified: Mon, 18 Jun 2018 09:04:15 GMT
Connection: keep-alive
ETag: "5b27758f-12b60"
Accept-Ranges: bytes
My nginx config is pretty much what is documented in the gunicorn deployment docs, i.e. I haven't enabled nginx caching (nginx -T | grep -i cache is empty) or done anything else I would think is out of the ordinary.
What am I missing?
The problem is that you have
location / {
try_files $uri #proxy_to_app;
}
directive in nginx config, so nginx just serves files himself and gunicorn doesn't even knows about it, and of course can't add headers.
It turns out I had forgotten the root directive I had configured many months ago, which was now picking up the static files. My error was in assuming that since I hadn't configured a location /static directive, nginx would be proxying all requests to the backend.
The solution for me was to remove the $uri reference from the try_files directive:
location / {
try_files /dev/null #proxy_to_app;
}
Alternatively, I could have simply put the contents of the #proxy_to_app location block directly inside the location / block.
Thanks to Alexandr Tatarinov for the suggestion in the comments.
I know about the SOAP -based webservices that SOAP messages, which are XML in turn, are transferred on the network, from client to the server. But what kind of data is transferred in case of RESTful webservice ?
from Wikipedia https://en.wikipedia.org/wiki/Representational_state_transfer
RESTful systems typically, but not always, communicate over the
Hypertext Transfer Protocol with the same HTTP verbs (GET, POST, PUT,
DELETE, etc.)
You can try yourself easily with curl or Fiddler. For example GitHub API is nice to experiment with.
Send this with Fiddler:
GET https://api.github.com/users/octocat HTTP/1.1
Host: api.github.com
User-Agent: Fiddler
and you will get this response:
HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 10 Jul 2015 10:23:10 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1155
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 58
X-RateLimit-Reset: 1436527371
Cache-Control: public, max-age=60, s-maxage=60
Last-Modified: Mon, 06 Jul 2015 23:59:25 GMT
ETag: "d811d5844be3eaf9ab1f60dd36198aa9"
Vary: Accept
X-GitHub-Media-Type: github.v3; format=json
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 3EAD7342:6BF1:E180441:559F9D0D
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: bd82876e9bf04990f289ba22f246ee9b
{"login":"octocat","id":583231,"avatar_url":"https://avatars.githubusercontent.com/u/583231?v=3","gravatar_id":"","url":"https://api.github.com/users/octocat","html_url":"https://github.com/octocat","followers_url":"https://api.github.com/users/octocat/followers","following_url":"https://api.github.com/users/octocat/following{/other_user}","gists_url":"https://api.github.com/users/octocat/gists{/gist_id}","starred_url":"https://api.github.com/users/octocat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/octocat/subscriptions","organizations_url":"https://api.github.com/users/octocat/orgs","repos_url":"https://api.github.com/users/octocat/repos","events_url":"https://api.github.com/users/octocat/events{/privacy}","received_events_url":"https://api.github.com/users/octocat/received_events","type":"User","site_admin":false,"name":"The Octocat","company":"GitHub","blog":"http://www.github.com/blog","location":"San Francisco","email":"octocat#github.com","hireable":false,"bio":null,"public_repos":5,"public_gists":8,"followers":1054,"following":6,"created_at":"2011-01-25T18:44:36Z","updated_at":"2015-07-06T23:59:25Z"}
The following link answered my query. Now I know that the data sent in case of RESTful services is 'raw http' data.
http://rest.elkstein.org/2008/02/how-simple-is-rest.html
I'm trying to send an HTTP GET request in C++ using sockets, and I'm getting in response a 301 Moved permently, but to the same address I've asked for!
Here is my GET request :
GET /watch?v=1cQh1ccqu8M HTTP/1.1
Host: www.youtube.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language:en-US;q=0.6,en;q=0.4
Connection: keep-alive
All the \r\n are perfectly in place, because this GET request used to work for me not long ago, and I have not touched it since...
The response I'm getting from youtube :
HTTP/1.1 301 Moved Permanently
Date: Mon, 08 Dec 2014 11:04:10 GMT
Server: gwiseguy/2.0
Content-Type: text/html; charset=utf-8
Expires: Tue, 27 Apr 1971 19:44:06 EST
Location: https://www.youtube.com/watch?v=1cQh1ccqu8M
X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
Cache-Control: no-cache
Content-Length: 0
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic,p=0.002
According to the page they tell me to check http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en, it says I need to add some kind of cookies now?
I've allways send this request without sending any cookies, so I am a bit confused...
me: Was your original request really done with https and not plain http...
#Amit: No, I was connecting to `www.youtube.com', then I've sent the GET request
Then you should look more closely at the redirect:
Location: https://www.youtube.com/watch?v=1cQh1ccqu8M
As you can see, this does redirect you to the same host, same page, but different protocol: you must use https instead of http.