Unable to decode CORS gzipped content from EC2 via Cloudfront - amazon-web-services

This is probablly related to Serving gzipped CSS and JavaScript from Amazon CloudFront via S3, but although I followed the doc I still can get this situation to work :
An S3 bucket contains files that I have gzipped (with grunt-contrib-compress, if that's of any relevance.)
My bucket has a CORS configuration :
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*.myorigin.net</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
The bucket is served via Cloudfront.
My Cloudfront has CORS configured to :
Allow GET,HEAD,OPTIONS,PUT,etc...
on HTTPS
to cache OPTIONS requests
to whitelist Access-Control-Request-Headers, Access-Control-Request-Method and Origin
Using curl, I'm able to get my resources, with the right header to request gzip, and I get a gzipped version
curl --insecure https:/whatever.cloudfront.net/.../foo.js --silent -H "Accept-Encoding:gzip,deflate,sdch" -I
HTTP/1.1 200 OK
Content-Type: application/javascript
Content-Length: 33990
Connection: keep-alive
Date: Wed, 06 May 2015 17:12:53 GMT
Content-Encoding: gzip
Last-Modified: Wed, 06 May 2015 16:36:40 GMT
ETag: "7a92919df6117827de9474851afe06c0"
Accept-Ranges: bytes
Server: AmazonS3
Age: 70
X-Cache: Hit from cloudfront
Via: 1.1 (redacted).cloudfront.net (CloudFront)
X-Amz-Cf-Id: iaghGGDg3f3l4njz7mpXeOzqIS5OCR5kaehk4Td4-Bxiv2KtljHVlQ==
I'am able to get the content decoded by adding the --compressed flag, so the files seem to be correct.
If I now do a CORS request, with curl, again, I get the proper encoding :
curl --insecure https://.../foo.js --silent -H "Accept-Encoding: gzip,deflate,sdch" -I -H "Origin: myorigin.net"
HTTP/1.1 200 OK
Content-Type: application/javascript
Content-Length: 33990
Connection: keep-alive
Date: Wed, 06 May 2015 17:16:06 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, HEAD
Access-Control-Max-Age: 3000
Content-Encoding: gzip
Last-Modified: Wed, 06 May 2015 16:36:40 GMT
ETag: "7a92919df6117827de9474851afe06c0"
Accept-Ranges: bytes
Server: AmazonS3
Vary: Origin,Access-Control-Request-Headers,Access-Control-Request-Method
X-Cache: Miss from cloudfront
Via: 1.1 redacted.cloudfront.net (CloudFront)
X-Amz-Cf-Id: kaUV4S9wiCGzXp8N_Gg2LhGK8uYw1-qRtSw6w_Ry4V8oHZiLRvWULA==
Still, for some reason, when my browser does the request... there is no "Content-Encoding", and so the content is not uncompressed, and since I'm loading it with requirejs, I get a bad error.
Request :
GET /resources-gz/foo.js HTTP/1.1
Host: d28p9e2ugw1o8w.cloudfront.net
Connection: keep-alive
Cache-Control: no-cache
Accept: */*
Pragma: no-cache
User-Agent: Mozilla/5.0 (X11; Linux i686 (x86_64)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Referer: http://localhost:3000/start
Accept-Encoding: gzip,deflate,sdch
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4,de;q=0.2
Response:
HTTP/1.1 200 OK
Content-Type: application/javascript
Content-Length: 33990
Connection: keep-alive
Date: Wed, 06 May 2015 16:31:27 GMT
Last-Modified: Wed, 06 May 2015 14:36:32 GMT
ETag: "7a92919df6117827de9474851afe06c0"
Accept-Ranges: bytes
Server: AmazonS3
Age: 1261
X-Cache: Hit from cloudfront
Via: 1.1 eazeaze.cloudfront.net (CloudFront)
X-Amz-Cf-Id: IjpNd37e-c38Moz3HQJ940KNTXfp1NN7O4enQJUYPd6Aet5Egw8XPg==
I'm obviously missing something...

Yes, you're missing something.
Read the response headers of the last response.
Last-Modified: Wed, 06 May 2015 14:36:32 GMT
ETag: "7a92919df6117827de9474851afe06c0"
Age: 1261
X-Cache: Hit from cloudfront
This is a CloudFront cache hit, and CloudFront fetched this object from S3 20+ minutes before you requested it ... and this version of the object had last been modified in S3 over two hours earlier... so, you're getting a copy that probably dates from before you "fixed" whatever you recently fixed.
When the etag of an object doesn't change, that means the body has not changed... but it doesn't imply anything about the response headers.
The easiest way to test this is to put a new file in S3 with a name you haven't used before... gzipped, of course, then try downloading it again.

Related

.htaccess block wp-json

I have one normal Wordpress website on which I am trying to block requests to wp-json. I am aware that such requests go via the core of the Wordpress. My request looks like this:
[root#SV-CentOS-01 ~]# curl -i https://www.website.com/wp-json/wp/v2/users/1
When I add RewriteRule ^wp-json.*$ - [L,R=404] on top of my htaccess I get 404 server response and the API returns me the users of my website. Is there actually a way to achieve what I want via .htaccess or we need to make it the Wordpress way?
Example:
[root#SV-CentOS-01 ~]# curl -i https://www.website.com/wp-json/wp/v2/users/1
HTTP/1.1 404 Not Found
Date: Mon, 20 Sep 2021 14:14:13 GMT
Server: Apache
Vary: Accept-Encoding,Cookie,Origin
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Robots-Tag: noindex
Link: <https://www.website.com/wp-json/>; rel="https://api.w.org/"
X-Content-Type-Options: nosniff
Access-Control-Expose-Headers: X-WP-Total, X-WP-TotalPages, Link
Access-Control-Allow-Headers: Authorization, X-WP-Nonce, Content-Disposition, Content-MD5, Content-Type
Allow: GET
Set-Cookie: PHPSESSID=5c07eaa455457ca0ef4b358d016c3b8d; path=/
Upgrade: h2,h2c
Connection: Upgrade
Transfer-Encoding: chunked
Content-Type: application/json; charset=UTF-8
{"id":1,"name":"User One","url":"","description":"","link":"https:\/\/www.website.com\/author\/admin\/","slug":"admin","meta":[],"_links":{"self":[{"href":"https:\/\/www.website.com\/wp-json\/wp\/v2\/users\/1"}],"collection":[{"href":"https:\/\/www.website.com\/wp-json\/wp\/v2\/users"}]}}[root#SV-CentOS-01 ~]#

Can't adjust buffer to fit data

I am trying to make an HTTP request the with EtherCard library, then get the full response. Using the code from the examples, I'm only able to capture the headers, which are then abruptly cut off. The issue seems to be that I can't make the buffer big enough to store the data, but the data, hence why it's cut off. But it's only 292 bytes.
Here is another question I asked trying to understand what the example code was doing: What is happening in this C/Arduino code?
Here is the data I'm trying to GET: http://jsonplaceholder.typicode.com/posts/1
String response;
byte Ethernet::buffer[800]; // if i raise this to 1000, response will be blank
static void response_handler (byte status, word off, word len) {
Serial.println("Response:");
Ethernet::buffer[off + 400] = 0; // if i raise 400 much higher, response will be blank
response = String((char*) Ethernet::buffer + off);
Serial.println(response);
}
See the comments above for what I've attempted.
Here is the output from the code above:
Response:
HTTP/1.1 404 Not Found
Date: Fri, 20 Jan 2017 12:15:19 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 2
Connection: close
Set-Cookie: __cfduid=d9714bd94284b999ceb0e87bc91705d501484914519; expires=Sat, 20-Jan-18 12:15:19 GMT; path=/; domain=.typicode.com; HttpOnly
X-Powered-By: Express
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no
As you can see, it's not the complete data, only some of the headers.
There are several problems here:
1) You get a HTTP 404 response, which means the resource was not found on the server. So you need to check your request.
2) You are cutting off the string at pos 400:
Ethernet::buffer[off + 400] = 0; // if i raise 400 much higher, response will be blank
That's why it stops after Cache-Control: no, which is exactly 400 bytes (byte 0-399).
You probably want Ethernet::buffer[off + len] = 0;, but you also need to check if that is not out of bounds (i.e. larger than your buffer size - that's probably why you get a 'blank' response).
For example, a 404 response from that server looks like this:
HTTP/1.1 404 Not Found
Date: Mon, 23 Jan 2017 07:00:00 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 2
Connection: keep-alive
x-powered-by: Express
Vary: Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
x-content-type-options: nosniff
Etag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
Via: 1.1 vegur
CF-Cache-Status: MISS
Server: cloudflare-nginx
CF-RAY: 32595301c275445d-xxx
{}
and the 200 response headers (from a browser):
HTTP/1.1 200 OK
Date: Mon, 23 Jan 2017 07:00:00 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
x-powered-by: Express
Vary: Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: public, max-age=14400
Pragma: no-cache
Expires: Mon, 23 Jan 2017 10:59:01 GMT
x-content-type-options: nosniff
Etag: W/"124-yv65LoT2uMHrpn06wNpAcQ"
Via: 1.1 vegur
CF-Cache-Status: HIT
Server: cloudflare-nginx
CF-RAY: 32595c4ff39b445d-xxx
Content-Encoding: gzip
So your buffer needs to be big enough to hold both the response headers and the data.
3) In the 200 response we see 2 things: the transfer is chunked, and gzipped (but the latter only happens when there is a Accept-Encoding: gzip header in the request.
The easiest way to handle this is to send a HTTP/1.0 request instead of HTTP/1.1 (chunked transfer and gzip are not allowed/available in HTTP/1.0).

Cross-Origin Request Blocked in Firefox only! [OPTIONS : 403 Forbidden]

It's a CORS problem again
[and I don't have much knowledge about CORS].
My environment :
I have JSON file stored in Amazon S3 server, and reading this JSON file from my web application.
My issue :
I am getting this in FireFox only
Cross-Origin Request Blocked: The Same Origin Policy disallows reading
the remote resource at server. (Reason: CORS header
‘Access-Control-Allow-Origin’ missing)
Note : I have configured CORSRule in the server [Amazon S3]; after that, Chrome and other browsers work properly, but Firefox gives this error.
In Firefox Network tab [inspect element]:
Status : 403
Method : OPTIONS
Cause : xhr
Type : xml
Request Headers :
Host: myWebApp.net
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0
Accept: text/plain, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
X-Requested-With: XMLHttpRequest
Referer: https://myWebApp.net/page.jsp
Connection: keep-alive
Response Headers :
Content-Length: 0
Date: Thu, 24 Nov 2016 07:27:30 GMT
Location: https://webApp.s3.amazonaws.com/file.json
Server: Apache-Coyote/1.1
I have looked around but didn't find any causes.
So I need your help.
As it works properly in the Chrome browser, the issue is in Firefox only.
Please describe step by step if possible [I am new to these things].
Opps!
Finally got solution.
The issue was in Amazon S3's CORSConfiguration
I don't know much about it, but now its allowing Cross-Origin from all browsers.
I have changes CORSConfiguration as :
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>https://myWebApp.net</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
<ExposeHeader>x-amz-request-id</ExposeHeader>
<ExposeHeader>x-amz-id-2</ExposeHeader>
</CORSRule>
</CORSConfiguration>
by this rules, browsers getting proper Request Headers and Response Headers.
Request Headers :
Host: myWebApp.s3.amazonaws.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: GET
Access-Control-Request-Headers: x-requested-with
origin: https://myWebApp.net
Connection: keep-alive
Cache-Control: max-age=0
Response Headers :
Access-Control-Allow-Headers: x-requested-with
Access-Control-Allow-Methods: GET, POST, PUT
Access-Control-Allow-Origin: https://myWebApp.net
Access-Control-Max-Age: 3000
Content-Length: 0
Date: Thu, 24 Nov 2016 13:21:17 GMT
Server: AmazonS3
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
access-control-allow-credentials: true
access-control-expose-headers: x-amz-server-side-encryption, x-amz-request-id, x-amz-id-2
x-amz-id-2: emYYMILNIdkCejpjuDXz4Haks87asdhj/7JL5AASt/8eIwKdgO1Gb/AzGRg7SU/GH55IVopScg=
x-amz-request-id: 307572CDFF39F443

what kind of data is transferred on the network in case of RESTful webservice

I know about the SOAP -based webservices that SOAP messages, which are XML in turn, are transferred on the network, from client to the server. But what kind of data is transferred in case of RESTful webservice ?
from Wikipedia https://en.wikipedia.org/wiki/Representational_state_transfer
RESTful systems typically, but not always, communicate over the
Hypertext Transfer Protocol with the same HTTP verbs (GET, POST, PUT,
DELETE, etc.)
You can try yourself easily with curl or Fiddler. For example GitHub API is nice to experiment with.
Send this with Fiddler:
GET https://api.github.com/users/octocat HTTP/1.1
Host: api.github.com
User-Agent: Fiddler
and you will get this response:
HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 10 Jul 2015 10:23:10 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1155
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 58
X-RateLimit-Reset: 1436527371
Cache-Control: public, max-age=60, s-maxage=60
Last-Modified: Mon, 06 Jul 2015 23:59:25 GMT
ETag: "d811d5844be3eaf9ab1f60dd36198aa9"
Vary: Accept
X-GitHub-Media-Type: github.v3; format=json
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 3EAD7342:6BF1:E180441:559F9D0D
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: bd82876e9bf04990f289ba22f246ee9b
{"login":"octocat","id":583231,"avatar_url":"https://avatars.githubusercontent.com/u/583231?v=3","gravatar_id":"","url":"https://api.github.com/users/octocat","html_url":"https://github.com/octocat","followers_url":"https://api.github.com/users/octocat/followers","following_url":"https://api.github.com/users/octocat/following{/other_user}","gists_url":"https://api.github.com/users/octocat/gists{/gist_id}","starred_url":"https://api.github.com/users/octocat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/octocat/subscriptions","organizations_url":"https://api.github.com/users/octocat/orgs","repos_url":"https://api.github.com/users/octocat/repos","events_url":"https://api.github.com/users/octocat/events{/privacy}","received_events_url":"https://api.github.com/users/octocat/received_events","type":"User","site_admin":false,"name":"The Octocat","company":"GitHub","blog":"http://www.github.com/blog","location":"San Francisco","email":"octocat#github.com","hireable":false,"bio":null,"public_repos":5,"public_gists":8,"followers":1054,"following":6,"created_at":"2011-01-25T18:44:36Z","updated_at":"2015-07-06T23:59:25Z"}
The following link answered my query. Now I know that the data sent in case of RESTful services is 'raw http' data.
http://rest.elkstein.org/2008/02/how-simple-is-rest.html

Stop watching google push notifications

I am using the Google Reports API to watch for changes to resources, such as a user's Google Calendar activity events. To achieve this, a channel (or watchpoint) was created by following the steps here: https://developers.google.com/admin-sdk/reports/v1/reference/activities/watch
Now, I would like to stop watching a resource before the channel expires. I sent a POST request along with a access token in the header and the body contains the channel id and the resource id (https://developers.google.com/admin-sdk/reports/v1/reference/channels/stop). However I keep getting a 404 Not Found. Am I doing something wrong here?
All other requests using the same access token are working fine.
My request/response looks like this:
POST /admin/reports/v1/channels/stop HTTP/1.1
Host: www.googleapis.com
Content-length: 97
Content-type: application/json
Authorization: Bearer ya29.rAExOpnO_gatfyJvKWEVt8OsQ-LyaCyN3UUFjYzm0-3ExEBZ9an7WWfdDLqJspChQaiiIQ
{
"id": "5cfc250b-2faf-4f86-91b3-398326c6b4fb",
"resourceId": "A_HZ7mQy0Zpd6-TkQjr3aQlWd94"
}
HTTP/1.1 404 Not Found
Content-length: 9
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Vary: Origin,X-Origin
Server: GSE
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Fri, 10 Jul 2015 17:10:40 GMT
X-frame-options: SAMEORIGIN
Content-type: text/html; charset=UTF-8
Not Found
Turns out the request URL specified in the documentation is wrong.
The correct one is
https://www.googleapis.com/admin/reports_v1/channels/stop
Issue reported here: https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3914