I am attempting to use S3's server-side encryption for customer keys. I created a bucket which allows anonymous users to upload objects, and then attempted to upload an object like so:
$ http -v PUT 'https://BUCKETNAME.s3.amazonaws.com/test.txt' \
"x-amz-server-side-encryption-customer-algorithm:AES256" \
"x-amz-server-side-encryption-customer-key:BASE64DKEY" \
"x-amz-server-side-encryption-customer-key-MD5:EmqLRYqvItSQUzWCBAdF+A==" \
< ~/test.txt
PUT /test.txt HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate, compress
Content-Length: 20
Content-Type: application/json; charset=utf-8
Host: BUCKETNAME.s3.amazonaws.com
User-Agent: HTTPie/0.8.0
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: BASE64KEY
x-amz-server-side-encryption-customer-key-MD5: EmqLRYqvItSQUzWCBAdF+A==
This is a test file
HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 21 Oct 2015 22:12:26 GMT
ETag: "5dd39cab1c53c2c77cd352983f9641e1"
Server: AmazonS3
x-amz-id-2: AUOQUfmHEwOPqqvDd5X7aTYk+SX043gVFvM3wlgbzfRcpQsXIxXOFjrTRAM+B2T9Ns6Z/C26lBg=
x-amz-request-id: 6063C14465E4B090
Everything seemed to be working, although the encryption headers didn't come back in the response. So, I attempted to fetch my new object:
$ curl 'https://BUCKETNAME.s3.amazonaws.com/test.txt'
This is a test file
Oh no! My encryption headers appear to have been completely ignored, and the object has been stored in plaintext. As far as I can tell from the documentation, I am uploading the object correctly. Any suggestions as to what I might be doing wrong?
What's really awful is that if I do a GET and include this key, I get back a 200. That's terrifying. I could easily have started using these calls and never noticed that no encryption was being performed.
I have discovered what I believe to be the cause. When I upload objects to S3 anonymously (but providing a key), the server-side encryption credentials are completely ignored, and my object is stored in plaintext. The credentials are also ignored when downloading, so the download works fine.
However, if I authenticate as any AWS user, the headers are respected, and my object is stored with appropriate encryption.
So, important note to SSE-C users: make sure you don't upload any objects anonymously, or the whole feature silently ignores your encryption keys entirely!
Related
The AWS S3 PUT REST API docs are lacking a clear example of the Authorization string in the Request Syntax.
Request Syntax
PUT /Key+ HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
Cache-Control: CacheControl
Content-Disposition: ContentDisposition
Content-Encoding: ContentEncoding
Content-Language: ContentLanguage
Content-Length: ContentLength
Content-MD5: ContentMD5
Content-Type: ContentType
Expires: Expires
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write-acp: GrantWriteACP
x-amz-server-side-encryption: ServerSideEncryption
x-amz-storage-class: StorageClass
x-amz-website-redirect-location: WebsiteRedirectLocation
x-amz-server-side-encryption-customer-algorithm: SSECustomerAlgorithm
x-amz-server-side-encryption-customer-key: SSECustomerKey
x-amz-server-side-encryption-customer-key-MD5: SSECustomerKeyMD5
x-amz-server-side-encryption-aws-kms-key-id: SSEKMSKeyId
x-amz-server-side-encryption-context: SSEKMSEncryptionContext
x-amz-request-payer: RequestPayer
x-amz-tagging: Tagging
x-amz-object-lock-mode: ObjectLockMode
x-amz-object-lock-retain-until-date: ObjectLockRetainUntilDate
x-amz-object-lock-legal-hold: ObjectLockLegalHoldStatus
Body
The docs show this request example further on...
PUT /my-image.jpg HTTP/1.1
Host: myBucket.s3.<Region>.amazonaws.com
Date: Wed, 12 Oct 2009 17:50:00 GMT
Authorization: authorization string
Content-Type: text/plain
Content-Length: 11434
x-amz-meta-author: Janet
Expect: 100-continue
[11434 bytes of object data]
But again, the doc does not have an example format for Auth String. I tried AccessKeyID Secret but that didn't work. I dont' even see logical parameters in the request syntax to pass the two parts of the credential (AccessKeyID and Secret) anywhere in the examples!
Does anyone have a simple example of how to use PUT to add a .json file to S3 using the REST API? Preferrably a screenshot of PostMan setup to better explain where values go (in URL vs. as headers).
From the AWS docs here, it appears it is not possible to create a PUT request to an S3 bucket using REST API alone:
For authenticated requests, unless you are using the AWS SDKs, you have to write code to calculate signatures that provide authentication information in your requests.
This is a new concept to me. I've used token requests and sending keys in headers before when authenticating via REST API's. It sounds like a more secure method of auth.
We are experiencing upload errors to BigQuery / cloud storage:
REQUEST
POST https://www.googleapis.com/upload/bigquery/v2/projects/XXX HTTP/1.1
Content-Type: multipart/related; boundary="PART_TAG_DATA_IMPORTER"
Host: www.googleapis.com
Content-Length: 652
--PART_TAG_DATA_IMPORTER
Content-Type: application/json; charset=UTF-8
{"configuration":{"load":{"createDisposition":"CREATE_IF_NEEDED","destinationTable":{"datasetId":"XX","projectId":"XX","tableId":"XX"},"schema":{"fields":[{"mode":"required","name":"xx1","type":"INTEGER"},{"mode":"required","name":"xx2","type":"STRING"},{"mode":"required","name":"xx3","type":"INTEGER"}]},"skipLeadingRows":1,"sourceFormat":"CSV","sourceUris":["gs://XXX/9f41d369-b63e-4858-9108-7d1243175955.csv"],"writeDisposition":"WRITE_TRUNCATE"}}}
--PART_TAG_DATA_IMPORTER--
RESPONSE:
HTTP/1.1 400 Bad Request
X-GUploader-UploadID: XXX
Content-Length: 77
Date: Fri, 15 Nov 2019 10:23:33 GMT
Server: UploadServer
Content-Type: text/html; charset=UTF-8
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Payload parts count different from expected 2. Request payload parts count: 1
Anyone else receiving this? Everything worked fine since last night. There were no changes in our codebase and error is happening in about 80% of the cases but after 5-6 attempts it (sometimes) goes through.
We are using .NET and have the latest Google.Apis libraries but this is reproducible by simple request to the server. It also sometimes goes through normally.
Google has added check in /upload/bigquery/v2/projects/{projectId}/jobs endpoint a rule that it cannot receive single part message.
/bigquery/v2/projects/{projectId}/jobs needs to be used when doing upload from GCS as per this documentation here (which does not say this explicitly):
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert
This looks quite odd. It appears you're using the inline upload endpoint but you're passing a reference to a GCS object in the load config, and not sending an inline upload.
Could you share a snippet of how you're constructing this from the .NET code?
I'm using cloudfront secure cookies to keep some files private. When cookie auth succeeds and the origin is hit cloudfront returns the proper cors headers (Access-Control-Allow-Origin) from the origin but how do I make cloudfront return CORS headers during a 403/Access Denied? This validation is entirely in cloudfront before the request to the origin, but is there a setting to enable it? I want to be able to make a XHR request to cloudfront and know why the request failed. Since cloudfront doesn't return cors headers on a 403 most modern browsers will prevent reading any information on the request including the status code and its tough to determine why the request failed.
Thanks!
As you know, CloudFront doesn't spontaneously emit CORS headers -- they need to come from the origin server -- so in order to see CORS headers in the response, the request needs to be allowed by CloudFront... but, of course, it can't be allowed, because the condition you're trying to catch is 403 Forbidden.
So, what we need in order to allow your unauthorized responses to be CORS-friendly is an additional origin that can provide us with an alternate error response, and that origin needs to be CORS-aware. The solution seems to be something we can accomplish with a little help from CloudFront Custom Error Responses and an otherwise-empty S3 bucket, created for the purpose.
Custom error responses allow you to configure CloudFront to fetch the custom error response from another origin, rather than generating it internally. As part of that process, some headers from the original request are included in the upstream fetch, and the response headers from the error document are returned.
S3 makes a handy origin, since it has configurable CORS support.
Create a new, empty bucket.
Enable CORS for the bucket, and configure CORS with the appropriate parameters. The default configuration may be fine for this purpose.
Create a simple file that your CloudFront distribution will be using instead of its built in response for a 403. For test purposes, that can just be a text file that says "Access denied."
Upload the file to the bucket with whatever name you like, such as 403.txt. Select the option to make the object publicly-readable. Set metadata Cache-Control: no-cache, no-store, private, must-revalidate and Content-Type: text/plain (or text/html, depending on what exactly you put in the error file).
In CloudFront, create a new Origin. For the Origin Domain Name, select the bucket from the list of buckets.
Create a new Cache Behavior, matching path /403.txt (or whatever you named the file). Whitelist the Origin, Access-Control-Request-Headers, and Access-Control-Request-Method headers for forwarding. Set Restrict Viewer Access to No, because for this one path, we don't require signed credentials. Note that this path needs to be exactly the same as the filename in the bucket (except the leading slash, which isn't shown in the bucket but should be included, here).
In CloudFront Custom Error Responses, choose Create Custom Error Response. Select error code 403, set Error Caching Minimum TTL to 0, choose Customize Error Response Yes, set Response Page Path /403.txt and set HTTP Response code to 403.
Profit!
Test:
$ curl -v dzczcnnnnexample.cloudfront.net -H 'Origin: http://example.com'
* Rebuilt URL to: dzczcnnnnexample.cloudfront.net/
* Trying 203.0.113.208...
* Connected to dzczcnnnnexample.cloudfront.net (203.0.113.208) port 80 (#0)
> GET / HTTP/1.1
> Host: dzczcnnnnexample.cloudfront.net
> User-Agent: curl/7.47.0
> Accept: */*
> Origin: http://example.com
>
< HTTP/1.1 403 Forbidden
< Content-Type: text/plain
< Content-Length: 16
< Connection: keep-alive
< Date: Sun, 08 Apr 2018 14:01:25 GMT
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, HEAD
< Access-Control-Max-Age: 3000
< Last-Modified: Sun, 08 Apr 2018 13:29:19 GMT
< ETag: "fd9e8f7be7b65381c4acc272b6afc858"
< x-amz-server-side-encryption: AES256
< Cache-Control: private, no-cache, no-store, must-revalidate
< Accept-Ranges: bytes
< Server: AmazonS3
< Vary: Origin,Access-Control-Request-Headers,Access-Control-Request-Method
< X-Cache: Error from cloudfront
< Via: 1.1 1234567890a26beddeac6bfc77b2d348.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: ExAmPlEIbQBtaqExamPLEQs4VwcxhvtU1YXBi47uUzUgami0Hj0MwQ==
<
Access denied.
* Connection #0 to host dzczcnnnnexample.cloudfront.net left intact
Here, Access denied. is what I put in the text file I created. You may want to get a little more creative, after confirming that this works for you, as it does for me. The content of this new file in S3 will always be returned whenever CloudFront throws a 403 error. Additionally, it will also be returned whenever your origin throws a 403, because custom error responses are designed to replace all errors with a given HTTP status code.
You note, above, that we see Access-Control-Allow-Origin: *. This is the default behavior of S3 CORS. If you provide explicit origins in the S3 CORS config, you get a response like this...
Access-Control-Allow-Origin: http://example.com
...but for GET requests, I assume this level of specificity would not be necessary and the wildcard would suffice. The scenario described here isn't setting CORS for the entire CloudFront distribution -- just for the error response.
I have some images that I need to do a HttpRequestMethod.HEAD in order to find out some details of the image.
When I go to the image url on a browser it loads without a problem.
When I attempt to get the Header info via my code or via online tools it fails
An example URL is http://www.adorama.com/images/large/CHHB74P.JPG
As mentioned, I have used the online tool Hurl.It to try and attain the Head request but I am getting the same 403 Forbidden message that I am getting in my code.
I have tried adding many various headers to the Head request (User-Agent, Accept, Accept-Encoding, Accept-Language, Cache-Control, Connection, Host, Pragma, Upgrade-Insecure-Requests) but none of this seems to work.
It also fails to do a normal GET request via Hurl.it. Same 403 error.
If it is relevant, my code is a c# web service and is running on the AWS cloud (just in case the adorama servers have something against AWS that I dont know about). To test this I have also spun up an ec2 (linux box) and run curl which also returned the 403 error. Running curl locally on my personal computer returns the binary image which is presumably just the image data.
And just to remove the obvious thoughts, my code works successfully for many many other websites, it is just this one where there is an issue
Any idea what is required for me to download the image headers and not get the 403?
same problem here.
Locally it works smoothly. Doing it from an AWS instance I get the very same problem.
I thought it was a DNS resolution problem (redirecting to a malfunctioning node). I have therefore tried to specify the same IP address as it was resolved by my client but didn't fix the problem.
My guess is that Akamai (the service is provided by an Akamai CDN in this case) is blocking AWS. It is understandable somehow, customers pay by traffic for CDN, by abusing it, people can generate huge bills.
Connecting to www.adorama.com (www.adorama.com)|104.86.164.205|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 403 Forbidden
Server: **AkamaiGHost**
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 301
Cache-Control: max-age=604800
Date: Wed, 23 Mar 2016 09:34:20 GMT
Connection: close
2016-03-23 09:34:20 ERROR 403: Forbidden.
I tried that URL from Amazon and it didn't work for me. wget did work from other servers that weren't on Amazon EC2 however. Here is the wget output on EC2
wget -S http://www.adorama.com/images/large/CHHB74P.JPG
--2016-03-23 08:42:33-- http://www.adorama.com/images/large/CHHB74P.JPG
Resolving www.adorama.com... 23.40.219.79
Connecting to www.adorama.com|23.40.219.79|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.0 403 Forbidden
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 299
Cache-Control: max-age=604800
Date: Wed, 23 Mar 2016 08:42:33 GMT
Connection: close
2016-03-23 08:42:33 ERROR 403: Forbidden.
But from another Linux host it did work. Here is output
wget -S http://www.adorama.com/images/large/CHHB74P.JPG
--2016-03-23 08:43:11-- http://www.adorama.com/images/large/CHHB74P.JPG
Resolving www.adorama.com... 23.45.139.71
Connecting to www.adorama.com|23.45.139.71|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.0 200 OK
Content-Type: image/jpeg
Last-Modified: Wed, 23 Mar 2016 08:41:57 GMT
Server: Microsoft-IIS/8.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
ServerID: C01
Content-Length: 15131
Cache-Control: private, max-age=604800
Date: Wed, 23 Mar 2016 08:43:11 GMT
Connection: keep-alive
Set-Cookie: 1YDT=CT; expires=Wed, 20-Apr-2016 08:43:11 GMT; path=/; domain=.adorama.com
P3P: CP="NON DSP ADM DEV PSD OUR IND STP PHY PRE NAV UNI"
Length: 15131 (15K) [image/jpeg]
Saving to: \u201cCHHB74P.JPG\u201d
100%[=====================================>] 15,131 --.-K/s in 0s
2016-03-23 08:43:11 (460 MB/s) - \u201cCHHB74P.JPG\u201d saved [15131/15131]
I would guess that the image provider is deliberately blocking requests from EC2 address ranges.
The reason the wget outgoing ip address is different in the two examples is due to DNS resolution on the cdn provider that adorama are providing
Web Server may implement ways to check particular fingerprint attributes to prevent automated bots . Here a few of them they can check
Geoip, IP
Browser headers
User agents
plugin info
Browser fonts return
You may simulate the browser header and learn some fingerprinting "attributes" here : https://panopticlick.eff.org
You can try replicate how a browser behave and inject similar headers/user-agent. Plain curl/wget are not likely to satisfied those condition, even tools like phantomjs occasionally get blocked. There is a reason why some prefer tools like selenium webdriver that launch actual browser.
I found using another url also being protected by AkamaiGHost was blocking due to certain parts in the user agent. Particulary using a link with protocol was blocked:
Using curl -H 'User-Agent: some-user-agent' https://some.website I found the following results for different user agents:
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 okay
facebookexternalhit/1.1 (+http\://www.facebook.com/externalhit_uatext.php): 403
https ://bar: okay
https://bar: 403
All I could find for now is this (downvoted) answer https://stackoverflow.com/a/48137940/230422 stating that colons (:) are not allowed in header values. That is clearly not the only thing happening here as the Mozilla example also has a colon, only not a link.
I guess that at least most webservers don't care and allow facebook's bot and other bots having a contact url in their user agent. But appearently AkamaiGHost does block it.
I try to write a C++-application and I have to do HTTP Digest Authentication. The problem is not primarily about C++, but about the fact, that the connection is not being established. The website I try to access is the following: httpbin.org/digest-auth/auth/user/passwd .
Consider the following server response to a simple GET /digest-auth/auth/user/passwd:
HTTP/1.1 401 UNAUTHORIZED
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Mon, 08 Sep 2014 15:10:09 GMT
Server: gunicorn/18.0
Set-Cookie: fake=fake_value
Www-Authenticate: Digest realm="me#kennethreitz.com", nonce="2a932bfb1f9a748a7b5ee590d0cf99e0", qop=auth, opaque="2d09668631b42bff8375523e7b27e45e"
Content-Length: 0
Connection: keep-alive
A1 is then computed to be user:me#kennethreitz.com:passwd and is hashed to 4de666b60f91e2444f549243bed5fa4b which I will refer to as HA1.
A2 is computed to be GET:/digest-auth/auth/user/passwd and hashed to b44272ea65ee4af7fb26c5dba58f6863 which I will refer to as HA2.
With this information, the response is computed as HA1:nonce:1:ac3yyj:auth:HA2, where HA1 and HA2 are the values we have just computed and nonce is taken from the server response above, which is in total: 4de666b60f91e2444f549243bed5fa4b:2a932bfb1f9a748a7b5ee590d0cf99e0:1:ac3yyj:auth:b44272ea65ee4af7fb26c5dba58f6863. The hash of that is 55f292e183ead0810528bb2a13b98e00.
Combining all that information should be sufficient to establish a http-connection using digest authentication. However, the following request is declined by the server and answered with another HTTP/1.1 401.
GET /digest-auth/auth/user/passwd HTTP/1.1
Host: httpbin.org
Authorization: Digest username="user", realm="me#kennethreitz.com",nonce="2a932bfb1f9a748a7b5ee590d0cf99e0",uri="/digest-auth/auth/user/passwd",qop=auth,nc=1,cnonce="ac3yyj",response="55f292e183ead0810528bb2a13b98e00",opaque="2d09668631b42bff8375523e7b27e45e"
Note that the formatting does not show the structure of the request. The block from Authorization to opaque is actually one line.
Feel free to re-do the md5-calculation - but I have redone the calculations manually and got the same hashes as my program did. I used that tool (http://md5-hash-online.waraxe.us/) for the manual computations.
Am I missing something obvious here, probably misinterpreting the standard in a way? Why can't I get authorized?
Finally got it. The authentication itself is completely correct.
The server demands a cookie to be set. Apparently one must show that cookie in the reponse, too. That explains why firefox (being a browser) can authenticate correctly while curl and lwp-request can't albeit sticking to the standard - RFC said nothing about cookies. Why do we have standards, that noone cares for?
Anyway, appending Cookie: fake=fake_value to the header solves the problem.