The AWS S3 PUT REST API docs are lacking a clear example of the Authorization string in the Request Syntax.
Request Syntax
PUT /Key+ HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
Cache-Control: CacheControl
Content-Disposition: ContentDisposition
Content-Encoding: ContentEncoding
Content-Language: ContentLanguage
Content-Length: ContentLength
Content-MD5: ContentMD5
Content-Type: ContentType
Expires: Expires
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write-acp: GrantWriteACP
x-amz-server-side-encryption: ServerSideEncryption
x-amz-storage-class: StorageClass
x-amz-website-redirect-location: WebsiteRedirectLocation
x-amz-server-side-encryption-customer-algorithm: SSECustomerAlgorithm
x-amz-server-side-encryption-customer-key: SSECustomerKey
x-amz-server-side-encryption-customer-key-MD5: SSECustomerKeyMD5
x-amz-server-side-encryption-aws-kms-key-id: SSEKMSKeyId
x-amz-server-side-encryption-context: SSEKMSEncryptionContext
x-amz-request-payer: RequestPayer
x-amz-tagging: Tagging
x-amz-object-lock-mode: ObjectLockMode
x-amz-object-lock-retain-until-date: ObjectLockRetainUntilDate
x-amz-object-lock-legal-hold: ObjectLockLegalHoldStatus
Body
The docs show this request example further on...
PUT /my-image.jpg HTTP/1.1
Host: myBucket.s3.<Region>.amazonaws.com
Date: Wed, 12 Oct 2009 17:50:00 GMT
Authorization: authorization string
Content-Type: text/plain
Content-Length: 11434
x-amz-meta-author: Janet
Expect: 100-continue
[11434 bytes of object data]
But again, the doc does not have an example format for Auth String. I tried AccessKeyID Secret but that didn't work. I dont' even see logical parameters in the request syntax to pass the two parts of the credential (AccessKeyID and Secret) anywhere in the examples!
Does anyone have a simple example of how to use PUT to add a .json file to S3 using the REST API? Preferrably a screenshot of PostMan setup to better explain where values go (in URL vs. as headers).
From the AWS docs here, it appears it is not possible to create a PUT request to an S3 bucket using REST API alone:
For authenticated requests, unless you are using the AWS SDKs, you have to write code to calculate signatures that provide authentication information in your requests.
This is a new concept to me. I've used token requests and sending keys in headers before when authenticating via REST API's. It sounds like a more secure method of auth.
Related
I'm trying to use AWS api to get keys from Secrets Manager using an internal linux server (using cURL). now i constructed a postman payload that works, but i noticed that the signature digest is changing with every request (because the time is also changing in the request). I tried to recreate this myself using an online hmac-sha256 generator tool, but i cant seem to create the correct payload to make the signatures match, even if i simply copy postman's http generated payload. this is my http payload according to postman:
POST https://secretsmanager.eu-central-1.amazonaws.com/ HTTP/1.1
Host: secretsmanager.eu-central-1.amazonaws.com
Content-Type: application/x-amz-json-1.1
X-Amz-Target: secretsmanager.GetSecretValue
Accept-Encoding: identity
X-Amz-Content-Sha256: beaead3198f7da1e70d03ab969765e0821b24fc913697e929e726aeaebf0eba3
X-Amz-Date: 20220809T093728Z
Authorization: AWS4-HMAC-SHA256 Credential=credentils/20220809/eu-central-1/secretsmanager/aws4_request, SignedHeaders=accept-encoding;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=signature
Content-Length: 98
{
"SecretId": "arn:aws:secretsmanager:eu-central-1:rest-of-arn"
}
hashing it with my secret key (hmac-sha256) just doesnt output the same signature as the one in the payload - and the request fails.
anyone knows how to construct the correct payload to make this work?
The signature is supposed to change as the timestamp of the request is included in the "string to sign" in the second step of the aws v4 signature.
As you can see, the signing process is quite complex and my suggestion is to avoid trying to re-implement it in a prod environment.
Instead, if you can, install the aws cli or update curl to a version that supports aws v4 signature.
An AWS Lamba function (NodeJS) returning 3 HTTP headers: aaa, Access-Control-Allow-Origin and bbb was created:
exports.handler = async (event) => {
const response = {
statusCode: 200,
headers: { "aaa":"aaa", "Access-Control-Allow-Origin":"*", "bbb":"bbb" },
body: JSON.stringify('Hello from Lambda!'),
};
return response;
};
The function is integrated into a HTTP API (not REST API). In the HTTP API Gateway Configuration, Section "Configure CORS", the HTTP header "Access-Control-Allow-Origin" was set to "*". Please see the screenshot:
Gateway Config
The command "curl -i https://xxxxxxxxxx.execute-api.eu-central-1.amazonaws.com" proves that the HTTP Header Access-Control-Allow-Origin is explicitly removed, because only HTTP headers aaa and bbb are returned:
HTTP/2 200
date: Tue, 14 Apr 2020 11:01:58 GMT
content-type: text/plain; charset=utf-8
content-length: 20
aaa: aaa
bbb: bbb
apigw-requestid: K-S2EjVWliAEJKw=
Why on earth is this header still not present, even after "Configure CORS" was done?
(I'm googling now for more than two days in order to find a solution and it makes me go nuts)
As per Configuring CORS for an HTTP API -
If you configure CORS for an API, API Gateway ignores CORS headers
returned from your backend integration.
That's why the CORS headers from your Lambda (integration) are being ignored. This is one of the differences between the new HTTP APIs from the original REST APIs. In case of these APIs -
For a CORS request, API Gateway adds the configured CORS headers to
the response from an integration.
When you do a simple curl, that is not actual doing a cross-origin request. Hence, you don't see the CORS headers that would be set by the HTTP API. To verify if a CORS request works, I passed an Origin header in the below request and I can see the CORS headers along with my custom headers from Lambda -
$ curl -v -X GET https://$API_ID.execute-api.$AWS_REGION.amazonaws.com -H "Origin: https://www.example.com"
< HTTP/2 200
< date: Tue, 14 Apr 2020 18:02:26 GMT
< content-type: text/plain; charset=utf-8
< content-length: 18
< aaa: aaa
< bbb: bbb
< access-control-allow-origin: https://www.example.com
< access-control-expose-headers: date, x-api-id
Below is a snippet of my CORS configuration on the API. I added Access-Control-Allow-Origin value as https://www.example.com and passed this as a part of the Origin header in my curl request. Such a request would qualify as CORS.
For POST/PUT requests, you'll need to white list the content-type header.
Putting the wildcard doesn't do the trick for some reason, you need to explicitly whitelist it.
For Googlers:
If your OPTIONS preflights succeed but no Access-Control- headers present, and if you are testing using CURL, take extra attention to spell required headers for a preflight:
Access-Control-Request-Method: GET
Access-Control-Request-Headers: authorization
Origin: https://theaws.blog
If wrongly spelt, OPTIONS succeeds with 204 No Content, but no Access-Control- headers thus render the preflight result invalid.
Also check you have enough scope for the parameters, as long as your requested method and headers are a subset of the parameter, you will get Access-Control- headers. If not, you will get 204 No Content instead, which I would say it's not very informative!
I am looking at example AWS STS API request calls like below from the AWS web page
GET / HTTP/1.1
Host: sts.amazonaws.com
Content-Type: application/json; charset=utf-8
URL: https://sts.amazonaws.com/?DurationS .......
I can't seem to understand if the URL HTTP Header is a custom header used only by AWS or if its something new in HTTP (like with CORS)
Either way I could not find any reference to this Header.
Can someone please explain what this header is ? (i.e custom to AWS or something more general)
I am attempting to use S3's server-side encryption for customer keys. I created a bucket which allows anonymous users to upload objects, and then attempted to upload an object like so:
$ http -v PUT 'https://BUCKETNAME.s3.amazonaws.com/test.txt' \
"x-amz-server-side-encryption-customer-algorithm:AES256" \
"x-amz-server-side-encryption-customer-key:BASE64DKEY" \
"x-amz-server-side-encryption-customer-key-MD5:EmqLRYqvItSQUzWCBAdF+A==" \
< ~/test.txt
PUT /test.txt HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate, compress
Content-Length: 20
Content-Type: application/json; charset=utf-8
Host: BUCKETNAME.s3.amazonaws.com
User-Agent: HTTPie/0.8.0
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: BASE64KEY
x-amz-server-side-encryption-customer-key-MD5: EmqLRYqvItSQUzWCBAdF+A==
This is a test file
HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 21 Oct 2015 22:12:26 GMT
ETag: "5dd39cab1c53c2c77cd352983f9641e1"
Server: AmazonS3
x-amz-id-2: AUOQUfmHEwOPqqvDd5X7aTYk+SX043gVFvM3wlgbzfRcpQsXIxXOFjrTRAM+B2T9Ns6Z/C26lBg=
x-amz-request-id: 6063C14465E4B090
Everything seemed to be working, although the encryption headers didn't come back in the response. So, I attempted to fetch my new object:
$ curl 'https://BUCKETNAME.s3.amazonaws.com/test.txt'
This is a test file
Oh no! My encryption headers appear to have been completely ignored, and the object has been stored in plaintext. As far as I can tell from the documentation, I am uploading the object correctly. Any suggestions as to what I might be doing wrong?
What's really awful is that if I do a GET and include this key, I get back a 200. That's terrifying. I could easily have started using these calls and never noticed that no encryption was being performed.
I have discovered what I believe to be the cause. When I upload objects to S3 anonymously (but providing a key), the server-side encryption credentials are completely ignored, and my object is stored in plaintext. The credentials are also ignored when downloading, so the download works fine.
However, if I authenticate as any AWS user, the headers are respected, and my object is stored with appropriate encryption.
So, important note to SSE-C users: make sure you don't upload any objects anonymously, or the whole feature silently ignores your encryption keys entirely!
I am trying to connect to a REST API, using C#.
I was able to successfully do some GET request but POST kept giving me 401 Authentication error.
I have gone ahead and downloaded fiddler and this is what my requests look like:
GET (Works)
Request Headers
GET https: //hello.myurl.com/api HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/xml
Authorization: Basic ***************************************************************************************************
Host: hello.myurl.com
-
POST (Doesn't work)
Request Headers
POST https: //hello.myurl.com/api HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/xml
Authorization: Basic ***************************************************************************************************
Host: hello.myurl.com
Content-Length: 12
Request Body
status=hello
(* same in both using
String authinfo = "username:password";
Convert.ToBase64String(Encoding.ASCII.GetBytes(authInfo));
Any ideas why?
I'd consult the API documentation for that particular URL. GET requests are simply requesting a readonly version of data, a POST request is implying that you are making a change to a certain URL, so it's possible that the API allows GET requests without authentication, but requires authentication on the POST request.
In your case I'd hazard a guess that your authentication is not correct, but it's just being ignored on the GET request as it doesnt need authentication.