I am curious why I am getting a SignatureDoesNotMatch error on a Versioned URL for an object in my S3 bucket (by appending versionId=abdcd).
When the URL has the following pattern:
https://s3.amazonaws.com/bucket/object.tar.gz
everything seems to work fine. However as soon as I add the versionId query string, I get the signature mismatch error
https://s3.amazonaws.com/bucket/object.tar.gz?versionId=abcde
The method for accessing the object is controlled by CloudFormation::Init's source property, as describe here
S3 Bucket
The following example downloads a zip file from an Amazon S3 bucket
and unpacks it into /etc/myapp:
Note You can use authentication credentials for a source. However, you
cannot put an authentication key in the sources block. Instead,
include a buckets key in your S3AccessCreds block. For an example, see
the example template. For more information on Amazon S3 authentication
credentials, see AWS::CloudFormation::Authentication. JSON
"sources" : { "/etc/myapp" :
"https://s3.amazonaws.com/mybucket/myapp.tar.gz" }
This is not a permissions issue because it works fine when there's no versionId query string. I am curious why the signing method is failing by just adding that? And how can I fix it?
I read up AWS's Signature V4 but I don't see anything that explains why this is not working? It seems that query strings are only supposed to be used for auth headers?
Related
Unfortunately i cannot use AWS SDK and i must use REST API AWS services(i am working with Flutter WEB).
So i start to do research on aws docs and this is what i did:
Create bucket.
make all permission to be public(for test only)
open access point
Then at the postman i am using at "Auth" aws signature and added current keys,
except session token(my bucket is open and i understood that i dont need it, if i am mistaking pls correct me).
when i am sending the request i am getting an error:
Missing required header for this request: x-amz-content-sha256
Here is attachment images:
AWS Signautre
Body Request
Error message
Just ran into this issue too - the Service Name must be 's3' vs 'S3'.
advanced authorization settings
According to the AWS documentation, you need to send the x-amz-content-sha256 header in the request:
The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there is no payload, you must provide the hash of an empty string.
Seems like Postman is not adding it for some reason. You should check the Headers and add manually then.
try replacing S3 with s3 for the service name.
I too faced this issue and after adding session token parameter it got resolved. Postman has the provision to generate the below parameters,
X-Amz-Content-Sha256
Authorization
x-amz-security-token
X-Amz-Date
How can I read a csv file placed in aws using Postman ?
I am able to get list of files and folders places in aws using
aws url:
https://s3.amazonaws.com/partnertrackingconfigurationdev/?list-type=2
access key, secret key and session token.
But, I would like to read the csv file placed at below url using Postman.
aws url:
https://s3.amazonaws.com/partnertrackingconfigurationdev/?list-type=2&Key=QAInputData/CarrierTrackingNumbers.csv
Please help
On s3 in AWS you can explicite configure folder/file permissions. Is this here what you are looking for?
https://docs.aws.amazon.com/workdocs/latest/userguide/web_share_link.html
https://docs.aws.amazon.com/workdocs/latest/userguide/permissions.html#doc_perms
EDIT
I am also a little bit confused, that in your URL that after the 'RequestParams' (?list-type=2&....) you use again a 'Path' (/CarrierTrackingNumbers.csv).
Should it not be:
https://s3.amazonaws.com/partnertrackingconfigurationdev/CarrierTrackingNumbers.csv?list-type=2&Key=QAInputData
Trying to download file using s3 pre-signed url. We have integration with google analytics because of that all the hyperlinked is appended with one extra query param, "_ga".
pre-signed url constructed is like https://someapprouter.herokuapp.com/ant/storage/1619958100/f8c40a75/file?response-content-disposition=attachment%3B%20fileName%3D%22PersonSampleFormat.csv%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20190403T191315Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ACKJOEK2Q%2F20190403%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=xxxx
But when we are trying to download file from browser the extra query param is getting added which is causing the error in download. Finla URL is like https://someapprouter.herokuapp.com/ant/storage/1619958100/f8c40a75/file?response-content-disposition=attachment%3B%20fileName%3D%22PersonSampleFormat.csv%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20190403T191315Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=AKJOEK2Q%2F20190403%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=xxxx&_ga=yyyyy
Exception :
Error>
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.
How can we exclude query params _ga when s3 is doing signature match.
This is happening because you have autolinker enabled for your GA implementation. To remove it, you need to disable autolinker.
Note, if you're doing cross-domain tracking, then you'll need to customize the autolinker to specific domains, so that the linker will know to only auto link those domains.
If you provide more details of how GA is implemented or provide a link, I can show you specific code.
I am creating a signed URL with AWS so I can safely pass this URL to another API for temporary use. The signed URL points to a S3 resource. The problem is the other API does not accept such long links. Therefore I am trying to shorten it. I tried to use shorteners like goo.gl or bit.ly to no avail because the URL was too long for them. I even built my own private shortener with AWS (AWS url shortener) but it had the same problem: "The length of website redirect location cannot exceed 2,048 characters.".
I am creating the signed URLs in iOS (Swift) with AWSS3PreSignedURLBuilder.default().getPreSignedURL(preSignedURLRequest) while using AWS Cognito as an unauthorised user.
I have tried the following things to no avail:
Choose the shortest possible S3 bucket name with 3 characters
Shorten the filename as much as possible. I limited the file name to 10 characters plus file extension name (14 characters in total). Shorter file names are not viable for me because they should be unique to a certain extent.
But even with all these minor tweaks the signed URL returned by AWS is sometimes too long. Especially the token parameter (X-Amz-Security-Token) seems to be really long. With my minor tweaks I sometimes get URLs shorter than 2,048 characters but sometimes slightly longer. I would like to find a solution which guarantees me that the URL is not too long and can be shortened.
In my own private AWS URL shortener the following code snippet creates the S3 object which redirects to the actual long URL.
s3.putObject({
Bucket: s3_bucket,
Key: key_short,
Body: "",
WebsiteRedirectLocation: url_long,
ContentType: "text/plain"
},
(err, data) => {
if (err) {
console.log(err);
done("", err.message);
} else {
const ret_url = "https://" + cdn_prefix + "/" + id_short;
console.log("Success, short_url = " + ret_url);
done(ret_url, "");
}
});
The method returns with the following error
The length of website redirect location cannot exceed 2,048
characters.
The documentation of putObject for the header "x-amz-website​-redirect-location" in the object meta states the following (see: put object documentation):
The length of the value is limited to 2 KB
How can I make sure that the initial AWS signed URL is not too long for the URL shorteners?
EDIT:
One of the problems I have identified is that I create the signed URL as an unauthenticated user in AWS Cognito. Therefore the signed URL includes this ridiculously long token as a parameter. I did not want to embed my accessKey and shortKey in the iOS App thats why I switched to AWS Cognito (see aws cognito). But currently there are no authorised users just unauthorised ones and I need to create the signed URL as an unauthorised AWS Cognito user. If I create the signed URL with with a regular credentials using accessKey and shortKey I get a much shorter URL. But for that I would have to embed my accessKey and shortKey in the iOS app which is not recommended.
I solved the problem by creating an AWS lambda for creating a presigned URL and returning the presigned URL. The presigned URL allows the caller to access (getObject) the S3 resource. There are two options regarding this:
The role assigned to the AWS lambda has the S3 permission for getObject. The resulting presigned URL will have a much shorter token included than the presigned URL created with the temporary credentials issued by AWS Cognito in the iOS app.
Embed the access key and secret key of a role with the S3 permission for getObject directly into the AWS lambda which will give you an even shorter URL because there is no token included in the resulting presigned URL. (e.g. sample AWS code)
I call this lambda from within my iOS app as an unauthorised cognito user. After receiving the presigned URL from the AWS lambda I am able to shorten it because with this method the presigned URLs are much shorter.
There is an older method of generating pre-signed URLs that produces a very short link, eg:
https://s3-ap-southeast-2.amazonaws.com/my-bucket/foo.png?AWSAccessKeyId=AKI123V12345RYTP123&Expires=1508620600&Signature=oB1/jca2JFXw5DbN7gBKEXkUQk8%3D
However, this pre-dates sigv4 so it does not work in the newer regions (Frankfurt onwards).
You can find sample code at:
mdwhatcott/s3_presign.py
S3 Generate Pre Signed Url
It can also be used to sign for uploads: Correct S3 Policy For Pre-Signed URLs
I am using CarrierWaveDirect to upload a high resolution images to s3. I then use that image to process multiple versions which are made public through Cloudfront urls.
The uploaded high res files need to remain private to anonymous users, but the web application needs to access the private file in order to do the processing for other versions.
I am currently setting all uploaded files to private in the CarrierWave initializer via
config.fog_public = false
I have an IAM policy for the web application that allows full admin access. I also have set the ACCESSKEY AND SECRETKEY in the app for that IAM user. Given these two criteria, I would think that the web app could access the private file and continue with processing, but it is denied access to the private file.
*When I log into the user account associated with the web app, I am able to access the private file because a token is added on to the URL.
I can't figure out why the app cannot access the private file given the ACCESSKEY AND SECRRETKEY
I was having a hard time getting to your problem. I am quite certain your question is not
unable to access private s3 file even though IAM policy grants access
but rather
how to handcraft a presigned URL for GETting a private file on S3
The gist shows you're trying to create the presigned URL for GET yourself. While this is perfectly fine, it's also very error-prone.
Please verify that what you're trying to do is working at all, using the AWS SDK for Ruby (I only post code known to work with version 1 here but if you aren't held back by legacy code, start with version 2):
s3 = AWS::S3.new
bucket = s3.buckets["your-bucket-name"]
obj = bucket.objects["your-object-path"]
obj.url_for(:read, expires: 10*60) # generate a URL that expires in 10 minutes
See the docs for AWS::S3::S3Object#url_for and Aws::S3::Object#presigned_url for details.
You may need to read up on passing args to AWS::S3.new here (for credentials, regions and so).
I'd advise you take the following steps:
Make it work locally using the access_key_id and secret_access_key
Make it work in your worker
If it works, you can compare the query string the SDK returned with the one you handcrafted yourself. Maybe you can spot an error.
But in any case, I suggest you use higher-level SDKs to do things like that for you.
If this doesn't get you anywhere, please post a comment with your new findings.