I am trying to generate a presigned url for the data object that I am uploading to the s3 bucket using S3 TransferManager. But I am getting the below error
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>mykey</Key>
<RequestId>9BB78BBE144063EA</RequestId>
<HostId>myhostid</HostId>
</Error>
If I use the com.amazonaws.services.s3.AmazonS3 to upload my data object I can generate the pre-signed url successfully. But I would like to use TransferManager as upload performance is far better as compared to AmazonS3 (near to 100 times in my case)
Here is the code that gives me the above error
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(region.getName())
.withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
.build();
TransferManager manager = TransferManagerBuilder
.standard()
.withS3Client(s3Client)
.build();
byte[] zippedBytes = compressLayers(layers); //here the compressed json is got
ObjectMetadata metaData = new ObjectMetadata();
metaData.setContentType(CONTENT_TYPE_JSON);
metaData.setContentEncoding(CONTENT_ENCODING_GZIP);
metaData.setContentLength(zippedBytes.length);
manager.upload(bucketName, uuidFileName, new ByteArrayInputStream(zippedBytes), metaData);
java.util.Date expiration = new Date(new Date().getTime()+1000*60*2);
GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(bucketName, uuidFileName)
.withMethod(HttpMethod.GET)
.withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(req);
In the above code if I user the s3Client to upload the data object,
s3Client.putObject(bucketName, uuidFileName, new ByteArrayInputStream(zippedBytes), metaData);
then it returns the pre-signed url correctly. Also is there any way to generate the pre-signed url using the TransferManager?
Any help will be highly appreciated.
Related
I have an issue with generation Presigned url for s3 file
ClientS3 configuration:
CryptoConfiguration cryptoConfig = new CryptoConfiguration()
.withAwsKmsRegion(RegionUtils.getRegion(Regions.EU_WEST_1.getName()));
AmazonS3 amazonS3 = AmazonS3EncryptionClient.encryptionBuilder()
.withCredentials(new DefaultAWSCredentialsProviderChain())
.withRegion(Regions.DEFAULT_REGION)
.withCryptoConfiguration(cryptoConfig)
.withEncryptionMaterials(new KMSEncryptionMaterialsProvider("alias/test-alias"))
.build();
but when I generate presigned url, I receive image, that are not decoded:
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest("test-uploads", "example.jpg")
.withMethod(HttpMethod.GET)
.withResponseHeaders(responseHeaderOverrides)
.withExpiration(Date.from(Instant.now().plus(3600*24, ChronoUnit.SECONDS)));
URL fileUrl = amazonS3.generatePresignedUrl(generatePresignedUrlRequest);
When I try to add SSEAlgorithm and KmsCmkId, than I receive error:
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest("test-uploads", "example.jpg")
.withMethod(HttpMethod.GET)
.withSSEAlgorithm(SSEAlgorithm.KMS)
.withKmsCmkId("alias/test-fs-user-uploads")
.withResponseHeaders(responseHeaderOverrides)
.withExpiration(Date.from(Instant.now().plus(3600*24, ChronoUnit.SECONDS)));
URL fileUrl = amazonS3.generatePresignedUrl(generatePresignedUrlRequest);
error:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>...</AWSAccessKeyId>
<StringToSign>...</SignatureProvided>
<StringToSignBytes>...</StringToSignBytes>
<CanonicalRequest>...</CanonicalRequest>
<CanonicalRequestBytes>...</CanonicalRequestBytes>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>
Could anybody help with it?
I am creating S3 presigned URl in one of my mvc controllers and that is to be consumed by a python lambda. But getting below error
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>There were headers present in the request which were not signed</Message>
<HeadersNotSigned>x-amz-date</HeadersNotSigned>
<RequestId>D7815D6EDC8BB6D0</RequestId>
<HostId>jViPAaaoh+UQDQoMzB9QZXAFyIxgvw7sLr7+Otj4tJFukLkynf07B1Z81iYE8aJ4i+p0B+1Dmwc=</HostId>
</Error>
Tried allowing all actions on S3 bucket.
//This is how I am creating presigned URL
public string BuildPreSignedURL(string bucketName, string objectKey)
{
string urlString = string.Empty;
IAmazonS3 s3Client = new AmazonS3Client();
try
{
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = objectKey,
Expires = DateTime.Now.AddMinutes(preSignedURLExpireTimeInMinutes),
};
urlString = s3Client.GetPreSignedURL(request);
}
catch (Exception ex)
{
throw;
}
return urlString;
}
//Below is now it is returned from controller
string preSignedURL = preSignedBuilder.BuildPreSignedURL(bucketName, entityS3Key);
return new RedirectResult(preSignedURL);
AWS lambda calling my api contoller should be able to download the file.
I don't know about C#, but with Python we need to set also the "method" when we generate the presigned URL. In the example below it uses get_object.
s3_client = session.client('s3')
try:
response = s3_client.generate_presigned_url('get_object',
Params={'Bucket': bucket_name,
'Key': object_name},
ExpiresIn=expiration)
except ClientError as e:
logging.error(e)
return None
I have created a customer key for AWS S3 Server side encryption using customer key (SSE-C).
I am able to upload the object using the key. But when I generate a presigned URL using AWS Java SDK the URL is getting created successfully but when I hit that URL I am getting the below error.
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.
import java.io.IOException;
import java.net.URI;
import java.net.URL;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.FileEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.HttpMethod;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.Headers;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.SSEAlgorithm;
import com.amazonaws.services.s3.model.SSECustomerKey;
import com.amazonaws.util.Base64;
import com.amazonaws.util.Md5Utils;
public class GeneratePresignedURL {
public GeneratePresignedURL() throws IOException {
String bucketName = "abctest";
String keyName = "testnew.mp4";
try {
SSECustomerKey SSE_KEY = new SSECustomerKey("KLgsVafKowMCfKDsbIh597CmMUSoPBn6QJ8OIGxAMBw=");
ClientConfiguration cnf = new ClientConfiguration();
cnf.withSignerOverride("AWSS3V4SignerType");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withClientConfiguration(cnf)
.withCredentials(new ProfileCredentialsProvider()).build();
// Set the presigned URL to expire after one hour.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
// Generate the presigned URL.
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName,
keyName).withMethod(HttpMethod.PUT).withExpiration(expiration).withSSECustomerKey(SSE_KEY);
generatePresignedUrlRequest.setSSECustomerKeyAlgorithm(SSEAlgorithm.AES256);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toExternalForm());
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}
I have tried to follow the code from https://aws.amazon.com/blogs/developer/generating-amazon-s3-pre-signed-urls-with-sse-c-part-5-finale/
I have no problem in using pre-signed url for SSE S3, I am only facing issue with SSE-C
I have tried setting default encryption and other config but of no help. Any pointers would be of great help.
Thanks,
AK
You can not call pre-signed URL direct via browser
You need to pass headers in the request
x-amz-server-side-encryption-customer-algorithm
x-amz-server-side-encryption-customer-key
x-amz-server-side-encryption-customer-key-MD5
Please check the document for more information
https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
When using the presigned URL to upload a new object, retrieve an existing object, or retrieve only object metadata, you must provide all the encryption headers in your client application.
Please check the sample code calling signed URL via Unirest library
https://github.com/pavanpawar4591/s3signurlwith-sse-c
public static void getPreSignedURL() throws URISyntaxException {
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyName)
.withMethod(HttpMethod.GET).withExpiration(expiration).withSSECustomerKey(SSE_KEY);
// generatePresignedUrlRequest.setContentType("video/mp4");
generatePresignedUrlRequest.setSSECustomerKeyAlgorithm(SSEAlgorithm.AES256);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toURI() + " With key: " + SSE_KEY);
System.out.println("------------------------------");
//https://aws.amazon.com/blogs/developer/generating-amazon-s3-pre-signed-urls-with-sse-c-part-4/
// refer to above doc
try {
HttpResponse<String> response = Unirest.get(url.toURI().toString())
.header(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, SSEAlgorithm.AES256.getAlgorithm())
.header(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, Base64.encodeAsString(SECRET_KEY.getEncoded()))
.header(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5,
Md5Utils.md5AsBase64(SECRET_KEY.getEncoded()))
.header("cache-control", "no-cache").header("postman-token", "d3453c38-1b59-a12e-fd97-dbe2150eadf5")
.asString();
System.out.println(response.getStatus());
System.out.println(response.getStatusText());
System.out.println(response.getBody());
} catch (UnirestException e) {
e.printStackTrace();
}
}
Thanks #pavan
Any client that needs to use SSE-C, the client must be capable of sending the Below headers.
When using the presigned URL to retrieve an existing object, or retrieve only object metadata, we need to provide all the encryption headers in your client application.
For S3 managed or KMS managed server side encryption, we can generate a presigned URL and directly paste that into a browser or player.
However, this is not true for SSE-C objects because in addition to
the presigned URL, you also need to include HTTP headers that are
specific to SSE-C objects. Therefore, you can use the presigned URL
for SSE-C objects only programmatically.
Upload:
I opted for S3 managed keys instead of customer provided client key.
FileInputStream fin = new FileInputStream(uploadFileName);
byte fileContent[] = new byte[(int) uploadFileName.length()];
// Reads up to certain bytes of data from this input stream into an array of
// bytes.
fin.read(fileContent);
// create string from byte array
// Specify server-side encryption.
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(fileContent.length);
objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName,
new ByteArrayInputStream(fileContent), objectMetadata);
// Upload the object and check its encryption status.
PutObjectResult putResult = s3Client.putObject(putRequest);
System.out.println("Object \"" + keyName + "\" uploaded with SSE.");
GET Presigned URL:
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyName)
.withMethod(HttpMethod.GET).withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toURI());
Reference:
https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
I am using AWS API gateway to invoke my aws lambda functions. They are REST web-services that will return response back to the caller. Some of the responses are more than 20 MB. Due to the nature of the response data (data is polylines that represents complex structures) I cannot use pagination here. AWS Lambda got the limitation of lambda response body cannot exceed 6MB size and due to this limitation some of my responses that are above 6MB are failing with 'body size is too long' message. I would like to know is there a way to increase the 6MB limitation of lambda responses?
Thanks for the comments, I have implemented in the same way as mentioned in the comments
Uploaded the response to s3 bucket, and provided a redirect url to the caller. Here is the sample code that works.
You may also enable CORS at s3 bucket level to make it work well. You may change to the correct region
public String uploadToS3(Set<MyClass> myclassSet, String bucketName) throws IOException {
Region region = Regions.getCurrentRegion();
if (region == null) {
region = Region.getRegion(Regions.US_EAST_1);
}
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(region.getName())
.withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
.build();
ObjectMapper mapper = new ObjectMapper();
String jsonClass = mapper.writeValueAsString(myclassSet);
s3Client.putObject(bucketName, "random_key.json", jsonClass);
GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(bucketName, "random_key.json");
URL url = s3Client.generatePresignedUrl(req);
String redirectUrl = url.toString();
return redirectUrl;
}
I was following the example from https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLJavaSDK.html to create pre-signed s3 urls (v4) and I get Access Denied error when I try to access the signed the url
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>0FB02ECDDF5EAC7B</RequestId>
<HostId>vA+mmsv9PCunNe5uPkPrmpqqN3vFctQ13c9dIRlKWTYsT0zNA1V9g+4YS+lCItrBlyQtdHpyspg=</HostId>
</Error>
The following is the code snippet
public class GeneratePresignedURL {
public static void main(String[] args) throws IOException {
String clientRegion = "us-east-1";
String bucketName = "com-example-bucket";
String objectKey = "path/to/file.img"; // No leading `/`
// https://com-example-bucket.s3.amazonaws.com/path/to/file.img
try {
AWSCredentialsProvider awsCredentialsProvider = new DefaultAWSCredentialsProviderChain();
// Assuming that us-east-1 defaults to v4, couldn't find a way to set it explicitly
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(clientRegion)
.withCredentials(awsCredentialsProvider)
.build();
// Set the presigned URL to expire after 10 minutes.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 10;
expiration.setTime(expTimeMillis);
// Generate the presigned URL.
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucketName, objectKey)
.withMethod(HttpMethod.GET)
.withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toString());
/*
sample signature:
https://com-example-bucket.s3.amazonaws.com/path/to/file.img?X-Amz-Security-Token=FQoDYXdzEDcaDLjUOdj2hDTZvWUQaiK3AxulqM%2BOPlp%2Bnq71P0LyuI0vj8tT%2F9i24Wd3jY8dUbudWbhUH9IAsPnl7asujO90GlaFP4dXujDDLwIakMjCJSfOFM4IoGJz8XtcjXkqJCNaenbrTA%2F3PfSl%2Fe9wQwJlY8gOu8%2Byioq2ElHULMKv52nEZj8s3v4dD0pGHQTYc4hGV7ty9CYwXNgz6w3TREhxuFdAewNgTRnY1uFNy7on6NDF5IE15vlJ2PxqrX53ZMLKP%2FdU8i5BcpZ3ySVhNpBpU3GJAPMOh%2B2ztCAk1zPjW4G0N5n9BlnjTMGs3vGBb9IW%2F8dzAoxaG9U9%2B%2FCp8euJN562dYYSZ9wmQgsfOVqc5OksdnHVkPJW400ObOcKmc9mqIRyqA%2B3Mv4z0%2Fx6iLYRJ3UaloFSGbmR6VlIxMl%2F67aHrmCnBE23a1%2BNMWgzLx%2FogqZy3CD%2F%2Fs6Jt1qkxUrRwC0RPK93LHD74qm8rjqZcEKFrBOrZsYtcl3zKgRIEHCbatQ7dwT634sdF0MwaD0vwTsbsStZDW903k5C%2FDuz4rEmkPv6c5CmFvxp4xOkUtMbDk4B8Z641CoeAMMOKICH%2FlW7%2F1as3nQo07Ow2QU%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180621T214222Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3000&X-Amz-Credential=<access_key>%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=<signature>
*/
}
catch(AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
}
catch(SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}
}
The signature format seems to be correct and I am not sure if I am missing some other s3 client config.
Since I don't get a signature mismatch error or invalid url error, I assume that the Access Key used to sign the url is correct.
Here are steps to generate aws-s3 pre-signed url to access the content stored in s3 through java can create with simple step
First add maven dependency in your pom
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.870</version>
</dependency>
Get S3 credential accessKey, secretKey, region of s3 storage it must needed
Write java class
First get s3Client connection using below code snippet providing your own credential
BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKey, secretKey);
final AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new
AWSStaticCredentialsProvider(awsCreds)).withRegion(region).build();
You must have your bucketName if you don't have create bucket and store your content
Use below code snippet to generate pre-signed url
// Set the expiry time
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
Pass your objectKey is noting but your existed s3 base filename
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucketName, objecKey)
.withMethod(HttpMethod.GET)
.withExpiration(expiration);
URL url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toString());
It is due to the region mismatch. You have set clientRegion to be us-east-1 but if you are accessing it from any other region it will give access denied error. You can only access the object from us-east-1 region or just change it to your region. It's ap-south-1 for Indian clients.
The issue seems to be that the role didn't have permissions to access the s3 bucket path/to/file
Try to add .withPathStyleAccessEnabled(true) as mentioned in below snap.
// Assuming that us-east-1 defaults to v4, couldn't find a way to set it explicitly
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(clientRegion)
.withPathStyleAccessEnabled(true)
.withCredentials(awsCredentialsProvider)
.build();
Change/check this change CROS properties of AWS S3 bucket.
<CORSConfiguration>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
</CORSRule>
</CORSConfiguration>
Be mindful of assigning the cognito auth-role with access to S3 else the permission will be denied in case you are using the cognito-credentials user pool session to provide the required access to s3client. This was an issue which i was facing as well.
import com.amazonaws.AmazonServiceException;
import com.amazonaws.HttpMethod;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.DeleteObjectRequest;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectSummary;
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.163</version>
</dependency>
String GeneratePresignedUrlAndUploadObject() {
String accesskey = "XXXXXXXXXXXXXXXXXXXXXX";
String secretkey = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
String bucketName = "your-bucket-Name";
Regions regions = Regions.US_EAST_1;
String objecKey = "your-file-name-you-need-url";
BasicAWSCredentials awsCreds = new BasicAWSCredentials(accesskey, secretkey);
//
final AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds)).withRegion(regions).build();
// Set the expiry time
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objecKey)
.withMethod(HttpMethod.GET).withExpiration(expiration);
//
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toString());
return url.toString();
}