Upload files into s3 using AWS Lambda function (Java) - amazon-web-services

I am currently making a react native application with aws service. I have few doubts about AWS Service use case and I searched and read about AWS service with my use cases but I didn't get any exact solution.
Here is my question,
I have to store photos and videos into S3 buckets. So I am making a simple mobile application that has access for getting photos and videos from gallery and making a API call to trigger AWS Lambda function (written in Java) to upload those photos and videos into the respective S3 buckets. But my question is, whenever I try to store all those photos and videos into S3 bucket using AWS Lambda function I keep getting no such file or directory exception. I know the reason why I am getting this exception because those photos and Videos are only available in local storage not in AWS cloud (ref Upload Image into AWS S3 bucket using Aws Lambda). So Is there any way to overcome this error? or else I need to get EC2 instance to store all the photos and videos?
My sample Lambda Function code (Upload Image into S3 bucket)
package com.amazonaws.lambda.demo;
import java.io.File;
import java.io.IOException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
public class LambdaFunctionHandler implements RequestHandler<String, PutObjectResult> {
#SuppressWarnings("deprecation")
public PutObjectResult UploadFile(String imageURI) throws IOException {
PutObjectResult resultObj = new PutObjectResult();
AWSCredentials credentials = new BasicAWSCredentials("Ak", "Sk");
//String clientRegion = "region";
String bucketName = "bucket_name";
String fileObjKeyName = "sample_image_pic.jpg";
try {
AmazonS3 s3Client = new AmazonS3Client(credentials);
// Upload a file as a new object with ContentType and title specified.
resultObj = s3Client.putObject(new PutObjectRequest(bucketName, fileObjKeyName, new File(imageURI)));
}
catch(AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
}
catch(SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
return resultObj;
}
#Override
public PutObjectResult handleRequest(String imageURI, Context context) {
PutObjectResult result = null;
try {
result = UploadFile(imageURI);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return result;
}
}
Thanks,

Related

Amazon SES Exception: IAM User is not authorized to perform `ses:SendRawEmail' on resource

I am new to AWS, I was trying to deploy Springboot application in AWS EC2 instance using Elastic beanstalk. I wanted to make use of AWS SES service to send notifications to the subscribed application users.
As part of this, with the help of AWS SDK SES API and the IAM credentials I was able to send the email for verified user emails using springboot application, but I wanted to send the emails for non-verified users as well, So I have requested AWS support team to get my IAM user out of AWS SES Sandbox and increase daily limit of sending emails and AWS support team honored the request.
After moving my IAM user out of SES Sandbox, when I have tried to send the email for verified and non-verified users I am getting following error.
org.springframework.mail.MailSendException: Failed messages: com.amazonaws.services.simpleemail.model.AmazonSimpleEmailServiceException:
User `arn:aws:sts::4***436****2:assumed-role/aws-elasticbeanstalk-ec2-role/i-066edeefc2ed72b10' is not authorized to perform
`ses:SendRawEmail' on resource `arn:aws:ses:ap-south-1:4***436****2:identity/ra******ar#gmail.com'
(Service: AmazonSimpleEmailService; Status Code: 403; Error Code: AccessDenied; Request ID: 5eeb1f17-a283-4d9f-bca9-ac981ee546c4;
Proxy: null); message exception details (1) are:
Failed message 1:
com.amazonaws.services.simpleemail.model.AmazonSimpleEmailServiceException:
User `arn:aws:sts::4***436****2:assumed-role/aws-elasticbeanstalk-ec2-role/i-066edeefc2ed72b10' is not authorized to perform
`ses:SendRawEmail' on resource `arn:aws:ses:ap-south-1:4***436****2:identity/ra******ar#gmail.com#gmail.com'
(Service: AmazonSimpleEmailService; Status Code: 403; Error Code: AccessDenied; Request ID: 5eeb1f17-a283-4d9f-bca9-ac981ee546c4; Proxy: null)
pom.xml:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-aws</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-ses</artifactId>
<version>1.11.777</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-mail</artifactId>
<version>2.2.6.RELEASE</version>
</dependency>
AWS Mail Configuration class:
import org.springframework.cloud.aws.mail.simplemail.SimpleEmailServiceJavaMailSender;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.mail.javamail.JavaMailSender;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailService;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClientBuilder;
#Configuration
public class AwsMailConfig {
// Used to fetch AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
private CustomPropertyConfig customPropertyConfig;
public AwsMailConfig(final CustomPropertyConfig customPropertyConfig) {
this.customPropertyConfig = customPropertyConfig;
}
#Profile("prod")
#Bean("AWSCredentialsProvider")
public AWSCredentialsProvider amazonAWSCredentialsProviderProduction() {
return new EC2ContainerCredentialsProviderWrapper();
}
#Bean
#Profile("prod")
public AmazonSimpleEmailService amazonSimpleEmailServiceProduction() {
return AmazonSimpleEmailServiceClientBuilder.standard()
.withRegion(Regions.AP_SOUTH_1)
.build();
}
#Bean
public JavaMailSender javaMailSender(AmazonSimpleEmailService amazonSimpleEmailService) {
return new SimpleEmailServiceJavaMailSender(amazonSimpleEmailService);
}
}
Email Sender service class:
import java.nio.charset.StandardCharsets;
import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;
import org.springframework.mail.javamail.JavaMailSender;
import org.springframework.mail.javamail.MimeMessageHelper;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
import com.ebike.aws.emailutil.Mail;
#Service
public class AWSEmailSenderService {
private JavaMailSender javaMailSender;
public AWSEmailSenderService(JavaMailSender javaMailSender) {
this.javaMailSender = javaMailSender;
}
#Async
public void sendEmail(Mail mail) throws MessagingException {
try {
MimeMessage message = getMimeMessage(mail);
javaMailSender.send(message);
} catch (Exception e) {
e.printStackTrace();
}
}
private MimeMessage getMimeMessage(Mail mail) throws MessagingException {
MimeMessage message = javaMailSender.createMimeMessage();
MimeMessageHelper helper = new MimeMessageHelper(message, MimeMessageHelper.MULTIPART_MODE_MIXED_RELATED,
StandardCharsets.UTF_8.name());
String html = mail.getEmailBody();
helper.setTo(mail.getTo());
helper.setText(html, true);
helper.setSubject(mail.getSubject());
helper.setFrom(mail.getFrom());
return message;
}
}
Mail.java has a method to get Mail Object
public Mail getMail() {
Mail mail = new Mail();
mail.setFrom(fromEmailID);
mail.setTo(toEmailId);
mail.setSubject(subject);
mail.setEmailBody(emailBody);
return mail;
}
After getting IAM user is not authorized to perform `ses:SendRawEmail' Exception, I have tried adding policy to grant all possible permissions but I know I have to modify in policy itself but after searching for a while I couldn't figure out and I have gone through this link but it didn't help or I was not able to understand.
Currently IAM User has following AWS policies:
I don't know, is the following is proper place to apply custom policy or not
Please help me to know to send emails for non-verified email-IDs using AWS SES and granting all possible permissions or policy for IAM user.
This error is not about your IAM user, but about aws-elasticbeanstalk-ec2-role role for Elastic Beanstalk.
Thus you have to go to IAM roles, find the role in question, and add the required permissions to it.

Getting error while generating presigned url for SSE-C

I have created a customer key for AWS S3 Server side encryption using customer key (SSE-C).
I am able to upload the object using the key. But when I generate a presigned URL using AWS Java SDK the URL is getting created successfully but when I hit that URL I am getting the below error.
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.
import java.io.IOException;
import java.net.URI;
import java.net.URL;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.FileEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.HttpMethod;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.Headers;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.SSEAlgorithm;
import com.amazonaws.services.s3.model.SSECustomerKey;
import com.amazonaws.util.Base64;
import com.amazonaws.util.Md5Utils;
public class GeneratePresignedURL {
public GeneratePresignedURL() throws IOException {
String bucketName = "abctest";
String keyName = "testnew.mp4";
try {
SSECustomerKey SSE_KEY = new SSECustomerKey("KLgsVafKowMCfKDsbIh597CmMUSoPBn6QJ8OIGxAMBw=");
ClientConfiguration cnf = new ClientConfiguration();
cnf.withSignerOverride("AWSS3V4SignerType");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withClientConfiguration(cnf)
.withCredentials(new ProfileCredentialsProvider()).build();
// Set the presigned URL to expire after one hour.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
// Generate the presigned URL.
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName,
keyName).withMethod(HttpMethod.PUT).withExpiration(expiration).withSSECustomerKey(SSE_KEY);
generatePresignedUrlRequest.setSSECustomerKeyAlgorithm(SSEAlgorithm.AES256);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toExternalForm());
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}
I have tried to follow the code from https://aws.amazon.com/blogs/developer/generating-amazon-s3-pre-signed-urls-with-sse-c-part-5-finale/
I have no problem in using pre-signed url for SSE S3, I am only facing issue with SSE-C
I have tried setting default encryption and other config but of no help. Any pointers would be of great help.
Thanks,
AK
You can not call pre-signed URL direct via browser
You need to pass headers in the request
x-amz-server-side-encryption-customer-algorithm
x-amz-server-side-encryption-customer-key
x-amz-server-side​-encryption​-customer-key-MD5
Please check the document for more information
https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
When using the presigned URL to upload a new object, retrieve an existing object, or retrieve only object metadata, you must provide all the encryption headers in your client application.
Please check the sample code calling signed URL via Unirest library
https://github.com/pavanpawar4591/s3signurlwith-sse-c
public static void getPreSignedURL() throws URISyntaxException {
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyName)
.withMethod(HttpMethod.GET).withExpiration(expiration).withSSECustomerKey(SSE_KEY);
// generatePresignedUrlRequest.setContentType("video/mp4");
generatePresignedUrlRequest.setSSECustomerKeyAlgorithm(SSEAlgorithm.AES256);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toURI() + " With key: " + SSE_KEY);
System.out.println("------------------------------");
//https://aws.amazon.com/blogs/developer/generating-amazon-s3-pre-signed-urls-with-sse-c-part-4/
// refer to above doc
try {
HttpResponse<String> response = Unirest.get(url.toURI().toString())
.header(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, SSEAlgorithm.AES256.getAlgorithm())
.header(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, Base64.encodeAsString(SECRET_KEY.getEncoded()))
.header(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5,
Md5Utils.md5AsBase64(SECRET_KEY.getEncoded()))
.header("cache-control", "no-cache").header("postman-token", "d3453c38-1b59-a12e-fd97-dbe2150eadf5")
.asString();
System.out.println(response.getStatus());
System.out.println(response.getStatusText());
System.out.println(response.getBody());
} catch (UnirestException e) {
e.printStackTrace();
}
}
Thanks #pavan
Any client that needs to use SSE-C, the client must be capable of sending the Below headers.
When using the presigned URL to retrieve an existing object, or retrieve only object metadata, we need to provide all the encryption headers in your client application.
For S3 managed or KMS managed server side encryption, we can generate a presigned URL and directly paste that into a browser or player.
However, this is not true for SSE-C objects because in addition to
the presigned URL, you also need to include HTTP headers that are
specific to SSE-C objects. Therefore, you can use the presigned URL
for SSE-C objects only programmatically.
Upload:
I opted for S3 managed keys instead of customer provided client key.
FileInputStream fin = new FileInputStream(uploadFileName);
byte fileContent[] = new byte[(int) uploadFileName.length()];
// Reads up to certain bytes of data from this input stream into an array of
// bytes.
fin.read(fileContent);
// create string from byte array
// Specify server-side encryption.
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(fileContent.length);
objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName,
new ByteArrayInputStream(fileContent), objectMetadata);
// Upload the object and check its encryption status.
PutObjectResult putResult = s3Client.putObject(putRequest);
System.out.println("Object \"" + keyName + "\" uploaded with SSE.");
GET Presigned URL:
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyName)
.withMethod(HttpMethod.GET).withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toURI());
Reference:
https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html

Amazon SES 403 error

I am trying to use Amazon SES to send email to my corporate account. I have copied by aws access keys to ~/.aws/credentials file. Seems i am consistently getting the error:
Error message: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'POST
/
amz-sdk-invocation-id:638ed10a-22cd-12f5-c9a5-25a672a6c38d
amz-sdk-retry:3/136/485
host:email.us-east-1.amazonaws.com
user-agent:aws-sdk-java/1.11.172 Mac_OS_X/10.11.5 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60
x-amz-date:20170808T040309Z
amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date
1c21e3af09924eec311e370c8e6710c0bc3fa2027fe213db11f02b22d79b6c91'
The String-to-Sign should have been
'AWS4-HMAC-SHA256
20170808T040309Z
20170808/us-east-1/ses/aws4_request
82a2179c0dda36b0a4f80c111a8119cc8a65d768a0ff65de4c9b7ccf69a6b68a' (Service: AmazonSimpleEmailService; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 7dbfbcaa-7bee-11e7-a445-27be327c79a0)
com.amazonaws.services.simpleemail.model.AmazonSimpleEmailServiceException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'POST
/
amz-sdk-invocation-id:638ed10a-22cd-12f5-c9a5-25a672a6c38d
amz-sdk-retry:3/136/485
host:email.us-east-1.amazonaws.com
user-agent:aws-sdk-java/1.11.172 Mac_OS_X/10.11.5 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60
x-amz-date:20170808T040309Z
amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date
1c21e3af09924eec311e370c8e6710c0bc3fa2027fe213db11f02b22d79b6c91'
The String-to-Sign should have been
'AWS4-HMAC-SHA256
20170808T040309Z
20170808/us-east-1/ses/aws4_request
82a2179c0dda36b0a4f80c111a8119cc8a65d768a0ff65de4c9b7ccf69a6b68a' (Service: AmazonSimpleEmailService; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 7dbfbcaa-7bee-11e7-a445-27be327c79a0)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1587)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257)
Code base:
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintStream;
import java.nio.ByteBuffer;
import java.util.Properties;
import java.util.UUID;
import javax.activation.DataHandler;
import javax.activation.DataSource;
import javax.activation.FileDataSource;
import javax.mail.Address;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Session;
import javax.mail.internet.AddressException;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeBodyPart;
import javax.mail.internet.MimeMessage;
import javax.mail.internet.MimeMultipart;
import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClient;
import com.amazonaws.services.simpleemail.model.RawMessage;
import com.amazonaws.services.simpleemail.model.SendRawEmailRequest;
public class SESClient {
private static String EMAIL_FROM = "##########";
private static String EMAIL_REPLY_TO = "##########";
private static String EMAIL_RECIPIENT = "##########";
// Remember to use two slashes in place of each slash.
private static Regions AWS_REGION = Regions.US_EAST_1;
private static String EMAIL_SUBJECT = "Amazon SES email test";
private static String EMAIL_BODY_TEXT = "This MIME email was sent through Amazon SES using SendRawEmail.";
public static void main(String[] args) throws AddressException, MessagingException, IOException {
Session session = Session.getDefaultInstance(new Properties());
MimeMessage message = new MimeMessage(session);
message.setSubject(EMAIL_SUBJECT, "UTF-8");
message.setFrom(new InternetAddress(EMAIL_FROM));
message.setReplyTo(new Address[]{new InternetAddress(EMAIL_REPLY_TO)});
message.setRecipients(Message.RecipientType.TO, InternetAddress.parse(EMAIL_RECIPIENT));
// Cover wrap
MimeBodyPart wrap = new MimeBodyPart();
// Alternative TEXT/HTML content
MimeMultipart cover = new MimeMultipart("alternative");
MimeBodyPart html = new MimeBodyPart();
cover.addBodyPart(html);
wrap.setContent(cover);
MimeMultipart content = new MimeMultipart("related");
message.setContent(content);
content.addBodyPart(wrap);
String[] attachmentsFiles = new String[]{
//EMAIL_ATTACHMENTS
};
StringBuilder sb = new StringBuilder();
for (String attachmentFileName : attachmentsFiles) {
String id = UUID.randomUUID().toString();
sb.append("<img src=\"cid:");
sb.append(id);
sb.append("\" alt=\"ATTACHMENT\"/>\n");
MimeBodyPart attachment = new MimeBodyPart();
DataSource fds = new FileDataSource(attachmentFileName);
attachment.setDataHandler(new DataHandler(fds));
attachment.setHeader("Content-ID", "<" + id + ">");
attachment.setFileName(fds.getName());
content.addBodyPart(attachment);
}
html.setContent("<html><body><h1>HTML</h1>\n" + EMAIL_BODY_TEXT + "</body></html>", "text/html");
try {
System.out.println("Attempting to send an email through Amazon SES by using the AWS SDK for Java...");
/*
* The ProfileCredentialsProvider will return your [default]
* credential profile by reading from the credentials file
* located at
* (~/.aws/credentials).
*
* TransferManager manages a pool of threads, so we create a
* single instance and share it throughout our application.
*/
AWSCredentials credentials = null;
try {
credentials = new ProfileCredentialsProvider().getCredentials();
} catch (Exception e) {
throw new AmazonClientException(
"Cannot load the credentials from the credential profiles file. " +
"Please make sure that your credentials file is at the correct " +
"location (~/.aws/credentials), and is in valid format.",
e);
}
AmazonSimpleEmailServiceClient client = new AmazonSimpleEmailServiceClient(credentials);
Region REGION = Region.getRegion(AWS_REGION);
client.setRegion(REGION);
// Print the raw email content on the console
PrintStream out = System.out;
message.writeTo(out);
// Send the email.
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
message.writeTo(outputStream);
RawMessage rawMessage = new RawMessage(ByteBuffer.wrap(outputStream.toByteArray()));
SendRawEmailRequest rawEmailRequest = new SendRawEmailRequest(rawMessage);
client.sendRawEmail(rawEmailRequest);
System.out.println("Email sent!");
} catch (Exception ex) {
System.out.println("Email Failed");
System.err.println("Error message: " + ex.getMessage());
ex.printStackTrace();
}
}
}

Spring Cloud for AWS - how to convert a message received from SQS and sent by S3

I have configured S3 to fire an event every time a file is uploaded. The event goes to SQS. Now I want to read messages from the queue with Spring Cloud for AWS. Is there a way to convert the payload to a known type which represents S3 Event record? I've tried with
package com.test;
import java.util.Map;
import org.springframework.messaging.handler.annotation.Headers;
import org.springframework.messaging.handler.annotation.MessageMapping;
import com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord;
public class SQSHandler {
#MessageMapping("MediaQueue")
private void receiveMessage(S3EventNotificationRecord message, #Headers Map<String, String> headers) {
System.out.println("MYK MYK MYK");
}
}
but I get error
ord, message=GenericMessage [payload={here was whole payload}]
at org.springframework.messaging.handler.annotation.support.PayloadArgumentResolver.resolveArgument(PayloadArgumentResolver.java:115)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:77)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:137)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:106)
at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMatch(AbstractMethodMessageHandler.java:447)
... 8 more
Do you know any method to do something like this?
The solution to the problem is following:
import com.amazonaws.services.s3.event.S3EventNotification;
#MessageMapping("${mediaQueueName}")
private void receiveMessage(String message, #Header(value = "MessageId") String messageId) {
S3EventNotification notification = S3EventNotification.parseJson(message);
//do whatever you need
}
According to the Spring Cloud Messaging docs, you need to configure your own MappingJackson2MessageConverter with strictContentTypeMatch set to false.
Then your code should work:
#MessageMapping("MediaQueue")
private void receiveMessage(S3EventNotificationRecord message, #Headers Map<String, String> headers) {
System.out.println("MYK MYK MYK");
}

File Upload to Amazon S3 via Groovy

I am trying to upload a file to Amazon S3 bucket using Groovy script. I tried the following code
#Grab( 'net.java.dev.jets3t:jets3t:0.9.0' )
import org.jets3t.service.impl.rest.httpclient.RestS3Service
import org.jets3t.service.security.AWSCredentials
import org.jets3t.service.model.*
import java.io.*;
bucketName='bucketname'
accessKey='accesskey'
secretKey='secretkey'
folder='D:/'
public putS3() {}
def login = new AWSCredentials( accessKey, secretKey )
def expiry = new GregorianCalendar( 2011,0,1 ).time
def s3 = new RestS3Service( login )
def bucket = new S3Bucket( bucketName )
args.each{fileName->
def key="$folder/$fileName"
def s3obj=new S3Object(bucket,newFile('D:/sample.txt'))
s3obj.key = key
println "\nUploading $fileName to $bucketName/$key"
s3obj = s3.putObject( bucket, s3obj )
def link = s3.createSignedGetUrl( bucketName, key, login, expiry, false )
  println "$fileName : $link"
}
code in the args block is not getting executed. When I execute this in Groovy Console it displays the result as []. Kindly help me where am I going wrong?
I don't have an S3 account to test with but here's a simplied example based upon the documenation:
import org.jets3t.service.impl.rest.httpclient.RestS3Service
import org.jets3t.service.model.S3Bucket
import org.jets3t.service.model.S3Object
import org.jets3t.service.security.AWSCredentials
#Grab('net.java.dev.jets3t:jets3t:0.9.0')
accessKey = 'accesskey'
secretKey = 'secretkey'
bucketName = 'bucketname'
fileName = 'D:\\sample.txt'
credentials = new AWSCredentials(accessKey, secretKey)
service = new RestS3Service(credentials)
bucket = new S3Bucket(bucketName)
file = new File(fileName)
fileObject = new S3Object(file)
fileObject.key = fileName
service.putObject(bucket, fileObject)
expiryTime = new Date() + 1 // 24 hours from current date
link = service.createSignedGetUrl(bucket.name, fileObject.key, expiryTime)
println "$fileName : $link"
I came accross that example recently as I attempt to code a quick groovy to upload to a S3 Bucket, however, all my attempts have ended up with 301
org.jets3t.service.S3ServiceException: Service Error Message. -- ResponseCode: 301, ResponseStatus: Moved Permanently, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>bucketname.s3.amazonaws.com</Endpoint><Bucket>bucketname</Bucket><RequestId>4CF5EDE9EF604DBB</RequestId><HostId>89KoQLvd93pXhnxJGcEaziSrSOPFRNXqbfPfY7LTe03z5rvVLAVx7UnFkts/Qe1fQ7eOWsaAL7A=</HostId></Error>
at org.jets3t.service.S3Service.putObject(S3Service.java:2358)
at org.jets3t.service.S3Service$putObject.call(Unknown Source)
at awsBucketDrop.run(awsBucketDrop.groovy:21)
At first I though this was a bucket location issue, as I found reference around 301 for that, however I've modified the code to swap to the bucket location, to no avail.
import org.jets3t.service.model.S3Bucket
import org.jets3t.service.model.S3Object
import org.jets3t.service.security.AWSCredentials
#Grab('net.java.dev.jets3t:jets3t:0.9.4')
accessKey = '<key>'
secretKey = '<secret>'
bucketName = '<bucketname>'
fileName = '<fileLocation>'
credentials = new AWSCredentials(accessKey, secretKey)
service = new RestS3Service(credentials)
bucket = new S3Bucket(bucketName,"eu-west-1")
println bucket.getLocation()
file = new File(fileName)
fileObject = new S3Object(file)
fileObject.key = fileName
service.putObject(bucket, fileObject)
expiryTime = new Date() + 1 // 24 hours from current date
link = service.createSignedGetUrl(bucket.name, fileObject.key, expiryTime)
println "$fileName : $link"
Now, this sdk has not been updated since 2015, so it's a fairly old SDK so I get the feeling it's no longer compatible (and might have some url hard coded into it) but if you had similar experience that you managed to solve out, let me know.
Thanks
If you come accross this and need a working groovy, I've loosely modified this code to work as a simple groovy
import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import java.io.File;
import java.io.IOException;
Regions clientRegion = Regions.EU_WEST_1;
String bucketName = "bucketname";
String stringObjKeyName = "stringObjKeyName";
String fileObjKeyName = "fileObjKeyName";
String fileName = "fileLocation";
try {
//This code expects that you have AWS credentials set up per:
// https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(clientRegion)
.build();
// Upload a text string as a new object.
s3Client.putObject(bucketName, stringObjKeyName, "Uploaded String Object");
// Upload a file as a new object with ContentType and title specified.
PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName));
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("plain/text");
metadata.addUserMetadata("x-amz-meta-title", "someTitle");
request.setMetadata(metadata);
s3Client.putObject(request);
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}