Grails AWS SDK plugin not resolving PutObjectRequest - amazon-web-services

I am trying to get my grails app working with Amazon S3, I have been following the following docs... http://agorapulse.github.io/grails-aws-sdk/guide/single.html
At the following step amazonWebService.s3.putObject(new PutObjectRequest('some-grails-bucket', 'somePath/someKey.jpg', new File('/Users/ben/Desktop/photo.jpg')).withCannedAcl(CannedAccessControlList.PublicRead))
The project can't resolve class PutObjectRequest, and I have tried importing com.amazonaws.services.s3.model.PutObjectRequest manually, but it still cant find the class. The only thing I can think of is I might have an older version of the SDK, though I only followed the tutorial.
My BuildConfig.groovy...
...
dependencies{
//dependencies for amazon aws plugin
build 'org.apache.httpcomponents:httpcore:4.3.2'
build 'org.apache.httpcomponents:httpclient:4.3.2'
runtime 'org.apache.httpcomponents:httpcore:4.3.2'
runtime 'org.apache.httpcomponents:httpclient:4.3.2'
}
plugins{
...
runtime ':aws-sdk:1.9.40'
}
has anyone else run into this issue and have a solution?

I don't use the plugin, I simply just use the SDK directly. Not sure what you would need a plugin for. You don't need httpcomponents for it to work
Add this to you dependencies block:
compile('com.amazonaws:aws-java-sdk-s3:1.10.2') {
exclude group: 'com.fasterxml.jackson.core'
}
Heres my bean I use. I set the key, access, and bucket data in the bean configuration
class AmazonStorageService implements FileStorageService {
String accessKeyId
String secretAccessKey
String bucketName
AmazonS3Client s3client
#PostConstruct
private void init() {
s3client = new AmazonS3Client(new BasicAWSCredentials(accessKeyId, secretAccessKey));
}
String upload(String name, InputStream inputStream) {
s3client.putObject(new PutObjectRequest(bucketName, name, inputStream, null).withCannedAcl(CannedAccessControlList.PublicRead));
getUrl(name)
}
String upload(String name, byte[] data) {
upload(name, new ByteArrayInputStream(data))
}
String getUrl(String name) {
s3client.getUrl(bucketName, name)
}
Boolean exists(String name) {
try {
s3client.getObjectMetadata(bucketName, name)
true
} catch(AmazonServiceException e) {
false
}
}
}

Related

java.lang.ClassNotFoundException on ClasspathPropertiesFileCredentialsProvider in a Kotlin project for Windows

I created a console application for Windows in Kotlin.
In that app, I want to send files to an AWS S3 bucket.
For that purpose, I initiate my S3 client like the following:
// credentials provider :
val classpathPropertiesFileCredentialsProvider = ClasspathPropertiesFileCredentialsProvider("AWSCredentials.properties")
// S3 client :
val amazonS3: AmazonS3 = AmazonS3ClientBuilder.standard()
.withRegion("eu-central-1")
.withCredentials(classpathPropertiesFileCredentialsProvider)
.build()
And here are all the imports in my build.gradle file:
implementation 'com.amazonaws:aws-java-sdk-opensdk:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-core:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-apigatewayv2:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-cognitoidentity:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-cognitoidp:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-s3:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-sts:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-s3control:1.12.58'
The project compiles without any error, but when I run it, I get the following error:
java.lang.ClassNotFoundException: com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider
which I don't understand, since I did the same in another Java Swing project, and it works fine.
Did I miss something?
Thanks.
To work with Kotlin and AWS Services such as Amazon S3, you should consider using the AWS SDK for Kotlin. This is a newer SDK built for Kotlin developers. You are currently using the AWS SDK for Java V1. If you are using Java, then you should move from V1 to Java V2.
To learn how to get started with the AWS SDK for Kotlin, see this topic:
https://github.com/awslabs/aws-sdk-kotlin/blob/main/docs/GettingStarted.md
To work with Amazon S3 and Kotlin, refer to the code examples in Github here.
To upload a file to an Amazon S3 bucket using the Kotlin SDK, use this code:
package com.kotlin.s3
// snippet-start:[s3.kotlin.s3_object_upload.import]
import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.PutObjectRequest
import aws.sdk.kotlin.services.s3.model.S3Exception
import aws.smithy.kotlin.runtime.content.ByteStream
import java.io.File
import java.io.FileInputStream
import java.io.IOException
import kotlin.system.exitProcess
// snippet-end:[s3.kotlin.s3_object_upload.import]
/**
To run this Kotlin code example, ensure that you have setup your development environment,
including your credentials.
For information, see this documentation topic:
https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/setup.html
*/
suspend fun main(args: Array<String>) {
val usage = """
Usage:
<bucketName> <objectKey> <objectPath>
Where:
bucketName - the Amazon S3 bucket to upload an object into.
objectKey - the object to upload (for example, book.pdf).
objectPath - the path where the file is located (for example, C:/AWS/book2.pdf).
"""
if (args.size != 3) {
println(usage)
exitProcess(0)
}
val bucketName = args[0]
val objectKey = args[1]
val objectPath = args[2]
val s3Client = S3Client { region = "us-east-1" }
putS3Object(s3Client, bucketName, objectKey, objectPath)
s3Client.close()
}
// snippet-start:[s3.kotlin.s3_object_upload.main]
suspend fun putS3Object(
s3Client: S3Client,
bucketName: String,
objectKey: String,
objectPath: String
){
try {
val metadataVal = mutableMapOf<String, String>()
metadataVal["myVal"] = "test"
val putOb = PutObjectRequest {
bucket = bucketName
key = objectKey
metadata = metadataVal
this.body = ByteStream.fromBytes(getObjectFile(objectPath))
}
val response =s3Client.putObject(putOb)
println("Tag information is ${response.eTag}")
} catch (e: S3Exception) {
println(e.message)
s3Client.close()
exitProcess(0)
}
}
fun getObjectFile(filePath: String): ByteArray {
var fileInputStream: FileInputStream? = null
lateinit var bytesArray: ByteArray
try {
val file = File(filePath)
bytesArray = ByteArray(file.length().toInt())
fileInputStream = FileInputStream(file)
fileInputStream.read(bytesArray)
} catch (e: IOException) {
println(e.message)
} finally {
if (fileInputStream != null) {
try {
fileInputStream.close()
} catch (e: IOException) {
println(e.message)
}
}
}
return bytesArray
}
NOTE that all examples assume that your creds are located in a file named credentials as explained here:
Setting the default credentials

Access Amazon S3 public bucket

Hello I am trying to download data from one of an Amazon S3 public bucket.
For example https://registry.opendata.aws/noaa-gfs-bdp-pds/
The bucket has web accessible folder and I want to download the files inside the bucket.
I know I can do this with AWS CLI tool
But I want to know if there anyway to do this with AWs SDK Api (s3 client) (c# visual studio)?
I think the issue is authentication when creating connection to s3 client it requires credentials like access key ,I don't have an AWS account,and the bucket I try to get to is public so
Does anyone know how to access to this public bucket without any credentials via API?
Thanks.
If you specify the AnonymousAWSCredentials as the credentials object, any requests that are made to S3 will be unsigned. After that, interacting with the bucket is done like any other call:
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using System;
namespace S3TestApp
{
class Program
{
static void Main(string[] args)
{
var unsigned = new AnonymousAWSCredentials();
var client = new AmazonS3Client(unsigned, Amazon.RegionEndpoint.USEast1);
var listRequest = new ListObjectsRequest
{
BucketName = "noaa-gfs-bdp-pds",
Delimiter = "/",
};
ListObjectsResponse listResponse;
do
{
listResponse = client.ListObjects(listRequest);
foreach (var obj in listResponse.CommonPrefixes)
{
Console.WriteLine("PRE {0}", obj);
}
foreach (var obj in listResponse.S3Objects)
{
Console.WriteLine("{0} {1}", obj.Size, obj.Key);
}
listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);
}
}
}

Automatically Grab Latest Google Cloud Platform Secret Version

I'm trying to grab the latest secret version. Is there a way to do that without specifying the version number? Such as using the keyword "latest". I'm trying to avoid having to iterate through all the secret versions with a for loop as GCP documentation shows:
try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
// Build the parent name.
SecretName projectName = SecretName.of(projectId, secretId);
// Get all versions.
ListSecretVersionsPagedResponse pagedResponse = client.listSecretVersions(projectName);
// List all versions and their state.
pagedResponse
.iterateAll()
.forEach(
version -> {
System.out.printf("Secret version %s, %s\n", version.getName(), version.getState());
});
}
Yes, you can use "latest" as the version number. This is called an "alias". At present, the only alias is "latest", but we may support more aliases in the future.
gcloud secrets versions access "latest" --secret "my-secret"
try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
SecretVersionName secretVersionName = SecretVersionName.of(projectId, secretId, "latest"); // <-- here
// Access the secret version.
AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
String payload = response.getPayload().getData().toStringUtf8();
System.out.printf("Plaintext: %s\n", payload);
}
import com.google.cloud.secretmanager.v1.AccessSecretVersionResponse;
import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretVersionName;
import java.io.IOException;
public class AccessSecretVersion {
public static void accessSecretVersion() throws IOException {
// TODO(developer): Replace these variables before running the sample.
String projectId = "your-project-id";
String secretId = "your-secret-id";
String versionId = "latest"; //<-- specify version
accessSecretVersion(projectId, secretId, versionId);
}
// Access the payload for the given secret version if one exists. The version
// can be a version number as a string (e.g. "5") or an alias (e.g. "latest").
public static void accessSecretVersion(String projectId, String secretId, String versionId)
throws IOException {
// Initialize client that will be used to send requests. This client only needs to be created
// once, and can be reused for multiple requests. After completing all of your requests, call
// the "close" method on the client to safely clean up any remaining background resources.
try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
SecretVersionName secretVersionName = SecretVersionName.of(projectId, secretId, versionId);
// Access the secret version.
AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
// Print the secret payload.
//
// WARNING: Do not print the secret in a production environment - this
// snippet is showing how to access the secret material.
String payload = response.getPayload().getData().toStringUtf8();
System.out.printf("Plaintext: %s\n", payload);
}
}
}
source: https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java

AWS SNS: Not sending sms to phone number

I am trying a small demo for sending SMS to numbers here in India. I used following code snippet but not working.
public void send(String to, String textBody) {
AWSCredentials awsCredentials;
awsCredentials = new EnvironmentVariableCredentialsProvider().getCredentials();
AmazonSNSClient snsClient = new AmazonSNSClient(awsCredentials);
PublishResult result = snsClient
.publish(new PublishRequest().withMessage(textBody)
.withPhoneNumber("+9195355*****"));
System.out.println(result);
}
I am not able to figure out what is missing?
Since you've confirmed that sending SMS using the SNS console,the messages are delivered successfully then this seems to suggest that there could be an issue with the provided code snippet!
I've modified your code snippet slightly and I've confirmed the following code snippet to be working without issues :
public static void main(String[] args) {
ProfileCredentialsProvider creds = new ProfileCredentialsProvider("syumaK");
AmazonSNS snsClient = AmazonSNSClientBuilder.standard().withCredentials(creds).withRegion("us-east-1").build();
String textBody = "My test SMS message from SNS";
String to = "+0123456789";
send(snsClient, to, textBody);
}
public static void send(AmazonSNS snsClient, String to, String textBody) {
try {
PublishResult result = snsClient
.publish(new PublishRequest()
.withMessage(textBody)
.withPhoneNumber(to));
System.out.println(result);
LOGGER.info("Message send with id {}."+ result.getMessageId());
} catch (AmazonSNSException e) {
LOGGER.info("Got an exception. " + e.getMessage())
}
}
Troubleshooting Steps:
-If using ProfileCredentialsProvider works just like above, consider switching back to EnvironmentVariableCredentialsProvider or using BasicAWSCredentials provider and check if you observe any success/failure?
I have tested the above code snippet using the following environment spec:
OS : Ubuntu 16.04
aws region: us-east-1
aws-java-sdk: "1.11.723"

How to prepare Pub/Sub emulator for tests?

I start the gcloud sdk docker
docker run -ti --rm --expose=8085 -p 8085:8085 google/cloud-sdk:latest
then i run:
gcloud beta emulators pubsub start --project=my-project --host-port=0.0.0.0:8085
then stop the sever and then:
gcloud beta emulators pubsub env-init
gives:
export PUBSUB_EMULATOR_HOST=0.0.0.0:8085
but there is no project id. How can I setup project for tests? How can i create topics and subscriptions?
version:
gcloud version
gives:
Google Cloud SDK 236.0.0
...
pubsub-emulator 2019.02.22
You are launching pubsub emulator with project my-project in your 2nd command. Once this is running, don't kill it, leave it running.
To create the topics and subscriptions, you have to use one of the SDKs. I created a demo project that does this using the Java SDK: https://github.com/nhartner/pubsub-emulator-demo/
The relevant code is this:
#Component
public class TestPubSubConfig {
private final TransportChannelProvider channelProvider;
private final CredentialsProvider credentialsProvider;
private String projectId;
private String topicName = "test-topic";
private String subscriptionName = "test-subscription";
TestPubSubConfig(#Autowired #Value("${spring.cloud.gcp.pubsub.emulator-host}") String emulatorHost,
#Autowired #Value("${spring.cloud.gcp.project-id}") String projectId) throws IOException {
this.projectId = projectId;
ManagedChannel channel = ManagedChannelBuilder.forTarget(emulatorHost).usePlaintext().build();
channelProvider = FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
credentialsProvider = NoCredentialsProvider.create();
createTopic(topicName);
createSubscription(topicName, subscriptionName);
}
#Bean
public Publisher testPublisher() throws IOException {
return Publisher.newBuilder(ProjectTopicName.of(projectId, topicName))
.setChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build();
}
private void createSubscription(String topicName, String subscriptionName) throws IOException {
ProjectTopicName topic = ProjectTopicName.of(projectId, topicName);
ProjectSubscriptionName subscription = ProjectSubscriptionName.of(projectId, subscriptionName);
try {
subscriptionAdminClient()
.createSubscription(subscription, topic, PushConfig.getDefaultInstance(), 100);
}
catch (AlreadyExistsException e) {
// this is fine, already created
}
}
private void createTopic(String topicName) throws IOException {
ProjectTopicName topic = ProjectTopicName.of(projectId, topicName);
try {
topicAdminClient().createTopic(topic);
}
catch (AlreadyExistsException e) {
// this is fine, already created
}
}
private TopicAdminClient topicAdminClient() throws IOException {
return TopicAdminClient.create(
TopicAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider).build());
}
private SubscriptionAdminClient subscriptionAdminClient() throws IOException {
return SubscriptionAdminClient.create(SubscriptionAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build());
}
}
A possible gotchya we uncovered while working with the Pub/Sub emulator is that the documentation says:
In this case, the project ID can be any valid string; it does not
need to represent a real GCP project because the Cloud Pub/Sub
emulator runs locally.
any valid string in this context is not any string, but specifically a valid one, meaning it looks like a valid GC Project Id. In our testing this was specifically strings that match the REGEX pattern:
/^[a-z]-[a-z]-\d{6}$/
Once supplied with a valid project ID, the emulator works as advertised. If you have a sandbox project in GC you can use that ID or you can make up your own that matches that pattern. Once you got that far you can follow the remainder of the Testing apps locally with the emulator documentation.