I created a console application for Windows in Kotlin.
In that app, I want to send files to an AWS S3 bucket.
For that purpose, I initiate my S3 client like the following:
// credentials provider :
val classpathPropertiesFileCredentialsProvider = ClasspathPropertiesFileCredentialsProvider("AWSCredentials.properties")
// S3 client :
val amazonS3: AmazonS3 = AmazonS3ClientBuilder.standard()
.withRegion("eu-central-1")
.withCredentials(classpathPropertiesFileCredentialsProvider)
.build()
And here are all the imports in my build.gradle file:
implementation 'com.amazonaws:aws-java-sdk-opensdk:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-core:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-apigatewayv2:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-cognitoidentity:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-cognitoidp:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-s3:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-sts:1.12.58'
implementation 'com.amazonaws:aws-java-sdk-s3control:1.12.58'
The project compiles without any error, but when I run it, I get the following error:
java.lang.ClassNotFoundException: com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider
which I don't understand, since I did the same in another Java Swing project, and it works fine.
Did I miss something?
Thanks.
To work with Kotlin and AWS Services such as Amazon S3, you should consider using the AWS SDK for Kotlin. This is a newer SDK built for Kotlin developers. You are currently using the AWS SDK for Java V1. If you are using Java, then you should move from V1 to Java V2.
To learn how to get started with the AWS SDK for Kotlin, see this topic:
https://github.com/awslabs/aws-sdk-kotlin/blob/main/docs/GettingStarted.md
To work with Amazon S3 and Kotlin, refer to the code examples in Github here.
To upload a file to an Amazon S3 bucket using the Kotlin SDK, use this code:
package com.kotlin.s3
// snippet-start:[s3.kotlin.s3_object_upload.import]
import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.PutObjectRequest
import aws.sdk.kotlin.services.s3.model.S3Exception
import aws.smithy.kotlin.runtime.content.ByteStream
import java.io.File
import java.io.FileInputStream
import java.io.IOException
import kotlin.system.exitProcess
// snippet-end:[s3.kotlin.s3_object_upload.import]
/**
To run this Kotlin code example, ensure that you have setup your development environment,
including your credentials.
For information, see this documentation topic:
https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/setup.html
*/
suspend fun main(args: Array<String>) {
val usage = """
Usage:
<bucketName> <objectKey> <objectPath>
Where:
bucketName - the Amazon S3 bucket to upload an object into.
objectKey - the object to upload (for example, book.pdf).
objectPath - the path where the file is located (for example, C:/AWS/book2.pdf).
"""
if (args.size != 3) {
println(usage)
exitProcess(0)
}
val bucketName = args[0]
val objectKey = args[1]
val objectPath = args[2]
val s3Client = S3Client { region = "us-east-1" }
putS3Object(s3Client, bucketName, objectKey, objectPath)
s3Client.close()
}
// snippet-start:[s3.kotlin.s3_object_upload.main]
suspend fun putS3Object(
s3Client: S3Client,
bucketName: String,
objectKey: String,
objectPath: String
){
try {
val metadataVal = mutableMapOf<String, String>()
metadataVal["myVal"] = "test"
val putOb = PutObjectRequest {
bucket = bucketName
key = objectKey
metadata = metadataVal
this.body = ByteStream.fromBytes(getObjectFile(objectPath))
}
val response =s3Client.putObject(putOb)
println("Tag information is ${response.eTag}")
} catch (e: S3Exception) {
println(e.message)
s3Client.close()
exitProcess(0)
}
}
fun getObjectFile(filePath: String): ByteArray {
var fileInputStream: FileInputStream? = null
lateinit var bytesArray: ByteArray
try {
val file = File(filePath)
bytesArray = ByteArray(file.length().toInt())
fileInputStream = FileInputStream(file)
fileInputStream.read(bytesArray)
} catch (e: IOException) {
println(e.message)
} finally {
if (fileInputStream != null) {
try {
fileInputStream.close()
} catch (e: IOException) {
println(e.message)
}
}
}
return bytesArray
}
NOTE that all examples assume that your creds are located in a file named credentials as explained here:
Setting the default credentials
Related
I know that there are many similar questions, and this one is no exception
But unfortunately I can't decide on the region for my case, how can I decide on the right region?
For example, when making a request to Postman, I encounter a similar error:
In my console i'm using EU (Frankfurt) eu-central-1 and also in terminal write smth like this:
heroku config:set region="eu-central-1"
And as I understand it, mine does not fit.
Also here is my AWS class:
class AmazonFileStorage : FileStorage {
private val client: S3Client
private val bucketName: String = System.getenv("bucketName")
init {
val region = System.getenv("region")
val accessKey = System.getenv("accessKey")
val secretKey = System.getenv("secretKey")
val credentials = AwsBasicCredentials.create(accessKey, secretKey)
val awsRegion = Region.of(region)
client = S3Client.builder()
.credentialsProvider(StaticCredentialsProvider.create(credentials))
.region(awsRegion)
.build() as S3Client
}
override suspend fun save(file: File): String =
withContext(Dispatchers.IO) {
client.putObject(
PutObjectRequest.builder().bucket(bucketName).key(file.name).acl(ObjectCannedACL.PUBLIC_READ).build(),
RequestBody.fromFile(file)
)
val request = GetUrlRequest.builder().bucket(bucketName).key(file.name).build()
client.utilities().getUrl(request).toExternalForm()
}
}
I think you may have the wrong region code; you do know that a Bucket is available in one and only one Region?
In your logging settings, set this scope to debug:
logging:
level:
org.apache.http.wire: debug
Then you should see something like this:
http-outgoing-0 >> "HEAD /somefile HTTP/1.1[\r][\n]"
http-outgoing-0 >> "Host: YOURBUCKETNAME.s3.eu-west-2.amazonaws.com[\r][\n]"
That log is from a bucket in the London region eu-west-2
To use Kotlin to interact with an Amazon S3 bucket (or other AWS services), consider using the AWS SDK for Kotlin. This SDK is meant for Kotlin developers. You are using the AWS SDK for Java.
To put an object into an Amazon S3 bucket using the AWS SDK for Kotlin, use this code. Notice the region that you want to use is specified in the code block where you define the aws.sdk.kotlin.services.s3.S3Client.
import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.PutObjectRequest
import aws.smithy.kotlin.runtime.content.asByteStream
import java.io.File
import kotlin.system.exitProcess
/**
Before running this Kotlin code example, set up your development environment,
including your credentials.
For more information, see the following documentation topic:
https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/setup.html
*/
suspend fun main(args: Array<String>) {
val usage = """
Usage:
<bucketName> <objectKey> <objectPath>
Where:
bucketName - The Amazon S3 bucket to upload an object into.
objectKey - The object to upload (for example, book.pdf).
objectPath - The path where the file is located (for example, C:/AWS/book2.pdf).
"""
if (args.size != 3) {
println(usage)
exitProcess(0)
}
val bucketName = args[0]
val objectKey = args[1]
val objectPath = args[2]
putS3Object(bucketName, objectKey, objectPath)
}
suspend fun putS3Object(bucketName: String, objectKey: String, objectPath: String) {
val metadataVal = mutableMapOf<String, String>()
metadataVal["myVal"] = "test"
val request = PutObjectRequest {
bucket = bucketName
key = objectKey
metadata = metadataVal
body = File(objectPath).asByteStream()
}
S3Client { region = "us-east-1" }.use { s3 ->
val response = s3.putObject(request)
println("Tag information is ${response.eTag}")
}
}
You can find this Kotlin example and many more in the AWS Code Library here:
Amazon S3 examples using SDK for Kotlin
ALso you can read the Kotlin DEV guide too. The link is at the start of the Code Example.
Hello I am trying to download data from one of an Amazon S3 public bucket.
For example https://registry.opendata.aws/noaa-gfs-bdp-pds/
The bucket has web accessible folder and I want to download the files inside the bucket.
I know I can do this with AWS CLI tool
But I want to know if there anyway to do this with AWs SDK Api (s3 client) (c# visual studio)?
I think the issue is authentication when creating connection to s3 client it requires credentials like access key ,I don't have an AWS account,and the bucket I try to get to is public so
Does anyone know how to access to this public bucket without any credentials via API?
Thanks.
If you specify the AnonymousAWSCredentials as the credentials object, any requests that are made to S3 will be unsigned. After that, interacting with the bucket is done like any other call:
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using System;
namespace S3TestApp
{
class Program
{
static void Main(string[] args)
{
var unsigned = new AnonymousAWSCredentials();
var client = new AmazonS3Client(unsigned, Amazon.RegionEndpoint.USEast1);
var listRequest = new ListObjectsRequest
{
BucketName = "noaa-gfs-bdp-pds",
Delimiter = "/",
};
ListObjectsResponse listResponse;
do
{
listResponse = client.ListObjects(listRequest);
foreach (var obj in listResponse.CommonPrefixes)
{
Console.WriteLine("PRE {0}", obj);
}
foreach (var obj in listResponse.S3Objects)
{
Console.WriteLine("{0} {1}", obj.Size, obj.Key);
}
listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);
}
}
}
I have stored AWS IAM user Access key's and Secret keys in a secret of AWS Secrets Manager.
This secret is helpful to get the data from an Amazon DynamoDB table, and keys's having full access to the Amazon DynamoDB table. I need to use this secret in java/.Net code to retrieve the data from DynamoDB table.
Secretname: dynamodbtesting
Below is the sample key names which I used while creating secret.
{
"aws_access_key_id": "value",
"aws_secret_access_key": "secret value"
}
How to use secret in java/.Net code to get the date from DynamoDB table?
Note: I could see one sample code after creation of secret in secret manager, is it helpful?
When using the AWS Java SDK, when you build the client which accesses dynamodb, you can pass credentials explicitly:
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-explicit
For example:
BasicAWSCredentials awsCreds = new BasicAWSCredentials("access_key_id", "secret_key_id");
AmazonS3 dynamodbClient = AmazonDynamoDBClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.build();
To answer your question: "How to use secret in java"
You can use the Secrets Manager Java API V2 to retrive a secret. The following Java code shows you how to perform this use case:
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;
import software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException;
//snippet-end:[secretsmanager.java2.get_secret.import]
/**
* To run this AWS code example, ensure that you have setup your development environment, including your AWS credentials.
*
* For information, see this documentation topic:
*
*https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class GetSecretValue {
public static void main(String[] args) {
final String USAGE = "\n" +
"Usage:\n" +
" GetSecretValue <secretName> \n\n" +
"Where:\n" +
" secretName - the name of the secret (for example, tutorials/MyFirstSecret). \n";
if (args.length != 1) {
System.out.println(USAGE);
System.exit(1);
}
String secretName = args[0];
Region region = Region.US_EAST_1;
SecretsManagerClient secretsClient = SecretsManagerClient.builder()
.region(region)
.build();
getValue(secretsClient, secretName);
secretsClient.close();
}
//snippet-start:[secretsmanager.java2.get_secret.main]
public static void getValue(SecretsManagerClient secretsClient,String secretName) {
try {
GetSecretValueRequest valueRequest = GetSecretValueRequest.builder()
.secretId(secretName)
.build();
GetSecretValueResponse valueResponse = secretsClient.getSecretValue(valueRequest);
String secret = valueResponse.secretString();
System.out.println(secret);
} catch (SecretsManagerException e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
}
//snippet-end:[secretsmanager.java2.get_secret.main]
}
You can find this example and others for this AWS Service here:
https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/secretsmanager
I want to create a youtube like Video streaming application but in a small scale. I am using Spring boot for backend rest endpoints and amazon S3 bucket for storing video files. I am able to upload and download video files to S3 bucket. But I am confused in streaming side. I want to show those video files in jsp page to play. I heard about Aws video on demand, aws kinesis, etc. Can someone suggest me or share some link which will be the best approach to follow for video streaming with spring boot. Or is there any other service apart from aws services which can be useful in this scenario. I am totally confused. Please help me out. Thank you.
I have created a sample project for streaming the AWS s3 resources using spring boot.
You can set a controller with mapping as required.
For this demo code the endpoint is http://localhost:port/bucket_name/object_key
#RestController("/")
public class ApiController {
#Value("${aws.region}")
private String awsRegion;
#GetMapping(value = "/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
public ResponseEntity<StreamingResponseBody> getObject(HttpServletRequest request) {
try {
AmazonS3 s3client = AmazonS3ClientBuilder.standard().withRegion(awsRegion).build();
String uri = request.getRequestURI();
String uriParts[] = uri.split("/", 2)[1].split("/", 2);
String bucket = uriParts[0];
String key = uriParts[1];
System.out.println("Fetching " + uri);
S3Object object = s3client.getObject(bucket, key);
S3ObjectInputStream finalObject = object.getObjectContent();
final StreamingResponseBody body = outputStream -> {
int numberOfBytesToWrite = 0;
byte[] data = new byte[1024];
while ((numberOfBytesToWrite = finalObject.read(data, 0, data.length)) != -1) {
outputStream.write(data, 0, numberOfBytesToWrite);
}
finalObject.close();
};
return new ResponseEntity<StreamingResponseBody>(body, HttpStatus.OK);
} catch (Exception e) {
System.err.println("Error "+ e.getMessage());
return new ResponseEntity<StreamingResponseBody>(HttpStatus.BAD_REQUEST);
}
}
}
You need to use StreamingResponseBody in your ResponseEntity.
If you need a ready to use microservice feel free to explore the github project s3-streamer I wrote for very same purpose.
when i make post request to upload image file to aws s3 bucket form my local dot net core aws lambda serverless application it works but form my deployed application the image still uploded to s3 bucket but the image broken (shows black empty image)
here is the code
[HttpPut("PostImageFile")]
public async Task FileImageAsync(string Id)
{
var s3Client = new AmazonS3Client("*******", "*******", Amazon.RegionEndpoint.USEast1);
try
{
var httpRequest = HttpContext.Request;
//posted file
var file = httpRequest.Form.Files[0];
byte[] fileBytes = new byte[file.Length];
file.OpenReadStream().Read(fileBytes, 0, Int32.Parse(file.Length.ToString()));
var fileName = Guid.NewGuid() + file.FileName;
PutObjectResponse response = null;
using (var stream = new MemoryStream())
{
file.CopyTo(stream);
var request = new PutObjectRequest
{
BucketName = "imageakal",
Key = fileName,
InputStream = stream,
ContentType = file.ContentType,
CannedACL = S3CannedACL.PublicReadWrite
};
response = await s3Client.PutObjectAsync(request);
};
}
catch (Exception ex)
{
Console.Write("Upload Failed: " + ex.Message);
}
}
Without many more details, I would guess that your AWS settings could have a list of permitted/denied domains. I would check that your AWS instance is configured to allow requests from your domain.
Just put "multipart/form-data" at 'Binary Media Type' section in Api Gateway setting tab, and deploy it(don't forget).