S3 CopyObjectRequest between regions - amazon-web-services

I'm trying to copy some objects between 2 S3 buckets that are in different regions.
I have this:
static void PutToDestination(string filename)
{
var credentials = new Amazon.Runtime.BasicAWSCredentials(awsAccessKeyId, awsSecretKey);
var client = new Amazon.S3.AmazonS3Client(credentials, Amazon.RegionEndpoint.GetBySystemName(awsS3RegionNameSource));
CopyObjectRequest request = new CopyObjectRequest();
request.SourceKey = filename;
request.DestinationKey = filename;
request.SourceBucket = awsS3BucketNameSource;
request.DestinationBucket = awsS3BucketNameDest;
try
{
CopyObjectResponse response = client.CopyObject(request);
}
catch (AmazonS3Exception x)
{
Console.WriteLine(x);
}
}
I get "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."
There doesn't seem to be a way to set separate endpoints for source and destination.
Is there a different method I should look at?
Thanks

Did you try this? https://github.com/aws/aws-sdk-java-v2/issues/2212 (Provide destination region when building the S3:Client). Worked for me :)

I think, if you don't specify the region explicitly, then Copy between cross region should work.
See documentation.
However, the accelaration will not work and copy will be
COPY=GET+PUT
Here is explanation excerpt from documentation that says-
Important
Amazon S3 Transfer Acceleration does not support cross-region copies.
In your code, you are specifying the region like below.
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new ProfileCredentialsProvider())
.withRegion(clientRegion)
.build();
Instead initialize S3client without region like below.
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new ProfileCredentialsProvider()).build();

Related

S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint

I know that there are many similar questions, and this one is no exception
But unfortunately I can't decide on the region for my case, how can I decide on the right region?
For example, when making a request to Postman, I encounter a similar error:
In my console i'm using EU (Frankfurt) eu-central-1 and also in terminal write smth like this:
heroku config:set region="eu-central-1"
And as I understand it, mine does not fit.
Also here is my AWS class:
class AmazonFileStorage : FileStorage {
private val client: S3Client
private val bucketName: String = System.getenv("bucketName")
init {
val region = System.getenv("region")
val accessKey = System.getenv("accessKey")
val secretKey = System.getenv("secretKey")
val credentials = AwsBasicCredentials.create(accessKey, secretKey)
val awsRegion = Region.of(region)
client = S3Client.builder()
.credentialsProvider(StaticCredentialsProvider.create(credentials))
.region(awsRegion)
.build() as S3Client
}
override suspend fun save(file: File): String =
withContext(Dispatchers.IO) {
client.putObject(
PutObjectRequest.builder().bucket(bucketName).key(file.name).acl(ObjectCannedACL.PUBLIC_READ).build(),
RequestBody.fromFile(file)
)
val request = GetUrlRequest.builder().bucket(bucketName).key(file.name).build()
client.utilities().getUrl(request).toExternalForm()
}
}
I think you may have the wrong region code; you do know that a Bucket is available in one and only one Region?
In your logging settings, set this scope to debug:
logging:
level:
org.apache.http.wire: debug
Then you should see something like this:
http-outgoing-0 >> "HEAD /somefile HTTP/1.1[\r][\n]"
http-outgoing-0 >> "Host: YOURBUCKETNAME.s3.eu-west-2.amazonaws.com[\r][\n]"
That log is from a bucket in the London region eu-west-2
To use Kotlin to interact with an Amazon S3 bucket (or other AWS services), consider using the AWS SDK for Kotlin. This SDK is meant for Kotlin developers. You are using the AWS SDK for Java.
To put an object into an Amazon S3 bucket using the AWS SDK for Kotlin, use this code. Notice the region that you want to use is specified in the code block where you define the aws.sdk.kotlin.services.s3.S3Client.
import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.PutObjectRequest
import aws.smithy.kotlin.runtime.content.asByteStream
import java.io.File
import kotlin.system.exitProcess
/**
Before running this Kotlin code example, set up your development environment,
including your credentials.
For more information, see the following documentation topic:
https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/setup.html
*/
suspend fun main(args: Array<String>) {
val usage = """
Usage:
<bucketName> <objectKey> <objectPath>
Where:
bucketName - The Amazon S3 bucket to upload an object into.
objectKey - The object to upload (for example, book.pdf).
objectPath - The path where the file is located (for example, C:/AWS/book2.pdf).
"""
if (args.size != 3) {
println(usage)
exitProcess(0)
}
val bucketName = args[0]
val objectKey = args[1]
val objectPath = args[2]
putS3Object(bucketName, objectKey, objectPath)
}
suspend fun putS3Object(bucketName: String, objectKey: String, objectPath: String) {
val metadataVal = mutableMapOf<String, String>()
metadataVal["myVal"] = "test"
val request = PutObjectRequest {
bucket = bucketName
key = objectKey
metadata = metadataVal
body = File(objectPath).asByteStream()
}
S3Client { region = "us-east-1" }.use { s3 ->
val response = s3.putObject(request)
println("Tag information is ${response.eTag}")
}
}
You can find this Kotlin example and many more in the AWS Code Library here:
Amazon S3 examples using SDK for Kotlin
ALso you can read the Kotlin DEV guide too. The link is at the start of the Code Example.

Connect to S3 bucket without giving region name

I need to download objects from S3 bucket and I have below information to access the S3 bucket.
Access Key, Secret Key and Bucket End point. There is no region name.
I was using minio package to access the bucket and was able to access it using below code :
public void getS3BucketObject() throws IOException, NoSuchAlgorithmException,
InvalidKeyException, ServerException, InsufficientDataException, InternalException, InvalidResponseException,
XmlParserException, ErrorResponseException {
//creating minioClient to access S3 bucket
minioClient =
MinioClient.builder()
.endpoint(s3BucketEndpoint)
.credentials(s3BucketAccessKey, s3BucketSecretKey)
.build();
//check for bucket existance
boolean found =
minioClient.bucketExists(BucketExistsArgs.builder().bucket(s3BucketName).build());
if (!found) {
System.out.println("Bucket doesn't exist ");
} else {
System.out.println("Bucket exists ");
}
}
But, I do need to use AWS SDK instead of minio but I am not sure as I don't have region information and not sure how to pass endpoint in the configuration setting, though I tried below.
final BasicAWSCredentials credentials = new BasicAWSCredentials(s3BucketAccessKey, s3BucketSecretKey);
final AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials));
boolean found = s3client.doesBucketExistV2(s3BucketName);
if (found){
System.out.println("The bucket is available ");
}else {
System.out.println("The bucket doesn't exist");
}
Have you tried using withRegion(AWS_S3_REGION) option while creating s3client using builder?
AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withRegion("us-east-1")
.build();
Ref. https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3ClientBuilder.html
AmazonS3ClientBuilder.defaultClient() fails to account for region?

How to use DefaultAWSCredentialsProviderChain() in java code to fetch credentials from instance profile and allow access to s3 bucket

I am working on a requirement where i want to connect to s3 bucket using springboot application.
When i am connecting through my local environment i am using seeting loadCredentials(true) which uses Amazon STS which fetches the temperoriy credentials using a role and allow access to s3 bucket.
When i am deploying to the qa/prd envirment i am setting loadCredentials(false) which uses DefaultAWSCredentialsProviderChain() class to fetch the credential from aws instance profile(role is assigned to ec2 instance) and allow access to s3 bucket. My code is
#Configuration
public class AmazonS3Config
{
static String clientRegion = "ap-south-1";
static String roleARN = "arn:aws:iam::*************:role/awss3acess";
static String roleSessionName = "bucket_storage_audit";
String bucketName = "testbucket";
//Depending on environment is set to true(for local environment) or false(for qa and prd environment)
private static AWSCredentialsProvider loadCredentials(boolean isLocal) {
final AWSCredentialsProvider credentialsProvider;
if (isLocal) {
AWSSecurityTokenService stsClient = AWSSecurityTokenServiceAsyncClientBuilder.standard()
.withCredentials(new ProfileCredentialsProvider())
.withRegion(clientRegion)
.build();
AssumeRoleRequest assumeRoleRequest = new AssumeRoleRequest().withDurationSeconds(3600)
.withRoleArn(roleARN)
.withRoleSessionName(roleSessionName);
AssumeRoleResult assumeRoleResult = stsClient.assumeRole(assumeRoleRequest);
Credentials creds = assumeRoleResult.getCredentials();
credentialsProvider = new AWSStaticCredentialsProvider(
new BasicSessionCredentials(creds.getAccessKeyId(),
creds.getSecretAccessKey(),
creds.getSessionToken())
);
} else {
System.out.println("inside default");
credentialsProvider = new DefaultAWSCredentialsProviderChain();
}
return credentialsProvider;
}
// Amazon s3client Bean return an instance of s3client
. #Bean
public AmazonS3 s3client() {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.fromName(clientRegion))
.withCredentials(loadCredentials(false))
.build();
return s3Client;
}
}
My question since the credentials of instance profile rotate after every 12 hours my application will fail after 12 hours.
What will i do to avoid this from happening in my code.
You can directly use ProfileCredentialsProvider instead of DefaultAWSCredentialsProviderChain as there is no need in your case to chain the credsproviders.
and about your question, AWSCredentialsProvider has refresh() method that will reread the config file.when an Authentication exception Occurs while using S3Client you can retry again and call refresh() first.

Location constraint exception while trying to access bucket from aws ec2 client

I am trying to create bucket from java web application. My tomcat is configured on AWS EC2 instance. It is giving following error, while it tries to connect to AWS S3:
com.amazonaws.services.s3.model.AmazonS3Exception:
The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
(Service: Amazon S3; Status Code: 400;..).
This is the code sample:
public class FileOperationsUtil {
private final BasicAWSCredentials awsCreds = new BasicAWSCredentials("xyz", "zyz");
private final AmazonS3 s3Client = new AmazonS3Client(awsCreds); private final String bucketName = "grex-prod";
//public static final Region ap-south-1;
public void uploadFile(InputStream fileInputStream,
String fileUploadLocation, String fileName) throws IOException {
s3Client.setRegion(Region.getRegion(Regions.AP_SOUTH_1));
// Region apsouth1 = Region.getRegion(Regions.ap-south-1); // s3Client.setRegion(apsouth1); // s3Client.setRegion(Region.getRegion(Regions.ap-south-1));
//s3Client.create_bucket(bucket, CreateBucketConfiguration={'LocationConstraint': 'ap-northeast-2'})
s3Client.createBucket(bucketName);
File fileToUpload = new File(fileUploadLocation);
fileToUpload.mkdirs();
// Full file path
String fullFilePath = (fileUploadLocation + fileName);
ObjectMetadata meta = new ObjectMetadata();
// meta.setContentLength(contents.length);
meta.setContentType("image/png");
// Upload files to a specific AWS s3 bucket
s3Client.putObject(new PutObjectRequest("grex-prod", fullFilePath,
fileInputStream, meta)
.withCannedAcl(CannedAccessControlList.Private));
}
public void deleteFolder(String oldFullFilePath) {
// System.out.println("inside");
ObjectListing objects = s3Client.listObjects(bucketName, oldFullFilePath);
for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
s3Client.deleteObject(bucketName, objectSummary.getKey());}
s3Client.deleteObject(bucketName, oldFullFilePath);}
In your example above:
s3Client.setRegion(Region.getRegion(Regions.AP_SOUTH_1));
// Region apsouth1 = Region.getRegion(Regions.ap-south-1); // s3Client.setRegion(apsouth1); // s3Client.setRegion(Region.getRegion(Regions.ap-south-1));
//s3Client.create_bucket(bucket, CreateBucketConfiguration={'LocationConstraint': 'ap-northeast-2'})
Both "region" and "LocationConstraint" should match. If you want to create the bucket in "ap-south-1" then both should be set to that value.
The error you received was due to the two values not matching, in other words you connected to one region (likely ap-south-1), and then tried to create a bucket which is intended to exist in another region (ap-northeast-2).
By excluding "LocationConstraint", the location where the bucket is created is entirely based on the "Region" which you are connected to. By using "LocationConstraint" you can ensure you are not trying to create the bucket in a region other than the one you intended.
There are some rules while using "locationConstraint":
region "us-east-1" is compatible with all regions in "locationConstraint"
With any other region, region and locationConstraint has to be same.
In short:
With "us-east-1" only, You can use any region as locationConstraint to create bucket in desired region.(region='us-east-1' locationConstraint=)
With all other region region and locationConstraint should be the same(region='us-west-2', locationConstraint='us-west-2')
any other region will throw "code:IllegalLocationConstraintException"

AWS S3 - Move an object to a different folder

Is there any way to move an object to a different folder in the same bucket using the AWS SDK (preferably for .Net)?
Searching around all I can see is the suggestion of Copy to new location and Delete of the original (Which is easy enough via "CopyObjectRequest" and "DeleteObjectRequest") however I'm just wondering is this the only way?
Turns out you can use Amazon.S3.IO.S3FileInfo to get the object and then call the "MoveTo" method to move the object.
S3FileInfo currentObject = new S3FileInfo(S3Client,Bucket,CurrentKey);
S3FileInfo movedObject = currentObject.MoveTo(Bucket,NewKey);
EDIT: It turns out the above "MoveTo" method just performs a Copy and Delete behind the scenes anyway :)
For further information:
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/index.html?page=S3/TS3IOS3FileInfo.html&tocid=Amazon_S3_IO_S3FileInfo
As #user1527762 mentions S3FileInfo is not available in .NET Core version of the AWSSDK.S3 library (v 3.7.2.2).
If you are in .net core you can use the CopyObject to move an object from one bucket to another.
var s3Client = new AmazonS3Client(RegionEndpoint.USEast1);
var copyRequest = new CopyObjectRequest
{
SourceBucket = "source-bucket",
SourceKey = "fileName",
DestinationBucket = "dest-bucket",
DestinationKey = "fileName.jpg",
CannedACL = S3CannedACL.PublicRead,
};
var copyResponse = await s3Client.CopyObjectAsync(copyRequest);
CopyObject does not delete the source object so you would have to call delete like this:
DeleteObjectRequest request = new DeleteObjectRequest
{
BucketName = "source-bucket",
Key = "fileName.jpg"
};
await s3Client .DeleteObjectAsync(request);
Details on CopyObject here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/copy-object.html
And DeleteObject here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-objects.html