Hello I am trying to download data from one of an Amazon S3 public bucket.
For example https://registry.opendata.aws/noaa-gfs-bdp-pds/
The bucket has web accessible folder and I want to download the files inside the bucket.
I know I can do this with AWS CLI tool
But I want to know if there anyway to do this with AWs SDK Api (s3 client) (c# visual studio)?
I think the issue is authentication when creating connection to s3 client it requires credentials like access key ,I don't have an AWS account,and the bucket I try to get to is public so
Does anyone know how to access to this public bucket without any credentials via API?
Thanks.
If you specify the AnonymousAWSCredentials as the credentials object, any requests that are made to S3 will be unsigned. After that, interacting with the bucket is done like any other call:
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using System;
namespace S3TestApp
{
class Program
{
static void Main(string[] args)
{
var unsigned = new AnonymousAWSCredentials();
var client = new AmazonS3Client(unsigned, Amazon.RegionEndpoint.USEast1);
var listRequest = new ListObjectsRequest
{
BucketName = "noaa-gfs-bdp-pds",
Delimiter = "/",
};
ListObjectsResponse listResponse;
do
{
listResponse = client.ListObjects(listRequest);
foreach (var obj in listResponse.CommonPrefixes)
{
Console.WriteLine("PRE {0}", obj);
}
foreach (var obj in listResponse.S3Objects)
{
Console.WriteLine("{0} {1}", obj.Size, obj.Key);
}
listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);
}
}
}
Related
I need to download objects from S3 bucket and I have below information to access the S3 bucket.
Access Key, Secret Key and Bucket End point. There is no region name.
I was using minio package to access the bucket and was able to access it using below code :
public void getS3BucketObject() throws IOException, NoSuchAlgorithmException,
InvalidKeyException, ServerException, InsufficientDataException, InternalException, InvalidResponseException,
XmlParserException, ErrorResponseException {
//creating minioClient to access S3 bucket
minioClient =
MinioClient.builder()
.endpoint(s3BucketEndpoint)
.credentials(s3BucketAccessKey, s3BucketSecretKey)
.build();
//check for bucket existance
boolean found =
minioClient.bucketExists(BucketExistsArgs.builder().bucket(s3BucketName).build());
if (!found) {
System.out.println("Bucket doesn't exist ");
} else {
System.out.println("Bucket exists ");
}
}
But, I do need to use AWS SDK instead of minio but I am not sure as I don't have region information and not sure how to pass endpoint in the configuration setting, though I tried below.
final BasicAWSCredentials credentials = new BasicAWSCredentials(s3BucketAccessKey, s3BucketSecretKey);
final AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials));
boolean found = s3client.doesBucketExistV2(s3BucketName);
if (found){
System.out.println("The bucket is available ");
}else {
System.out.println("The bucket doesn't exist");
}
Have you tried using withRegion(AWS_S3_REGION) option while creating s3client using builder?
AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withRegion("us-east-1")
.build();
Ref. https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3ClientBuilder.html
AmazonS3ClientBuilder.defaultClient() fails to account for region?
I want to create a youtube like Video streaming application but in a small scale. I am using Spring boot for backend rest endpoints and amazon S3 bucket for storing video files. I am able to upload and download video files to S3 bucket. But I am confused in streaming side. I want to show those video files in jsp page to play. I heard about Aws video on demand, aws kinesis, etc. Can someone suggest me or share some link which will be the best approach to follow for video streaming with spring boot. Or is there any other service apart from aws services which can be useful in this scenario. I am totally confused. Please help me out. Thank you.
I have created a sample project for streaming the AWS s3 resources using spring boot.
You can set a controller with mapping as required.
For this demo code the endpoint is http://localhost:port/bucket_name/object_key
#RestController("/")
public class ApiController {
#Value("${aws.region}")
private String awsRegion;
#GetMapping(value = "/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
public ResponseEntity<StreamingResponseBody> getObject(HttpServletRequest request) {
try {
AmazonS3 s3client = AmazonS3ClientBuilder.standard().withRegion(awsRegion).build();
String uri = request.getRequestURI();
String uriParts[] = uri.split("/", 2)[1].split("/", 2);
String bucket = uriParts[0];
String key = uriParts[1];
System.out.println("Fetching " + uri);
S3Object object = s3client.getObject(bucket, key);
S3ObjectInputStream finalObject = object.getObjectContent();
final StreamingResponseBody body = outputStream -> {
int numberOfBytesToWrite = 0;
byte[] data = new byte[1024];
while ((numberOfBytesToWrite = finalObject.read(data, 0, data.length)) != -1) {
outputStream.write(data, 0, numberOfBytesToWrite);
}
finalObject.close();
};
return new ResponseEntity<StreamingResponseBody>(body, HttpStatus.OK);
} catch (Exception e) {
System.err.println("Error "+ e.getMessage());
return new ResponseEntity<StreamingResponseBody>(HttpStatus.BAD_REQUEST);
}
}
}
You need to use StreamingResponseBody in your ResponseEntity.
If you need a ready to use microservice feel free to explore the github project s3-streamer I wrote for very same purpose.
I am working on a requirement where i want to connect to s3 bucket using springboot application.
When i am connecting through my local environment i am using seeting loadCredentials(true) which uses Amazon STS which fetches the temperoriy credentials using a role and allow access to s3 bucket.
When i am deploying to the qa/prd envirment i am setting loadCredentials(false) which uses DefaultAWSCredentialsProviderChain() class to fetch the credential from aws instance profile(role is assigned to ec2 instance) and allow access to s3 bucket. My code is
#Configuration
public class AmazonS3Config
{
static String clientRegion = "ap-south-1";
static String roleARN = "arn:aws:iam::*************:role/awss3acess";
static String roleSessionName = "bucket_storage_audit";
String bucketName = "testbucket";
//Depending on environment is set to true(for local environment) or false(for qa and prd environment)
private static AWSCredentialsProvider loadCredentials(boolean isLocal) {
final AWSCredentialsProvider credentialsProvider;
if (isLocal) {
AWSSecurityTokenService stsClient = AWSSecurityTokenServiceAsyncClientBuilder.standard()
.withCredentials(new ProfileCredentialsProvider())
.withRegion(clientRegion)
.build();
AssumeRoleRequest assumeRoleRequest = new AssumeRoleRequest().withDurationSeconds(3600)
.withRoleArn(roleARN)
.withRoleSessionName(roleSessionName);
AssumeRoleResult assumeRoleResult = stsClient.assumeRole(assumeRoleRequest);
Credentials creds = assumeRoleResult.getCredentials();
credentialsProvider = new AWSStaticCredentialsProvider(
new BasicSessionCredentials(creds.getAccessKeyId(),
creds.getSecretAccessKey(),
creds.getSessionToken())
);
} else {
System.out.println("inside default");
credentialsProvider = new DefaultAWSCredentialsProviderChain();
}
return credentialsProvider;
}
// Amazon s3client Bean return an instance of s3client
. #Bean
public AmazonS3 s3client() {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.fromName(clientRegion))
.withCredentials(loadCredentials(false))
.build();
return s3Client;
}
}
My question since the credentials of instance profile rotate after every 12 hours my application will fail after 12 hours.
What will i do to avoid this from happening in my code.
You can directly use ProfileCredentialsProvider instead of DefaultAWSCredentialsProviderChain as there is no need in your case to chain the credsproviders.
and about your question, AWSCredentialsProvider has refresh() method that will reread the config file.when an Authentication exception Occurs while using S3Client you can retry again and call refresh() first.
I am working on my AWS cert and I'm trying to figure out how the following bit of js code works:
var AWS = require('aws-sdk');
var uuid = require('node-uuid');
// Create an S3 client
var s3 = new AWS.S3();
// Create a bucket and upload something into it
var bucketName = 'node-sdk-sample-' + uuid.v4();
var keyName = 'hello_world.txt';
s3.createBucket({Bucket: bucketName}, function() {
var params = {Bucket: bucketName, Key: keyName, Body: 'Hello'};
s3.putObject(params, function(err, data) {
if (err)
console.log(err)
else
console.log("Successfully uploaded data to " + bucketName + "/" + keyName);
});
});
This code successfully loads a txt file containing the words "Hello" in it. I do not understand how this ^ can identify MY AWS account. It does! But how! It somehow is able to determine that I want a new bucket inside MY account, but this code was taken directly from the AWS docs. I don't know how it could figure that out....
As per Class: AWS.CredentialProviderChain, the AWS SDK for JavaScript looks for credentials in the following locations:
AWS.CredentialProviderChain.defaultProviders = [
function () { return new AWS.EnvironmentCredentials('AWS'); },
function () { return new AWS.EnvironmentCredentials('AMAZON'); },
function () { return new AWS.SharedIniFileCredentials(); },
function () {
// if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set
return new AWS.ECSCredentials();
// else
return new AWS.EC2MetadataCredentials();
}
]
Environment Variables (useful for testing, or when running code on a local computer)
Local credentials file (useful for running code on a local computer)
ECS credentials (useful when running code in Elastic Container Service)
Amazon EC2 Metadata (useful when running code on an Amazon EC2 instance)
It is highly recommended to never store credentials within an application. If the code is running on an Amazon EC2 instance and a role has been assigned to the instance, the SDK will automatically retrieve credentials from the instance metadata.
The next best method is to store credentials in the ~/.aws/credentials file.
I am using aws unity sdk for my unity game to download the assets stored in amazon server.I used GetObjectAsync() for download.In the aws sdk for iOS,we get the progress value from NSUrlSessionDelegates.But I want to use aws mobile sdk for unity and get the progress value of download.How do I do this? Please help.
private void GetObject()
{
ResultText.text = string.Format("fetching {0} from bucket {1}",
SampleFileName, S3BucketName);
Client.GetObjectAsync(S3BucketName, SampleFileName, (responseObj) =>
{
string data = null;
var response = responseObj.Response;
if (response.ResponseStream != null)
{
using (StreamReader reader = new StreamReader(response.ResponseStream))
{
data = reader.ReadToEnd();
}
ResultText.text += "\n";
ResultText.text += data;
}
});
}
You can do that. Here is snippet from Amazon develper website.
public event EventHandler<WriteObjectProgressArgs> WriteObjectProgressEvent
TransferUtilityDownloadRequest request = new TransferUtilityDownloadRequest();
request.WriteObjectProgressEvent += displayProgress;
private void displayProgress(object sender, WriteObjectProgressArgs args)
{
Console.WriteLine(args);
}
I have the same problem and I tried to ask on AWS forum.
Unfortunately they doesn't support yet.
Here the link:
https://forums.aws.amazon.com/thread.jspa?threadID=248187&tstart=0
To access of forum is necessary to login.