I am trying to access an Amazon S3 bucket programmatically through Java libraries. (to do basic cloud management from a third-party application). As a first step, I tried to print whether a bucket exists or not(3rd line)
AWSCredentials credentials=new BasicAWSCredentials("my-Access- Key","My- Secret-Key");
AmazonS3 s3client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials)).withRegion(Regions.AP_EAST_1).build();
String bucketExists=String.valueOf(s3client.doesBucketExistV2("newBucketName"));
When I run this line of code, I am getting an exception saying that
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID:RequestId...)
I don't want to maintain a credentials file in the .aws folder for
the following reason:
I am trying to variablilize the access credentials
based on the logged-in user from a secure LDAP system, so I can
confirm the feasibility only when I test it with hard-coded
credentials.
I have checked that the issue is not one of the below
I have created an IAM user with a valid Access ID and Secret Key in
the AWS console and have enabled the user for programmatic access.
I have also given applied the AmazonS3FullAccess policy for IAM user
The key is in Active state(have checked it through the console)
I have added the dependency for the AWS SDK to gradle (implementation group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.12.15')
Please let me know what the issue might be. My apologies if it is an amateur issue.
As indicated in the question comments, your code looks fine and it should work properly.
The most likely reason of the problem is that AWS is picking up other credentials from somewhere else. Please, try removing other credentials like the ones stored in the home directory in order to be sure that the SDK is using the right credentials when contacting S3.
In addition, please, verify that you are providing the right region according to your S3 bucket as well.
You are using the older V1 API. The Service Client name for V1 is AmazonS3. Likewise, the V2 service client name is S3Client. Amazon strongly recommends moving to V2:
The AWS SDK for Java 2.x is a major rewrite of the version 1.x code base. It’s built on top of Java 8+ and adds several frequently requested features. These include support for non-blocking I/O and the ability to plug in a different HTTP implementation at run time.
Try following this step by step set of instructions, which is based on V2.
Get started with the AWS SDK for Java 2.x
WHen working with V2, you can put your creds in a file located in a .aws folder named credentials, as explained in this document. Once you follow all steps in this document, you can programmatically access an Amazon S3 bucket.
Related
So I got into a situation working for a client which does not provide in any way AWS_ACCESS_KEY_ID as security protection. We have only available for development AWS Web Console. So I started searching for another way of the programmatic script(speed-up) my dev tasks.
Note: we cannot use AWS client without AWS_ACCESS_KEY_ID and secret.
My assumptions: If the AWS web console can do the same thing as aws cli (.eg create bucket, load data into bucket, etc.), why not use web console auth mechanism (visible in http request headers) and bind it to aws cli (or some other api call code) to make it work even without aws keys?
Question: Is this possible? For sure I can see in http headers following artifacts:
aws-session-token
aws-session-id
awsccc
and dozen of others...
My idea is to automate this by:
Go to the web console and login have a script that will
automatically output from browser session required parameters to
some text file
Use this extracted information by some dev script
If this is not supported or impossible to achieve with aws cli, can I use some SDK or raw AWS Api calls with extracted information?
I can extract SAML content which has above mentioned aws-creds header also I see oauth client call with following headers:
https://signin.aws.amazon.com/oauth?
client_id=arn%3Aaws%3Asignin%3A%3A%3Aconsole%2Fcanvas&
code_challenge=bJNNw87gBewdsKnMCZU1OIKHB733RmD3p8cuhFoz2aw&
code_challenge_method=SHA-256&
response_type=code&
redirect_uri=https%3A%2F%2Fconsole.aws.amazon.com%2Fconsole%2Fhome%3Ffromtb%3Dtrue%26isauthcode%3Dtrue%26state%3DhashArgsFromTB_us-east-1_c63b804c7d804573&
X-Amz-Security-Token=hidden content&
X-Amz-Date=20211223T054052Z&
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=ASIAVHC3TML26B76NPS4%2F20211223%2Fus-east-1%2Fsignin%2Faws4_request&
X-Amz-SignedHeaders=host&
X-Amz-Signature=3142997fe7212f041ef90c1a87288f53cecca9236098653904bab36b17fa53ef
Can I use it with AWS SDK somehow?
To reset an S3 bucket to a known state, I would suggest looking at the AWS cli s3 sync command and the -delete switch. Create a "template" bucket with your default contents, then sync that bucket into your Dev Bucket to reset your Dev bucket.
As for your key problems, i would look at IAM Roles rather trying to hack the console auth.
As to how to run the AWS CLI, you have several options. It can be done from Lambda, ECS (containers running on your own Ec2) or an ec2 instance. All 3 allow you to attach an IAM role. That role can have policies attached (for your S3 bucket) - but there is no key to manage.
Thx for feedback to #MisterSmith! It kinda helped with follow up.
I have found also SAML call during analysis of Chrome traffic from login page to AWS console, I have found this project: https://github.com/Versent/saml2aws#linux
Which extracted all ~/.aws/credentials variables needed for aws cli to work.
I have a Java program which needs to access Amazon S3 to put some files there. Please note that this Java program is running in my desktop (not in EC2). What's the best secure way to access Amazon S3 using credentials? Following are the ways I am aware of.
Using access token and secret
a. In sdk properties file
b. As environment variables
c. In command line system properties
d. Directly hard coding in program
Of course I'd prefer options b and c for security reasons.
Is there a role based permissions possible here? My understanding is that it's not possible since my Java program is running in an external machine which AWS doesn't know.
Any other method of access possible?
Thanks in advance.
The best way is to use the default provider chain, which means that the [DefaultCredentialsProvider] (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html) class will decide from where to take the credentials based on a specific hierarchy:
1. Java System Properties - aws.accessKeyId and aws.secretAccessKey
2. Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3. Web Identity Token credentials from system properties or environment variables
4. Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
5. Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
6. Instance profile credentials delivered through the Amazon EC2 metadata service
For local development the recommended way is to set up your credentials using the aws configure command and let the default provider chain take advantage of that.
Although environment variables may be a reasonable choice in some cases (and the default chain will be able to use them), please NEVER ever hardcode any credentials in your code!
Yes it is. We can assume a role using the AWS CLI:
aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session
This will provide a temporary AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN which can be provided to the application. The application will communicate with AWS services using the permissions provided by the assumed role.
Yes, there is another way if the goal is to access S3. We can use presigned urls.
When working with the AWS SDK for Java V2, refer to the AWS SDK for Java
Developer Guide V2. This developer guide contains a lot of information that answers questions like this.
To learn how credentials work, please refer to this topic:
Using credentials
All AWS Examples in Github assume credentials are loaded from the credential file. As explained in the docs, the credentials file is located in
Windows - C:\Users<yourUserName>.aws\credentials
Linux, macOS, Unix - ~/.aws/credentials
See this topic that will show you how to get up and running using the Amazon S3 API- including setting up your credentials.
Get started with the AWS SDK for Java 2.x
The Amazon S3 Java API has methods like pubObject that lets you place objects into an Amazon S3 bucket.
I am currently curious about if there any risks when using S3 GUI client tool like S3Browser or Cyberduck to explore our S3 file with AWS secret keys.
https://s3browser.com/
https://cyberduck.io/
Can those aws keys will be stolen by the tool?
Both of these services store the keys locally on your host machine.
From experience I know that S3 Browser allows your credentials to be encrypted locally on the disk using a password of your choosing as the decryption key.
Obviously goes without saying but when using any service ensure the following to increase your security:
Regularly rotate your credentials.
Your IAM policies should locked down to the minimum settings you require.
If you're concerned about API calls enabled S3 API calls in CloudTrail
If its a 1 time key, immediately remove it afterwards from both the software and your AWS account.
S3 browser also supports the assuming of a role which allows it to generate temporary credentials.
Nothing is 100% risk free. As a developer who has built one of those s3 browser, this is what I'm doing to mitigate the risk:
the entire code is 100% open source (https://github.com/mickael-kerjean/filestash) so you can audit the code that actually runs
there's a special page from the application that shows the hash of the binary alongside the commit from which this version is built from to enable anyone to see the built they run and verify it as well
I support assume role so you can use short lived credentials
I'm beginning to work with Secrets Manager and created my first secret in AWS. During the process, it gave me some sample code to work with. I put that in a small application and ran it. The code:
String region = "us-east-1";
string secret = "";
MemoryStream memoryStream = new MemoryStream();
IAmazonSecretsManager client = new AmazonSecretsManagerClient(
RegionEndpoint.GetBySystemName(region));
GetSecretValueRequest request = new GetSecretValueRequest();
request.SecretId = "MySecretNameExample";
GetSecretValueResponse response = null;
response = client.GetSecretValue(request);
The problem is that:
I was able to successfully retrieve the secret that I created and
nowhere am I creating a Credentials object with any valid AWS credential data
Where is this code getting the credential information from?
If you refer to the documenation for the API for this line of code:
IAmazonSecretsManager client = new AmazonSecretsManagerClient(
RegionEndpoint.GetBySystemName(region));
AmazonSecretsManagerClient
You will find the following description:
Constructs AmazonSecretsManagerClient with the credentials loaded from
the application's default configuration, and if unsuccessful from the
Instance Profile service on an EC2 instance.
This means that you are either running on an EC2 or ECS service (or related service such as Beanstalk, ...) with a role assigned to the instance or your have configured your credentials in the standard method in a credentials file. The AWS SDK is helping you locate credentials.
This document link will explain in more detail how AWS credentials are managed and selected.
Working with AWS Credentials
I have seen a lot of developers get the little details wrong with how credentials work and how they are used within the SDKs. Given that AWS credentials hold the keys to the AWS kingdom, managing and protecting them is vitally important.
The AWS SDK uses the a resolution strategy that looks in a number of locations until it finds credentials it can use. Typically the DefaultProviderChain class is responsible for performing the resolution. More information is here, but the gist is the lookup is performed in the following order (for Java, other languages are similar):
environment variables
Java system properties
credentials file (e.g. in the home directory)
instance profile credentials (only available when running in AWS)
When you run within AWS infrastructure, you can assign a profile or role to the resource that's running your code. Doing that makes credentials automatically available to your code. The idea is that they've made it easy to avoid putting credentials directly into your code.
AWS .Net SDK specifies several ways to specify credentials. One of them is SDK store - http://docs.aws.amazon.com/AWSSdkDocsNET/V3/DeveloperGuide/net-dg-config-creds.html#net-dg-config-creds-sdk-store
In that same document two seemingly contradictory things are written:
"SDK Store profiles are specific to a particular user on a particular host. They cannot be copied to other hosts or other users. For this reason, SDK Store profiles cannot be used in production applications."
"The associated credentials are incorporated into the application during the build process."
IF the credentials are incorporated in the program at the build time, then why can't I use the SDK store method in production environment?
That leaves me with either storing credentials in .config files, or credentials files, or using roles.
To clarify, the documentation is stating that if you are only using the SDK store, you can't simply pick up your application and move it to any other machine while expecting it to work as-is. If you are using profile names, your application is referencing the profile name in the local SDK store (or, failing that, the credentials file used by other AWS resources like the AWS CLI).
For example, if you created a profile named dev-Amit on your personal computer and then moved to another machine in production, dev-Amit wouldn't exist there unless you had already configured the SDK store on that machine with the same profile name.
This behavior is contrary to the recommended practice of passing credentials in EC2, using temporary credentials via IAM Roles. Using this method, you can reliably pass credentials to your applications on EC2 instances by simply attaching an IAM role to the instance that has sufficient permissions to do whatever your application needs to do. This method is preferred because you don't have to login to the instance to configure anything; you simply attach the IAM role at creation time.
Further Resources:
AWS Documentation - Tutorial: Grant Access Using an IAM Role and the AWS SDK for .NET
AWS Documentation - Order in which credentials are searched for