How can I make the GO SDK fetch the access keys for AWS from the Instance Metadata Service (169.254.169.254) provided by AWS.
I checked the official AWS SDK for go documentation and there seems to be only ways of fetching the access keys from environment variables, but no credentials retriever from IMS.
How is this done in go?
I checked the official AWS SDK for go documentation and there seems to be only ways of fetching the access keys from environment variables, but no credentials retriever from IMS.
You just missed it. The Go SDK supports the instance metadata service as well as every other common credentials provider.
From https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html:
If you have configured your instance to use IAM roles, the SDK uses these credentials for your application automatically.
You don't have to do anything to configure this. It should just work. If you're having problems, make sure that you're not manually configuring some other credentials source.
Usually you don't have to do anything more than something like:
sess := session.Must(
session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}),
)
And with or without CLI configuration, metadata service, or environment variables, it should just work wherever you run it.
Related
I have a Java program which needs to access Amazon S3 to put some files there. Please note that this Java program is running in my desktop (not in EC2). What's the best secure way to access Amazon S3 using credentials? Following are the ways I am aware of.
Using access token and secret
a. In sdk properties file
b. As environment variables
c. In command line system properties
d. Directly hard coding in program
Of course I'd prefer options b and c for security reasons.
Is there a role based permissions possible here? My understanding is that it's not possible since my Java program is running in an external machine which AWS doesn't know.
Any other method of access possible?
Thanks in advance.
The best way is to use the default provider chain, which means that the [DefaultCredentialsProvider] (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html) class will decide from where to take the credentials based on a specific hierarchy:
1. Java System Properties - aws.accessKeyId and aws.secretAccessKey
2. Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3. Web Identity Token credentials from system properties or environment variables
4. Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
5. Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
6. Instance profile credentials delivered through the Amazon EC2 metadata service
For local development the recommended way is to set up your credentials using the aws configure command and let the default provider chain take advantage of that.
Although environment variables may be a reasonable choice in some cases (and the default chain will be able to use them), please NEVER ever hardcode any credentials in your code!
Yes it is. We can assume a role using the AWS CLI:
aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session
This will provide a temporary AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN which can be provided to the application. The application will communicate with AWS services using the permissions provided by the assumed role.
Yes, there is another way if the goal is to access S3. We can use presigned urls.
When working with the AWS SDK for Java V2, refer to the AWS SDK for Java
Developer Guide V2. This developer guide contains a lot of information that answers questions like this.
To learn how credentials work, please refer to this topic:
Using credentials
All AWS Examples in Github assume credentials are loaded from the credential file. As explained in the docs, the credentials file is located in
Windows - C:\Users<yourUserName>.aws\credentials
Linux, macOS, Unix - ~/.aws/credentials
See this topic that will show you how to get up and running using the Amazon S3 API- including setting up your credentials.
Get started with the AWS SDK for Java 2.x
The Amazon S3 Java API has methods like pubObject that lets you place objects into an Amazon S3 bucket.
I am having properties file specific for dev, test and other environments. I have to store this files in some secure place in aws. I am using AWS Native tools for build and deployment. Please let me know how to store these files in aws
There are many ways to deal with a secret in case of AWS, but one thing is clear it depends on the service where you will use and consume these secret.
But you explore these two
The simplest way is to use the Environment variable.
AWS Secrets Manager
s3 ( for keeping files)
One common approach is to pass your secret as an environment variables, but in case of AWS I will recommend to go with AWS Secrets Manager
AWS Secrets Manager is an AWS service that makes it easier for you to
manage secrets. Secrets can be database credentials, passwords,
third-party API keys, and even arbitrary text. You can store and
control access to these secrets centrally by using the Secrets Manager
console, the Secrets Manager command line interface (CLI), or the
Secrets Manager API and SDKs.
Basic Secrets Manager Scenario
The following diagram illustrates the most basic scenario. It shows
how you can store credentials for a database in Secrets Manager, and
then use those credentials in an application that needs to access the
database.
Compliance with Standards
AWS Secrets Manager has undergone auditing for the these standards and can be part of your solution when you need to obtain compliance certification.
You can explore this article to read and write secret.
If you need to maintain files, not only object then you can store in s3 and pull files during deployment. but better to enable server-side encprtion.
But I will prefer secret manager instead of s3 and environment variable.
You can for s3 here and here
Bajjuri,
As Adil said in answer,
AWS Secret Manager -- if you want to store data as key, value pairs.
AWS S3 -- if you want to store files securely.
Adding to his answer, you can use AWS CodeDeploy Environment
Variables to fetch the files according to the your environment.
Let's say you've CodeDeploy deployment group for dev environment with
name "DEV" and deployment group for prod environment with name "PROD",
you can use this variable in bash script and call it in life cycle
hooks of appspec file to fetch the files or secret accordingly.
I've been using this technique in production for long and it works like a charm.
Below code creates AWS Credentials where access and secret keys are explicitly supplied.
AWSCredentials credentials = new BasicAWSCredentials(
"<AWS accesskey>",
"<AWS secretkey>"
);
But the issue with this approach is in production I can not use it.
So what will be an equivalent java code that will obtain credential automatically from the aws credentials chain and create a credential object or some EC2 client.
There are several alternatives you can use to store and recover your credentials in production. You should check the official documentation by AWS about 'Using the Default Credential Provider Chain'. Basically, you can use any of these alternatives:
Environment variables.
Java system properties.
The default credential profiles file.
Amazon ECS container credentials.
Instance profile credentials.
Depending on which one you choose it would use a different code. You have guidelines on how to use them in the above link.
According to many advices, we should not configure IAM USER but using IAM Role instead to avoid someone managed to grab the user confidential in .aws folder.
Lets say I don't have any EC2 instances. Can I still able to perform S3 operation via AWS CLI? Says aws s3 ls
MacBook-Air:~ user$ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".
You are correct that, when running applications on Amazon EC2 instances or as AWS Lambda functions, an IAM role should be assigned that will provide credentials via the EC2 metadata service.
If you are not running on EC2/Lambda, then the normal practice is to use IAM User credentials that have been created specifically for your application, with least possible privilege assigned.
You should never store the IAM User credentials in an application -- there have been many cases of people accidentally saving such files into GitHub, and bad actors grab the credentials and have access to your account.
You could store the credentials in a configuration file (eg via aws configure) and keep that file outside your codebase. However, there are still risks associated with storing the credentials in a file.
A safer option is to provide the credentials via environment variables, since they can be defined through a login profile and will never be included in the application code.
I don't think you can use service roles on your personal machine.
You can however use multi-factor authentication for AWS CLI
You can use credentials on any machine not just EC2.
Follow the steps as described by the documentation for your OS.
http://docs.aws.amazon.com/cli/latest/userguide/installing.html
My use case is as follows:
I need to push some data into AWS SQS queue using JAVA SDK and by help of IAM role (not using credential provider implementation).
Is there any way to do that?
Thanks for help in advance.
It's been a while, but this is not currently the case, it is now possible to use assume role with the Java SDK with a user. You can configure credentials in your .aws/credentials file as follows:
[useraccount]
aws_access_key_id=<key>
aws_secret_access_key=<secret>
[somerole]
role_arn=<the ARN of the role you want to assume>
source_profile=useraccount
Then, when you launch, set an environment variable: AWS_PROFILE=somerole
The SDK will use the credentials defined in useraccount to call assumeRole with the role_arn you provided. You'll of course need to be sure that the user with those credentials has the permissions to assume that role.
Note that if you're not including the full Java SDK in your project (i.e. you're including just the libraries for the services you need), you also need to include the aws-java-sdk-sts library in your classpath for this to work.
It is also possible to do all of this programmatically using STSAssumeRoleSessionCredentialsProvider, but this would require you to directly configure all of the services so it might not be as convenient as the profile approach which should just work for all services.
You can use role based authentication only on EC2 Instances, ECS Containers and Lambda functions. It is not possible to use them locally or on on premise servers.
DefaultAWSCredentialsProviderChain will automatically pick the EC2 Instance Role if it can't find the credentials via any of other methods. You can also create a custom AWSCredentialsProviderChain object with only injecting a instance of InstanceProfileCredentialsProvider to it like here
AWSCredentialsProviderChain myCustomChain = new AWSCredentialsProviderChain(new InstanceProfileCredentialsProvider());
For more info: https://docs.aws.amazon.com/java-sdk/latest/developer-guide/java-dg-roles.html