In my development environment, I regularly switch between several AWS access keys. So in my ~/.aws/credentials file, I have several profiles. I can then use these profiles with aws-cli by specifying the --profile <name> flag.
I develop several apps that use the Ruby AWS SDK. They are setup to read credentials from the ~/.aws/credentials file (the SDK does this by default, see this configuration article).
The problem is that the SDK always reads from the default profile. I would like to be able to configure it to read from some of the other profiles. Is this possible?
I've tried setting the default profile in my environment with:
export AWS_DEFAULT_PROFILE=<profile name>
as outlined in configuring the AWS Command Line Interface but it seems that only works with the aws-cli and not with the SDK.
For SDK:
By setting the AWS_PROFILE environment variable to the profile you
want to use.
For CLI:
From: A New and Standardized Way to Manage Credentials in the AWS SDKs
"AWS_DEFAULT_PROFILE is correct for the AWS CLI. The AWS CLI respects
AWS_PROFILE if it is set, but AWS_DEFAULT_PROFILE takes precedence."
Related
I found way to configure AWS credentials by
aws configure
command. But this is not very comfortable for me since I'm using multiple AWS accounts. Is there any way to make it easy to configure AWS credentials and switch between them?
Yes. You can configure multiple profiles.
The easiest way is to use:
aws configure --profile <name>
You can then use it with:
aws s3 ls --profile <name>
If --profile is not specified, it will use the default profile.
All configuration information is stored in the ~/.aws/credentials and ~/.aws/config files.
See: Named profiles - AWS Command Line Interface
I want to install a Python3 package from a private Github repository onto an AWS EMR Spark cluster.
I know how to do this the dirty way by hardcoding credentials but what is the recommended best practice to do this safely ? I don't want to store credentials in a bootstrap script...
Thanks in advance.
Thanks to Maurice I've successfully implemented a safe process, following his option #2.
Create an access token with read credentials on github.
Store this in AWS Secrets Manager. In my case I named this secret "github-read-access"
Give access to this secret to the user that is going to query it, or in the case of a bootstrap EMR script, to the EMR roles.
Using aws CLI I store the token as an environment variable and install the package with the following commands:
export GITHUB_TOKEN=`aws secretsmanager get-secret-value --secret-id github-read-access |grep SecretString|cut -d ":" -f 3|cut -d '"' -f 2 |cut -d '\' -f1`
sudo pip3 install git+https://${GITHUB_TOKEN}#github.com/<USER_NAME>/<REPO_NAME>.git
Caveat: I haven't worked with custom EMR bootstrapping scripts, but I assume they're not too different from regular user data scripts.
There are some options:
Systems Manager Parameter Store: This is essentially something like the windows registry in AWS, a regional key-value store. You can store your credentials here under a name such as my/git/credentials and even encrypt them using the Key Management Service. In your bootstrapping script you can then request the credentials using the AWS CLI and use them to connect to the private git repository. This requires the instance role of the cluster to have permissions to access that parameter (and the KMS-Key if you've encrypted the value)
Secrets Manager: The general idea is similar to the SSM parameter store. The secrets manager also allows you to store your credentials in a secure way, in this case encryption is mandatory. It even offers lifecycle hooks to periodically renew the credentials should you require that. You can use the same technique I described in option 1) in the bootstrapping script. The requirements in terms of permissions are similar, although you definitely have to add KMS permissions and Secrets Manger permissions here. In this case you'd have to parse the JSON response from the Secrets Manager though.
I'd personally start with option 1, it will be cheaper. If you have specific audit/regulatory requirements, I'd look at option 2 - it's slightly more complex.
I had an AWS account configured to work with the CLI. The free tier expired so I setup another account. I created an IAM user ran aws configure and put in the credentials for that user. I have the default profile setup with that users credentials as well.
From the cli if I run the command aws s3 ls it will always show the buckets from the old account. If I specify the profile using aws s3 ls --profile GrantM then it lists the buckets from the correct account and IAM user.
The environment variables are set to the new user also. Can someone explain this and how to switch it to use my new account?
create or edit this file:
% vim ~/.aws/credentials
list as many key pairs as you like:
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
[user1]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
set a local variable to select the pair of keys you want to use:
% export AWS_PROFILE=user1
do what you like:
aws s3api list-buckets # any aws cli command now using user1 pair of keys
more details:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html
When you use aws configure without any additional arguments it should allow you to amend the default profile, which is the one that is being accessed when you specify no profile. By amending this you will not need to specify the --profile flag.
If you would also like to amend over named profiles you would simply use aws configure --profile $PROFILE_NAME, where you can just as above replace the credentials currently stored in the configuration.
Alternatively for Linux/MacOS you can access your credentials in ~/.aws/credentials or for Windows in %USERPROFILE%\.aws\credentials. You can modify these files to replace any values.
More information is available on the Named profiles documentation page.
I would not mix environment variables and credentials profiles, you'll just get confused.
Remove the environment variables, ensure that the default profile in your ~/.aws/credentials file (or %USERPROFILE%\.aws\credentials on Windows) is set to the new credentials, then run aws s3 ls. If it's not what you expected, then run aws s3 ls --debug to work out what you did wrong.
According to Credentials — Boto 3 Docs documentation, the Environment Variables will be used in preference to the configuration files.
Therefore, I suggest you remove the credentials from your Environment Variables, and just use the configuration files.
Depending upon your operating system, you could use unset, or remove them from where ever you put them in the Environment Variables.
run on terminal where you be running the cli commands
export AWS_PROFILE='PROFILE_NAME'
move this the bashrc/zshrc file to make this permanent or just add a default section to the .aws/config and .aws/configure. Run following command and input the credentials you want.
aws configure
works on mac and windows.
ok so I am baffled by this aws cli behavior. Basically what is going on is that when I set my AWS creds related in environment variable, AWS CLI forces me to pass --profile flag each time I use the CLI.
So basically when AWS_ACCESS_KEY_ID AND AWS_SECRET_ACCESS_KEY then I cannot run commands like aws s3 ls without passing --profile flag to it even though my profile is [default]
Also, jus to note the environment variable values and the values inside my /.aws/credentials
file is exactly same. Also, I tried to set both AWS_PROFILE and AWS_DEFAULT_PROFILE to default hoping that if all values such as keys,secret and profile are set in environment variable then I do not have to pass any --profile flag explicitly. Not having to pass this flag explicitly is very important for me at this point because if I am running an application which connects with aws and picks up default credentials, there is no easy way to pass profile information to that app.
my credentials file look like following:
[default]
aws_access_key_id = AKIA****
aws_secret_access_key = VpR***
My config file looks like following:
[default]
region = us-west-1
output = json
And my environment variables do have the same values for corresponding entries. for key, secret and profile at least.
Any idea on how to solve this issue?
The AWS CLI looks for credentials using a series of providers in a particular order. (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#config-settings-and-precedence)
Specifically:
Command line options – You can specify --region, --output, and --profile as parameters on the command line.
Environment variables – You can store values in the environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. If they are present, they are used.
CLI credentials file – This is one of the files that is updated when you run the command aws configure. The file is located at ~/.aws/credentials on Linux or macOS, or at C:\Users\USERNAME\.aws\credentials on Windows. This file can contain the credential details for the default profile and any named profiles.
CLI configuration file – This is another file that is updated when you run the command aws configure. The file is located at ~/.aws/config on Linux or macOS, or at C:\Users\USERNAME\.aws\config on Windows. This file contains the configuration settings for the default profile and any named profiles.
Container credentials – You can associate an IAM role with each of your Amazon Elastic Container Service (Amazon ECS) task definitions. Temporary credentials for that role are then available to that task's containers. For more information, see IAM Roles for Tasks in the Amazon Elastic Container Service Developer Guide.
Instance profile credentials – You can associate an IAM role with each of your Amazon Elastic Compute Cloud (Amazon EC2) instances. Temporary credentials for that role are then available to code running in the instance. The credentials are delivered through the Amazon EC2 metadata service. For more information, see IAM Roles for Amazon EC2 in the Amazon EC2 User Guide for Linux Instances and Using Instance Profiles in the IAM User Guide.
Another potential option for you would be to unset any colliding variables in your env and rely on the aws credentials file to provide the appropriate access credentials from the default entry.
I have deployed an application on EB with my own AWS account, and have done the same with another one.
I don't know how to switch out of my command line eb cli account. There are no eb CLI options to switch profiles like the normal AWS CLI...
Thanks!
Use named profiles via eb --profile x to use a specific profile from your AWS credentials file.
Or you can use multiple, distinct credentials files (see here) and change the AWS_CREDENTIAL_FILE environment variable to point to the relevant credentials file. You could even write a couple of shell scripts such as eb-project1 and eb-project2 that each export/set the relevant AWS_CREDENTIAL_FILE value and then simply execute eb so that you don't have to switch credentials manually - just run eb-project1 or eb-project2 as needed instead of running eb.