I am wanting to set up a recursive sync from a Linux machine (Fedora) to an AWS S3 bucket. I am logged into Linux as root and have an AWS Key and Secret associated with a specific AWS user "Lisa".
I have installed aws-cli, s3cmd, and attempted to configure both. I have verified the aws/configure and aws/credentials files both have a default user and a "Lisa" user with Access Key and Secret pairs. I receive errors stating that Access is Denied, access key and secret pair not found. I have researched this on the web and verified that there are no environment variables that could be overriding the configure & credential files. I have also granted full access permissions to the bucket created through the AWS Console to all logged in users. I have not rotated the keys, as they were first created a week ago, and I was able to log-in & set-up the AWS console using that same key pair.
What else should I be doing before rotating the keys?
It looks like you haven't configured AWS credentials correctly. Make sure that you have correct access keys in your credentials file. If you don't specify any profiles, awscli uses the default profile.
~/.aws/credentials
[default]
aws_access_key_id=AKIAIDEFAULTKEY
aws_secret_access_key=Mo9T7WNO….
[Lisa]
aws_access_key_id=AKIAILISASKEY
aws_secret_access_key=H0XevhnC….
This command uses the default profile:
aws s3 ls
This command uses Lisa profile:
aws s3 ls --profile Lisa
You can set an environment variable to override the default profile.
export AWS_DEFAULT_PROFILE=Lisa
Now this command uses the profile Lisa:
aws s3 ls
If you don't know which profile is active, you can just invoke the following command:
aws sts get-caller-identity
You seem to have several terms intermixed, so it's worth knowing the difference:
Username and password is used to login to the web-based management console. They are short, to be human-readable and easy to remember.
Access Key (starting with AKIA) and Secret Key is used for making API calls. It is also used by the AWS CLI (which makes API calls on your behalf)
Key pair consists of a public and private key, used for authenticating SSH connections. It is a very long block of text.
You mention that an Access Key is not found. This could be because the wrong type of credential is being provided.
Related
I am on a federated account that only allows for 60 minutes access tokens. This makes using AWS difficult since I have to constantly relog in with MFA, even for the AWS CLI on my machine. I'm fairly certain that any programmatic secret access key and token I generate would be useless after an hour.
I am writing a .NET program (.NET framework 4.8) that will run on a EC2 instance to read and write from an S3 bucket. As per the documentation example, they give this example to initalize the AmazonS3Client:
// Before running this app:
// - Credentials must be specified in an AWS profile. If you use a profile other than
// the [default] profile, also set the AWS_PROFILE environment variable.
// - An AWS Region must be specified either in the [default] profile
// or by setting the AWS_REGION environment variable.
var s3client = new AmazonS3Client();
I've looked into SecretManager and ParameterStore, but that would matter if the programmatic access keys go inactive after an hour. Perhaps there is another way to give the program access to S3 and the SDK...
If I cannot use access keys and tokens stored in a file, could I use the IAM access that awscli uses? For example, I can type into powershell aws s3 ls s3://mybucket to list and read files from s3 to the ec2 instance. Could the .NET SDK use the same credentials to access the S3 bucket?
My goal is to access my s3 buckets from the command line using my AWS educate account.
I expected to get a list of my s3 buckets in the command prompt. I typed in this command. aws s3 ls
I actually received an error message saying Invalid Access Key ID.
They shown do not match they key on the home age of my AWS educate account.
How do I change the keys listed to match the ones on my AWS Educate home page? I think if I correct this then I will be able to access my s3 buckets with aws cli.
Run:
aws configure
And follow the prompts to configure a new Access Key and Secret Access Key.
If this isn't working, there are two other things you can check:
Have you set any of the following environment variables? https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html. These can override the ones set using aws configure
If that fails, check that $HOME/.aws/credentials is not write protected, or even try updating the credentials manually.
My requirement to upload file from local to s3 using aws cli but don't want to use access ID and secret access key while running in command line.
Any suggestions!
It is recommended that you never put AWS credentials in program code.
If the code is running on an Amazon EC2 instance, assign an IAM Role to the instance. The code will automatically detect and use these credentials.
If the code is running on your own computer, run the AWS Command-Line Interface (CLI) aws configure command and enter your IAM credentials (Access Key + Secret Key). They will be stored in the ~/.aws/credentials file and will be automatically accessed by your code.
SETUP
I created a new aws user via the aws web console, and selected both console and programmatic/cli access
I have added the AdministratorAccess policy directly to it.
I have not enabled MFA for this user
I have verified that my credentials file within the aws directory contains the proper values for aws_access_key_id and aws_secret_access_key
I have verified that my config file within the aws directory does not contain any lines that would overwrite data for the profile
I am verifying I am using the correct profile info by with aws configure list
THE ISSUE
Executing aws ec2 describe-regions returns:
An error occurred (UnauthorizedOperation) when calling the DescribeRegions operation: You are not authorized to perform this operation.
The error is pretty straightforward, but I'm not sure what else I can do to authorize this user. I had a coworker follow the same steps and the CLI worked as expected for him.
I researched the steps from This S.O. post but am still scratching my head.
Your AWS CLI is getting credentials from somewhere else. See Configuration Settings and Precedence:
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedence
Make sure it is not getting the credentials from environment variables or from other locations. The AWS CLI looks for credentials and configuration settings in the following order:
Command Line Options – region, output format and profile can be specified as command options to override default settings.
Environment Variables – AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.
1)The AWS credentials file – ~/.aws/credentials on Linux, OS X, or Unix, or at C:\Users\USERNAME .aws\credentials on Windows.
Can contain multiple named profiles in addition to a default profile.
2)The CLI configuration file – at ~/.aws/config on Linux, OS X, or Unix, or at C:\Users\USERNAME .aws\config on Windows. Can contain a default profile, named profiles, and CLI specific configuration parameters for each.
3)Instance profile credentials – these credentials can be used on EC2 instances with an assigned instance role
For different AWS services, I need different IAM users to secure the access control. Sometimes, I even need to use different IAM user credentials within a single project in a EC2 instance. What's the proper way to manage this and how I can deploy/attach these IAM user credentials to a single EC2 instance?
While I fully agree with accepted answer that using static credentials is one way of solving this problem, I would like to suggest some improvements over it (and proposed Secrets Manager).
What I would advise as architectural step forward to achieve full isolation of credentials, having them dynamic, and not stored in central place (Secrets Manager proposed above) is dockerizing application and running on AWS Elastic Container Service (ECS). This way you can assign different IAM role to different ECS Tasks.
Benefits over Secrets Manager solution
- use case of someone tampering with credentials in Secrets Manager is fully avoided, as credentials are of dynamic nature (temporary, and automatically assumed through SDKs)
Credentials are managed on AWS side for you
Only ECS Service can assume this IAM role, meaning you can't have actual person stealing the credentials, or developer connecting to production environment from his local machine with this credentials.
AWS Official Documentation for Task Roles
The normal way to provide credentials to applications running on an Amazon EC2 instance is to assign an IAM Role to the instance. Temporary credentials associated with the role when then be provided via Instance Metadata. The AWS SDKs will automatically use these credentials.
However, this only works for one set of credentials. If you wish to use more than one credential, you will need to provide the credentials in a credentials file.
The AWS credentials file can contain multiple profiles, eg:
[default]
aws_access_key_id = AKIAaaaaa
aws_secret_access_key = abcdefg
[user2]
aws_access_key_id = AKIAbbbb
aws_secret_access_key = xyzzzy
As a convenience, this can also be configured via the AWS CLI:
$ aws configure --profile user2
AWS Access Key ID [None]: AKIAbbbb
AWS Secret Access Key [None]: xyzzy
Default region name [None]: us-east-1
Default output format [None]: text
The profile to use can be set via an Environment Variable:
Linux: export AWS_PROFILE="user2"
Windows: set AWS_PROFILE="user2"
Alternatively, when calling AWS services via an SDK, simply specify the Profile to use. Here is an example with Python from Credentials — Boto 3 documentation:
session = boto3.Session(profile_name='user2')
# Any clients created from this session will use credentials
# from the [user2] section of ~/.aws/credentials.
dev_s3_client = session.client('s3')
There is an equivalent capability in the SDKs for other languages, too.