I'm currently using Traefik and Lego in order to have HTTPS connection for my docker containers (as mentioned here)
In the following documentation, it's mentioned that I need to use the following provider to do DNS Challenge.
But I get this error:
AccessDeniedException: User: arn:aws:sts::<USER_ID>:assumed-role/AmazonLightsailInstanceRole/<AN_ID> is not authorized to perform: lightsail:CreateDomainEntry on resource: arn:aws:lightsail:us-east-1:<INSTANCE_ID>:*
and another for DeleteDomainEntry, even though I have lightsail:* on Resource: * permission on the IAM user used for configuration.
If I understand correctly Lightsail is managed separately for the other AWS services and thus we need to use STS for connecting to it (tell me if I'm wrong). So my question is this, how can I set the permissions for the temporary token to be able to do CreateDomainEntry and DeleteDomainEntry?
Further information
My instance's region is eu-west-3 (I tried changing the region in Lego config, doesn't work)
The <USER_ID> seen in the error does not correspond to the id found in the ARN of the domain. It correspond to the first number in the supportCode when doing aws lightsail get-domains --region us-east-1 in the CLI.
Lego and Traefik do not call the AssumeRole directly and do not create the temporary token (checked source code)
I'm using AWS_ACCESS_KEY_ID_FILE and AWS_SECRET_ACCESS_KEY_FILE in Traefik environment configuration.
The error message tells that Lego made the request using the IAM role assigned to your lightsail instance. I guess your instance lacks permissions to modify DNS settings for lightsail.
You should create a new user in AWS IAM and enable programmatic access in order to obtain AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Then, pass those values as environment variables to your containers running Lego. Lego will use those env vars to authenticate with Lightsail APIs in us-east-1. [1]
My instance's region is eu-west-3 (I tried changing the region in Lego config, doesn't work)
Your Lego instance must call AWS APIs in us-east-1, see [2][3].
Lego and Traefik do not call the AssumeRole directly and do not create the temporary token
I guess Traefik/Lego assume the lightsail instance role automatically using EC2 instance metadata service, see [4]:
For applications, AWS CLI, and Tools for Windows PowerShell commands that run on the instance, you do not have to explicitly get the temporary security credentials—the AWS SDKs, AWS CLI, and Tools for Windows PowerShell automatically get the credentials from the EC2 instance metadata service and use them. To make a call outside of the instance using temporary security credentials (for example, to test IAM policies), you must provide the access key, secret key, and the session token.
I'm using AWS_ACCESS_KEY_ID_FILE and AWS_SECRET_ACCESS_KEY_FILE in Traefik environment configuration.
I could not find those env vars in the Lego source code [1]. Make sure that Lego is actually using your configured AWS credentials. The error message posted above suggests it's not using them and falls back to the instance profile instead.
[1] https://github.com/go-acme/lego/blob/master/providers/dns/lightsail/lightsail.go#L81
[2] https://docs.aws.amazon.com/cli/latest/reference/lightsail/create-domain-entry.html#examples
[3] https://github.com/go-acme/lego/blob/master/providers/dns/lightsail/lightsail.go#L69
[4] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials
There were too many confusing things. Martin Löper's answer and answer on the github issue I opened helped me to clear things out.
Here is what was confusing:
Lego lightsail provider documentation is listing the environment variable and then say The environment variable names can be suffixed by _FILE to reference a file instead of a value. Turns out, Lego's code never call their getOrFile method on the credentials. Furthermore, AWS sdk does not check variables with the _FILE suffix for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
The error message from AWS is little bit misleading. I thought all that time that it was a permission problem but in fact it was a authentication problem (little bit different in my opinion).
And here is how to solve it (little bit different from what proposed):
I use the AWS_SHARED_CREDENTIALS_FILE (mentioned here) environment variable so that I can use docker secrets by specifying /run/secrets/aws_shared_credentials file. This is more secure (more info here). AWS sdk will automatically detect this env variable and initialize this new session correctly.
I am having issues deploying my docker images to aws ecr as part of a terraform deployment and I am trying to think through the best long term strategy.
At the moment I have a terraform remote backend in s3 and dynamodb on let's call it my master account. I then have dev/test etc environments in separate accounts. The terraform deployment is currently run off my local machine (mac) and uses the aws 'master' account and its credentials which in turn assumes a role in the target deployment account to create the resources as per:
provider "aws" { // tell terraform which SDK it needs to load
alias = "target"
region = var.region
assume_role {
role_arn = "arn:aws:iam::${var.deployment_account}:role/${var.provider_env_deployment_role_name}"
}
}
I am creating a number of ecs services with Fargate deployments. The container images are built in separate repos by GitHub Actions and saved as GitHub packages. These package names and versions are being deployed after the creation of the ecr and service (maybe that's not ideal thinking about it) and this is where the problems arise.
The process is to pull the image from GitHub Packages, retag it and upload to the ecr using multiple executions of a null_resource local-exec. Works fine stand alone but has problems as part of the terraform process. I think the reason is that the other resources use the above provider to get permissions but as null_resource does not accept a provider it cannot get the permissions this way. So I have been passing the aws creds values into the shell. Not convinced this is really secure but that's currently moot as it ain't working either. I get this error:
Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `The specified item already exists in the keychain.``
Part of me thinks this is the wrong approach and that as I migrate to deploying via a Github action I can separate the infrastructure deployment via terraform from what is really the application deployment and just use GitHub secrets to reset the credentials values then run the script.
Alternatively, maybe the keychain thing just goes away and my process will work fine? Secure ??
That's fine for this scenario but it isn't really a generic approach for all my use cases.
I am shortly going to start deploying multiple aws lambda functions with docker containers. Haven't done it before but it looks like the process is going to be: create ecr, deploy container, deploy lambda function. This really implies that the container deployment should integral to the terraform deployment which loops back to my issue with the local-exec??
I found Actions to deploy to ECR which would imply splitting the deployments into multiple files but that seems inelegant and potentially brittle.
Maybe there is a simple solution, but given where I am trying to go with this, what is my best approach?
I know this isn't a complete answer, but you should be pulling your AWS creds from environment variables. I don't really understand if you need credentials for different accounts, but if you do then swap them during the progress of your action. See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html . Terraform should pick these up and automatically use them for AWS access.
Instead of those hard coded access key/secret access keys I'd suggest making use of Github/AWS's ability to assume role through temporary credentials with OIDC https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
You'd likely only define one initial role that you'd authenticate into and from there assume into the other accounts you're deploying into.
These the assume role credentials are only good for an hour and do not have the operation overhead of having to rotate them.
As suggested by Kevin Buchs answer...
My primary issue was related to deploying from a mac and the use of the keychain. As this was not on the critical path I went round it and set up a GitHub Action.
The Action loaded environmental variables from GitHub secrets for my 'master' aws account credentials.
AWS_ACCESS_KEY_ID: ${{ secrets.NK_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.NK_AWS_SECRET_ACCESS_KEY }}
I also loaded the target accounts credentials into environmental variables in the same way BUT with the prefix TF_VAR.
TF_VAR_DEVELOP_AWS_ACCESS_KEY_ID: ${{ secrets.DEVELOP_AWS_ACCESS_KEY_ID }}
TF_VAR_DEVELOP_AWS_SECRET_ACCESS_KEY: ${{ secrets.DEVELOP_AWS_SECRET_ACCESS_KEY }}
I then declare terraform variables which will be automatically populated from the environment variables.
variable "DEVELOP_AWS_ACCESS_KEY_ID" {
description = "access key for the dev account"
type = string
}
variable "DEVELOP_AWS_SECRET_ACCESS_KEY" {
description = "secret access key for the dev account"
type = string
}
And when I run a shell script with a local exec:
resource "null_resource" "image-upload-to-importcsv-ecr" {
provisioner "local-exec" {
command = "./ecr-push.sh ${var.DEVELOP_AWS_ACCESS_KEY_ID} ${var.DEVELOP_AWS_SECRET_ACCESS_KEY} "
}
}
Within the script I can then use these arguments to set the credentials eg
AWS_ACCESS=$1
AWS_SECRET=$1
.....
export AWS_ACCESS_KEY_ID=${AWS_ACCESS}
export AWS_SECRET_ACCESS_KEY=${AWS_SECRET}
and the script now has credentials to do whatever.
I was able to install CLI on windows 16 AWS instance. when I try "aws ec2 describe-instances" CLI command, I get the following error
CLI command "describe-instances" throw error "An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials"
In .aws\config file I have following content:
[default]
region = us-west-2
How can authorization fail when it took my access key id and secret access key without any issue.
Verify if your datetime is sync ok.
use: ntpdate ntp.server
bests
I deleted my two configuration files from .aws directory and re-ran "aws config"
That fixed the problem for me.
My Steps:
Go to your .aws directory under Users e.g. "c:\Users\Joe\.aws"
Two files: configure and credential. Delete both files
Rerun configure: "aws configure"
Note when you run aws configure you will need the AWS Access and Secret Key. If you don't have them you can just create another.
Steps:
Goto "My Security Credentials" Under you Account Name in AWS Console.
Expand Access Key panel.
Create New Access Key.
When you first ran aws configure, it just populated the local credentials in %UserProfile%\.aws\credentials; it didn't validate them with AWS.
(aws-cli doesn't know what rights your user has until it tries to do an operation -- all of the access control happens on AWS's end. It just tries to do what you ask, and tells you if it doesn't have access, like you saw.)
That said, if you're running the CLI from an AWS instance, you might want to consider applying a role to that instance, so you don't have to store your keys on the instance.
My Access and Security keys are correct. My server time was good. I got error while using Ap-south-1 region. After I changed my region to us-west-2, it worked without any problem.
I tried setting that too on my windows environment. didn't work and getting error above.
so I tried setting my environment
SET AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
SET AWS_SECRET_ACCESS_KEY=***YOUR_SECRET_ACCESS_KEY*
and then tried running command like "aws ec2 describe-instance"
I tried many things. Finally, just uninstalling and installing again (not repairing) did the trick. Just make sure to save a copy of your credentials (key and key ID) to use later when calling aws configure.
I created a new Access Key and configured that in the AWS CLI with aws configure. It created the .ini file in ~/.aws/config. When I run aws s3 ls it gives:
A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.
AmazonS3FullAccess policy is also attached to the user. How to fix this?
It might be happening that you have the old keys exported via env variables (bash_profile) and since the env variables have higher precedence over credential files it is giving the error "the access key id does not exists".
Remove the old keys from the bash_profile and you would be good to go.
Happened with me once earlier when I forgot I have credentials in bash_profile and gave me headache for quite some time :)
It looks like some values have been already set for the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
If it is like that, you could see some values when executing the below commands.
echo $AWS_SECRET_ACCESS_KEY
echo $AWS_ACCESS_KEY_ID
You need to reset these variables, if you are using aws configure
To reset, execute below commands.
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
Need to add aws_session_token in credentials, along with aws_access_key_id,aws_secret_access_key
None of the up-voted answers work for me. Finally I pass the credentials inside the python script, using the client API.
import boto3
client = boto3.client(
's3',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
aws_session_token=SESSION_TOKEN)
Please notice that the aws_session_token argument is optional. Not recommended for public work, but make life easier for simple trial.
For me, I was relying on IAM EC2 roles to give access to our machines to specific resources.
I didn't even know there was a credentials file at ~/.aws/credentials, until I rotated/removed some of our accessKeys at the IAM console to tighten our security, and that suddenly made one of the scripts stop working on a single machine.
Deleting that credentials file fixed it for me.
I made the mistake of setting my variables with quotation marks like this:
AWS_ACCESS_KEY_ID="..."
You may have configured AWS credentials correctly, but using these credentials, you may be connecting to some specific S3 endpoint (as was the case with me).
Instead of using:
aws s3 ls
try using:
aws --endpoint-url=https://<your_s3_endpoint_url> s3 ls
Hope this helps those facing the similar problem.
you can configure profiles in the bash_profile file using
<profile_name>
aws_access_key_id = <access_key>
aws_secret_access_key = <acces_key_secret>
if you are using multiple profiles. then use:
aws s3 ls --profile <profile_name>
You may need to set the AWS_DEFAULT_REGION environment variable.
In my case, I was trying to provision a new bucket in Hong Kong region, which is not enabled by default, according to this:
https://docs.aws.amazon.com/general/latest/gr/s3.html
It's not totally related to OP's question, but to topic per se, so if anyone else like myself finds trapped on this edge case:
I had to enable that region manually, before operating on that AWS s3 region, following this guide: https://docs.aws.amazon.com/general/latest/gr/rande-manage.html
I have been looking for information about this problem and I have found this post. I know it is old, but I would like to leave this post in case anyone has problems.
Okay, I have installed the AWS CLI and opened:
It seems that you need to run aws configure to add the current credentials. Once changed, I can access
Looks like ~/.aws/credentials was not created. Try creating it manually with this content:
[default]
aws_access_key_id = sdfesdwedwedwrdf
aws_secret_access_key = wedfwedwerf3erfweaefdaefafefqaewfqewfqw
(on my test box, if I run aws command without having credentials file, the error is Unable to locate credentials. You can configure credentials by running "aws configure".)
Can you try running these two commands from the same shell you are trying to run aws:
$ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
and then try aws command.
another thing that can cause this, even if everything is set up correctly, is running the command from a Makefile. for example, I had a rule:
awssetup:
aws configure
aws s3 sync s3://mybucket.whatever .
when I ran make awssetup I got the error: fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.. but running it from the command line worked.
Adding one more answer since all the above cases didn't work for me.
In AWS console, check your credentials(My Security Credentials) and see if you have entered the right credentials.
Thanks to this discussion:
https://forums.aws.amazon.com/message.jspa?messageID=771815
This could happen because there's an issue with your AWS Secret Access Key. After messing around with AWS Amplify, I ran into this issue. The quickest way is to create a new pair of AWS Access Key ID and AWS Secret Access Key and run aws configure again.
I works for me. I hope this helps.
To those of you who run aws s3 ls and getting this exception. Make sure You have permissions to all regions under the provided AWS Account. When running aws s3 ls you try to pull all the s3 buckets under the AWS Account. therefore, in case you don't have permissions to all regions, you'll get this exception - An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.
Follow Describing your Regions using the AWS CLI for more info.
I had the same problem in windows and using the module aws-sdk of javascript. I have changed my IAM credentials and the problem persisted even if i give the new credentials through the method update like this
s3.config.update({
accessKeyId: 'ACCESS_KEY_ID',
secretAccessKey: 'SECRET_ACCESS_KEY',
region: 'REGION',
});
After a while i found that the module aws-sdk had created a file inside the folder User on windows with this path
C:\Users\User\.aws\credentials
. The credentials inside this file take precedence over the other data passed through the method update.
The solution for me was to write here
C:\Users\User\.aws\credentials
the new credentials and not with the method s3.config.update
Kindly export the below variables from the credential file from the below directory.
path = .aws/
filename = credentials
export aws_access_key_id = AK###########GW
export aws_secret_access_key = g#############################J
Hopefully this saves others from hours of frustration:
call aws.config.update({ before initializing s3.
const AWS = require('aws-sdk');
AWS.config.update({
accessKeyId: 'AKIAW...',
secretAccessKey: 'ptUGSHS....'
});
const s3 = new AWS.S3();
Credits to this answer:
https://stackoverflow.com/a/61914974/11110509
I tries below steps and it worked:
1. cd ~
2. cd .aws
3. vi credentials
4. delete
aws_access_key_id =
aws_secret_access_key =
by placing cursor on that line and pressing dd (vi command to delete line).
Delete both the line and check gain.
If you have an AWS Educate account and you get this problem:
An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records".
The solution is here:
Go to your C:/ drive and search for .aws folder inside your main folder in windows.
Inside that folder you get the "credentials" file and open it with notepad.
Paste the whole key credential from AWS account to the same notepad and save it.
Now you are ready to use you AWS Educate account.
Assuming you already checked Access Key ID and Secret... you might want to check file team-provider-info.json which can be found under amplify/ folder
"awscloudformation": {
"AuthRoleName": "<role identifier>",
"UnauthRoleArn": "arn:aws:iam::<specific to your account and role>",
"AuthRoleArn": "arn:aws:iam::<specific to your account and role>",
"Region": "us-east-1",
"DeploymentBucketName": "<role identifier>",
"UnauthRoleName": "<role identifier>",
"StackName": "amplify-test-dev",
"StackId": "arn:aws:cloudformation:<stack identifier>",
"AmplifyAppId": "<id>"
}
IAM role being referred here should be active in IAM console.
If you get this error in an Amplify project, check that "awsConfigFilePath" is not configured in amplify/.config/local-aws-info.json
In my case I had to remove it, so my environment looked like the following:
{
// **INCORRECT**
// This will not use your profile in ~/.aws/credentials, but instead the
// specified config file path
// "dev": {
// "configLevel": "project",
// "useProfile": false,
// "awsConfigFilePath": "/Users/dev1/.amplify/awscloudformation/cEclTB7ddy"
// },
// **CORRECT**
"dev": {
"configLevel": "project",
"useProfile": true,
"profileName": "default",
}
}
Maybe you need to active you api keys in the web console, I just saw that mine were inactive for some reason...
Thanks, everyone. This helped to solve.
Something somehow happened which changed the keys & I didn't realize since everything was working fine until I connected to S3 from a spark...then from the command line also error started coming even in AWS s3 ls
Steps to solve
Run AWS configure to check if keys are set up (verify from last 4 characters & just keep pressing enter)
AWS console --> Users --> click on the user --> go to security credentials--> check if the key is the same that is showing up in AWS configure
If both not the same, then generate a new key, download csv
run --> AWS configure, set up new keys
try AWS s3 ls now
Change keys at all places in my case it was configs in Cloudera.
I couldn't figure out how to get the system to accept my Vocareum credentials so I took advantage of the fact that if you configure your instance to use IAM roles, the SDK automatically selects the IAM credentials for your application, eliminating the need to manually provide credentials.
Once a role with appropriate permissions was applied to the EC2 instance, I didn't need to provide any credentials.
Open the ~/.bash_profile file and edit the info with the new values that you received at the time of creating the new user:
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_DEFAULT_REGION=us-east-1
Afterward, run the command:
source ~/.bash_profile
This will enable the new keys for the local machine. Now, we will need to configure the info in the terminal as well. Run the command -
aws configure
Provide the new values as requested and you are good to go.
In my case, I was using aws configure
However, I hand-edited the .aws/config file to export the KeyID and key environment variables.
This apparently caused a silent error and saw the error listed above.
I solved this by destroying the .aws directory and running aws configure again.
I have encountered this issue when trying to export RDS Postgres data to S3 following this official guide.
TL;DR Troubleshooting tips:
Reset RDS credentials using:
DROP EXTENSION aws_s3 CASCADE;
DROP EXTENSION aws_commons CASCADE;
CREATE EXTENSION aws_s3 CASCADE;
Delete and add DB instance role used for s3Export feature. Optionally reset RDS credentials (previous action point) once again after that.
Below you will find more details on my case.
In particular, I have encountered:
[XX000] ERROR: could not upload to Amazon S3
Details: Amazon S3 client returned 'The AWS Access Key Id you provided does not exist in our records.'.
To be able to perform export to S3, RDS DB instance should be configured to assume a role with permission to write to S3 bucket, the guide describes these steps.
The reason of an error was in aws_s3.query_export_to_s3 Postgres procedure using some (cached?) invalid assumed credentials. I am still not aware which credentials has it been using but I have managed to achieve the same behaviour using AWS CLI:
I have assumed a role (aws sts assume-role),
And then tried to perform another action (aws s3 cp in particular) with this credentials without session token (only AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY without AWS_SESSION_TOKEN).
This resulted in the same error from AWS CLI: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.
In short: hard resetting RDS credentials helped.
I just found another cause/remedy for this error/situation. I was getting the error running a PowerShell script. The error was happening on an execution of Write-S3Object. I have been working with AWS for a while now and have been running this script with success, but had not run it in a while.
My usual method of setting AWS credentials is:
Set-AWSCredential -ProfileName <THE_PROFILE_NAME>
I tried the "aws configure" command and every other recommendation in this forum post. No luck.
Well, I am aware of the .aws\credentials file and took a look in there. I have only three profiles, with one being [default]. Everything was looking good, but then I noticed a new element in there, present in all 3 profiles, that I had not seen before:
toolkit_artifact_guid=64GUID3-GUID-GUID-GUID-004GUID236
(GUID redacting added by me)
Then I noticed this element differed between the profile I was running with and the [default] profile, which was the same profile, except for that.
On a hunch I changed the toolkit_artifact_guid in the [default] to match it to my target profile, and no more error. I have no idea why.