I have database credentials stored in AWS Secrets Manager which I'd like to pull and set as environment variables in a localised Codebuild container. I am using the 4.0 Codebuild image and the codebuild_build.sh script to test Codebuild on my local machine.
Currently my buildspec.yml looks like this at the start:
version: 0.2
env:
secrets-manager:
DATABASE_USER: <secret_name>:<secret_key_for_database_user>
When I run the script I get this:
agent_1 | [Container] 2020/12/06 21:30:16 Phase context status code: Secrets Manager Error Message: ResourceNotFoundException: Secrets Manager can't find the specified secret.
agent_1 | [Container] 2020/12/06 21:30:16 Runtime error (*clienterr.PhaseContextError: ResourceNotFoundException: Secrets Manager can't find the specified secret.)
I've tried obtaining the secret using the aws client and that works fine. I don't think it's an issue with my credentials because I get to this step when running the codebuild_build.sh script with the -c option and when hardcoding my aws secret key id and the secret key itself into the buildspec.yml. I have also tried using the full ARN of the secret but have had no luck.
Is there a trick that I'm missing here? Do I need to update the resource permissions on the secret itself inside the AWS Secrets Manager or on Codebuild, (both locally and on AWS)?
Related
I am running a cdk deploy build on circleCi, and when the step CDK deploy comes it gives me "Need to perform AWS calls for account ************, but no credentials have been configured".
But for the troubleshooting i tried other commands as well like
aws s3 ls
aws aws cloudformation list-stacks
These above commands we working fine, also able to run command to create a cloudformation with same config but not able to run cdk deploy. the access key and secret i am using has Admin access.
Set the creds with a profile name using aws-cli Orb in CircleCI and
try using the below command to deploy with CDK
cdk deploy --all --profile cdkprofile
For reference, in CircleCI
orbs:
aws-cli: circleci/aws-cli#2.0.3
commands:
env-setup:
description: AWS Env Setup
steps:
- aws-cli/setup:
profile-name: cdkprofile
aws-access-key-id: AWS_ACCESS_KEY_ID
aws-secret-access-key: AWS_SECRET_ACCESS_KEY
And assumption is AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set as CircleCI env variables
As a starting note: The best way to troubleshoot is with cdk [command] --verbose (see CLI ref)
CDK has an internal mechanism for finding credentials not directly using AWS CLI (AWS CLI is not a requirement for CDK to run)
In a similar situation with a CI tool, the issue was simply that the ~/.aws/credentials file did not exist (not that you need it with AWS CLI, but in the situation for CDK, it was required)
Credit to this issue reporting: https://github.com/aws/aws-cdk/issues/6947#issue-586402006
Solution tested for above:
For an EC2 running CI tool, with EC2 IAM role
Where ~/.aws/config exists and defined profile(s) with:
credential_source = Ec2InstanceMetadata
role_arn = arn:aws:iam:::role/role-to-assume-in-acctId
Create empty ~/.aws/credentials file
Example error for the problem solved above (from verbose output)
Resolving default credentials
Notices refreshed
Unable to determine the default AWS account: ProcessCredentialsProviderFailure: Profile myprofile did not include credential process
Other causes found in other issues/comments could relate to:
Duplicate profiles
Having credential_process in the profile, set to empty
Needing --profile parameter to be added
I have an app using:
SAM
AWS S3
AWS Lambda based on Docker
AWS SAM pipeline
Github function
In the Dockerfile I have:
RUN aws s3 cp s3://mylambda/distilBERT distilBERT.tar.gz
Resulting in the error message:
Step 6/8 : RUN aws s3 cp s3://mylambda/distilBERT distilBERT.tar.gz
---> Running in 786873b916db
fatal error: Unable to locate credentials
Error: InferenceFunction failed to build: The command '/bin/sh -c aws s3 cp s3://mylambda/distilBERT distilBERT.tar.gz' returned a non-zero code: 1
I need to find a way to store the credential in a secured manner. Is it possible with GitHub secrets or something?
Thanks
My solution may be a bit longer but I feel it solves your problem, and
It does not expose any secrets
It does not require any manual work
It is easy to change your AWS keys later if required.
Steps:
You can add the environment variables in Github actions(since you already mentioned Github actions) as secrets.
In your Github CI/CD flow, when you build the Dockerfile, you can create a aws credentials file.
- name: Configure AWS credentials
echo "
[default]
aws_access_key_id = $ACCESS_KEY
aws_secret_access_key = $SECRET_ACCESS_KEY
" > credentials
with:
ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY_ID }}
SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
In your Dockerfile, you can add instructions to COPY this credentials file and store it
COPY credentials credentials
RUN mkdir ~/.aws
RUN mv credentials ~/.aws/credentials
Changing your credentials requires just changing your github actions.
Docker by default does not have access to the .aws folder running on the host machine. You could either pass the AWS credentials as environment variables to the Docker image:
ENV AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
ENV AWS_SECRET_ACCESS_KEY=...
Keep in mind, hardcoding AWS credentials in a Dockerfile is a bad practice. In order to avoid this, you can pass the environment variables at runtime with using docker run -e MYVAR1 or docker run --env MYVAR2=foo arguments. Other solution would be to use an .env file for the environment variables.
A more involved solution would be to map a volume for the ~/.aws folder from the host machine in the Docker image.
I cannot seem to get my AWS provider to use my AWS credentials in the Docker Containers
I have ran all of these so far
serverless config credentials --provider provider --key key --secret secret
&
aws configure
☝️ ran through the prompts and plugged in my info
&
export AWS_ACCESS_KEY_ID=blah
export AWS_SECRET_ACCESS_KEY=blah
export AWS_DEFAULT_REGION=east-whatever
But still, get stopped by AWS
ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here:.
I have also generated and regenerated my IAM creds multipule time, also whne I run aws configure list it returns
Name Value Type Location
---- ----- ---- --------
profile <not set> None None
access_key ****************67GT shared-credentials-file
secret_key ****************6glt shared-credentials-file
region us-east-2 config-file ~/.aws/config
I am not sure what else to do, I am on Ubuntu 20.04 using Bash
I will take approach of setting everything using aws-cli ( hence aws cli installation is imp)
install aws cli
run aws configure -> enter your creds, this info will be stored in profile called default in ~/.aws/credentials file. docs.
verify your configuration by checking ~/.aws/config ~/.aws.credentials file
for serverless frame work deployment use this serverless deploy.
if mutiple profiles needs to be used use sls deploy --aws-profile profile-name
The structure of the wait command is:
$ aws <command> wait <subcommand> [options and parameters]
However in DevOps it only seems to support:
$ aws <command> <subcommand> [options and parameters]
See example below where there is a Command and Subcommand. Where does the Wait go? I'm trying to run this command https://awscli.amazonaws.com/v2/documentation/api/latest/reference/elasticbeanstalk/wait/environment-updated.html
I had to set the Subcommand to wait and move the environment-updated down into the Options and parameters
It looks that you want be able to do this using extension. However, you have aws CLI installed on the agent so what you need is to setup few variables and then call your commands from powershell step.
Supply standard AWS environment variables in the build agent process
You can specify credentials with standard named AWS environment variables. These variables can be used to get credentials from a custom credentials store.
The following are all the supported standard named AWS environment variables:
AWS_ACCESS_KEY_ID – IAM access key ID.
AWS_SECRET_ACCESS_KEY – IAM secret access key.
AWS_SESSION_TOKEN – IAM session token.
AWS_ROLE_ARN – Amazon Resource Name (ARN) of the role you want to assume.
AWS_REGION – AWS Region code, for example, us-east-2.
You can also create a feature request on github to support wait command by extension.
I have configured serverless with key and secret.
When i try to run serverless deploy it says:
ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here:.
Please help me what to do
This is how to setup AWS credentials on the local:
Sign in to AWS console
Go to your AWS account overview
Account menu in the upper-right (has your name on it)
sub-menu: Security Credentials
Copy <Access Key ID>
Copy <Secret Access Key>
Run on local serverless config credentials --provider aws --key <Access Key ID> --secret <Secret Access Key>
You can set up your serverless credentials by using the serverless config credentials command, with --provider aws and using --key and --secret to set your AWS Access Key ID and Secret Access Key, respectively.
See the Serverless AWS - Config Credentials documentation for Serverless specifics, and this AWS document on security credentials in case you need to figure out what your credentials are.
Also give your profile and region configuration in your yml file
provider:
name: aws
runtime: python2.7
lambdaHashingVersion: 20201221
profile: 'serverless-admin'
region: 'us-east-1'
I had the same problem. I try everything. I configure my credentials (AWS key and secret) with both aws cli and serverless cli but still got that same error when I run:
sls deploy -v
I was using CMD before then I realize something when I run the command
aws configure list
the output is
Name Value Type Location
---- ----- ---- --------
profile <not set> None None
access_key ****************SOU5 shared-credentials-file
secret_key ****************Cki5 shared-credentials-file
region eu-west-1 config-file ~/.aws/config
the path of the config file is
~/.aws/config
and CMD don't know what ~/ PATH IS but Bash does know
😈
then I run the
sls deploy -v
on bash terminal and it worked 🎉🙌🙌🎉🐱🏍
For those who has the same problem, even when already configured the AWS credentials for serverless, by running the command serverless config credentials:
I had the same problem; even when I had set up properly my AWS credentials on serverless, when I run the serverless deploy command, I got this: ServerlessError: AWS provider credentials not found.
How did I solved it? By installing and configuring the AWS CLI Tool. Once I installed it and configured it with my AWS credentials, and then run again the serverless deploy command, it just works fine.
I guess the serverless command, behind the scenes, uses the AWS CLI to access AWS and provision the AWS resources.
I hope this could be helpful for someone.