I am deploying a django app to AWS Elastic Beanstalk and initially I am defining my environment variable in .ebextensions/django.config
Some of those variable are sensitive and I don't want to push them to Git, so I want to encapsulate those variables in a single file (for example: .env) that will not be pushed to Git.
Plan A:
A way to include .env in my config file, but I didn't find a way to do it supposedly like:
option_settings:
aws:elasticbeanstalk:application:environment:
include .env
aws:elasticbeanstalk:container:python:
WSGIPath: mydjangoapp.wsgi:application
Cons:
The environment variables are shown as plain text in AWS console at Elastic Beanstalk > Environments > my-environment > Configuration > Environment properties, although I know the fact that they are only readable by the authorised AWS users who have permission to it.
Pros:
Ability to update only and directly the environment variables from AWS console without requiring new deployment.
Plan B:
Nearly same as plan A, but without including .env file in config file. It is by using eb setenv to set the sensitive environments, but it is should be typed explicitly one-by-one, not from an external file, so it is headache if they are a lot
Plan C:
Remove the sensitive variables at all from my config file and load the .env file from my django app itself.
Cons:
If I want to update any of those variables, I have to deploy a new version of my application.
Although .env file will not pushed to Git and it can be shared between developers internally, it should be pushed with the deployment package and it will appear in the application ec2 instance directory.
Pros:
Hide sensitive information even from the AWS console
Questions:
Is plan A applicable by any means? I could do the same in google cloud app engine yaml files, but I could not find a way to do it in AWS EB configurations https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/beanstalk-environment-configuration-advanced.html
What is the best practice here? Is there another plan to do?
Regarding plan B - this answer shows a great way to dump all vars from a .env into the eb setenv ... command!
I require files to be downloaded from AWS S3 during container build, however I've been unsuccessful to provide the AWS credentials to the build process without actually hardcoding them in the Dockerfile. I get the error:
docker fatal error: Unable to locate credentials
despite previously having executed:
aws configure
Moreover, I was not able to use --build-arg for this purpose.
My question: is it possible to have these credentials in build time without hardcoding them in the Dockerfile and if so how?
Thank you for your attention.
Using the --build-arg flag is the correct way to do it, if you don't mind that the values can be seen by everyone using docker history, however you must use the ARG directive, not the ENV directive to specify them in your Dockerfile.
Here is an example Dockerfile that I have used with AWS credentials. It takes in the aws credentials as build arguments, including a default argument for the AWS_REGION build argument. It then performs a basic aws action, in this case logging into ecr.
FROM <base-image>:latest # an image I have that has `aws` installed
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_REGION=us-west-2
RUN aws ecr get-login --no-include-email | bash
CMD ["npm", "start"]
You then build the image with the following command:
docker build -t testing --build-arg AWS_ACCESS_KEY_ID=<Your ID Here> \
--build-arg AWS_SECRET_ACCESS_KEY=<Your Key Here> .
Please be aware that the values of the --build-arg arguments can be seen by anyone with access to the image later on using docker history.
It is possible to hide the values from docker history. In order to achieve this you must use multistage-build. This will make your history only visible from the second FROM on.
Based on Jack's snippet example:
FROM <base-image>:latest AS first
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_REGION=us-west-2
[do something]
FROM <base-image>:latest
COPY --from=first /dir/file_from_first /dir/file
This is a way to hide all the layers created during the first FROM.
I'm deploying the front-end of my website to amazon s3 via Gitlab pipelines. My previous deployments have worked successfully but the most recent deployments do not. Here's the error:
Completed 12.3 MiB/20.2 MiB (0 Bytes/s) with 1 file(s) remaining
upload failed: dist/vendor.bundle.js.map to s3://<my-s3-bucket-name>/vendor.bundle.js.map Unable to locate credentials
Under my secret variables I have defined four. They are S3 credential variables (AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY) for two different buckets. One pair is for the testing branch and the other is for the production branch.
Not - the production environment variables are protected and the other variables are not.
Here's the deploy script that I run:
#/bin/bash
#upload files
aws s3 cp ./dist s3://my-url-$1 --recursive --acl public-read
So why am I getting this credential location error? Surely it should just pick up the environment variables automatically (the unprotected ones) and deploy them. Do I need to define the variables in the job and refer to them?
(I encountered this issue many times - Adding another answer for people that have the same error - from other reasons).
A quick checklist.
Go to Setting -> CI/CD -> Variables and check:
If both AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY environment variables exist.
If both names are spelled right.
If their state is defined as protected - they can only be ran against protected branches (like master).
If error still occurs:
Make sure the access keys still exist and active on your account.
Delete current environment variables and replace them with new generated access keys and make sure AWS_SECRET_ACCESS_KEY doesn't contain any special characters (can lead to strange errors).
The actual problem was a collision to do with naming the variables. For both branches the variables were called AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. However the problem wasn't just to rename them as the pipeline still didn't pick them up.
I printed the password to the logs to determine which password was being picked up by which branch but found that neither was being taken up. The solution was to have a unique name for each password for each branch (e.g. PRODUCTION_ACCESS_KEY_ID and TESTING_ACCESS_KEY_ID) and in the build script refer to them:
deploy_app_production:
environment:
name: production
url: <url>
before_script:
- echo "Installing ruby & dpl"
- apt-get update && apt-get install -y ruby-full
- gem install dpl
stage: deploy
tags:
- nv1
script:
- echo "Deploying to production"
- sh deploy.sh production $PRODUCTION_ACCESS_KEY_ID $PRODUCTION_SECRET_ACCESS_KEY
only:
- master
And in the deploy.sh I referred to the passed in variables (though I did end up switching to dpl):
dpl --provider=s3 --access-key-id=$2 --secret-access-key=$3 --bucket=<my-bucket-name>-$1 --region=eu-west-1 --acl=public_read --local-dir=./dist --skip_cleanup=true
Have you tried running the Docker image you use in your GitLab pipelines script locally, in interactive mode?
That way, you could verify that the environment variables not being picked up are indeed the problem. (I.e. if you set the same environment variables locally, and it works, then yes, that's the problem.)
I'm suspecting that maybe the credentials are picked up just fine, and maybe they just don't have all the required permissions to perform the requested operation. I know the error message says otherwise, but S3 error messages tend to be quite misleading when it comes to permission problems.
For anyone still struggling after reading the other answers, make sure your branch is protected if you are trying to read protected variables. Gitlab doesn't expose those variables to non-protected branches due to which you might see the error.
https://docs.gitlab.com/ee/ci/variables/index.html#add-a-cicd-variable-to-a-project
To protect your branch go to Settings -> Repository -> Protected Branches and mark your required branch as protected. (This step might change in future)
So, i'm trying not to put sensitive information on the dockerfile. A logical approach is to put the creds in the ebs configuration (the GUI) as a ENV variable. However, docker build doesn't seem to be able to access the ENV variable. Any thoughts?
FROM jupyter/scipy-notebook
USER root
ARG AWS_ACCESS_KEY_ID
RUN echo {$AWS_ACCESS_KEY_ID}
I assume that for every deployment you create a new Dockerrun.aws.json file with the correct docker image tag for that deployment. At deployment stage, you can inject environment values which will then be used in docker run command by EB agent. So your docker containers can now access to these environment variables.
Putting sensitive information (for a Dockerfile to use) can be either for allowing a specific step of the image to run (build time), or for the resulting image to have that secret still there at runtime.
For runtime, if you can use the latest docker 1.13 in a swarm mode configuration, you can manage secrets that way
But the first case (build time) is typically for passing credentials to an http_proxy, and that can be done with --build-arg:
docker build --build-arg HTTP_PROXY=http://...
This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN instruction of the Dockerfile.
Also, these values don’t persist in the intermediate or final images like ENV values do.
In that case, you would not use ENV, but ARG:
ARG <name>[=<default value>]
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag
I am running docker-container on Amazon EC2. Currently I have added AWS Credentials to Dockerfile. Could you please let me know the best way to do this?
A lot has changed in Docker since this question was asked, so here's an attempt at an updated answer.
First, specifically with AWS credentials on containers already running inside of the cloud, using IAM roles as Vor suggests is a really good option. If you can do that, then add one more plus one to his answer and skip the rest of this.
Once you start running things outside of the cloud, or have a different type of secret, there are two key places that I recommend against storing secrets:
Environment variables: when these are defined on a container, every process inside the container has access to them, they are visible via /proc, apps may dump their environment to stdout where it gets stored in the logs, and most importantly, they appear in clear text when you inspect the container.
In the image itself: images often get pushed to registries where many users have pull access, sometimes without any credentials required to pull the image. Even if you delete the secret from one layer, the image can be disassembled with common Linux utilities like tar and the secret can be found from the step where it was first added to the image.
So what other options are there for secrets in Docker containers?
Option A: If you need this secret only during the build of your image, cannot use the secret before the build starts, and do not have access to BuildKit yet, then a multi-stage build is a best of the bad options. You would add the secret to the initial stages of the build, use it there, and then copy the output of that stage without the secret to your release stage, and only push that release stage to the registry servers. This secret is still in the image cache on the build server, so I tend to use this only as a last resort.
Option B: Also during build time, if you can use BuildKit which was released in 18.09, there are currently experimental features to allow the injection of secrets as a volume mount for a single RUN line. That mount does not get written to the image layers, so you can access the secret during build without worrying it will be pushed to a public registry server. The resulting Dockerfile looks like:
# syntax = docker/dockerfile:experimental
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
And you build it with a command in 18.09 or newer like:
DOCKER_BUILDKIT=1 docker build -t your_image --secret id=aws,src=$HOME/.aws/credentials .
Option C: At runtime on a single node, without Swarm Mode or other orchestration, you can mount the credentials as a read only volume. Access to this credential requires the same access that you would have outside of docker to the same credentials file, so it's no better or worse than the scenario without docker. Most importantly, the contents of this file should not be visible when you inspect the container, view the logs, or push the image to a registry server, since the volume is outside of that in every scenario. This does require that you copy your credentials on the docker host, separate from the deploy of the container. (Note, anyone with the ability to run containers on that host can view your credential since access to the docker API is root on the host and root can view the files of any user. If you don't trust users with root on the host, then don't give them docker API access.)
For a docker run, this looks like:
docker run -v $HOME/.aws/credentials:/home/app/.aws/credentials:ro your_image
Or for a compose file, you'd have:
version: '3'
services:
app:
image: your_image
volumes:
- $HOME/.aws/credentials:/home/app/.aws/credentials:ro
Option D: With orchestration tools like Swarm Mode and Kubernetes, we now have secrets support that's better than a volume. With Swarm Mode, the file is encrypted on the manager filesystem (though the decryption key is often there too, allowing the manager to be restarted without an admin entering a decrypt key). More importantly, the secret is only sent to the workers that need the secret (running a container with that secret), it is only stored in memory on the worker, never disk, and it is injected as a file into the container with a tmpfs mount. Users on the host outside of swarm cannot mount that secret directly into their own container, however, with open access to the docker API, they could extract the secret from a running container on the node, so again, limit who has this access to the API. From compose, this secret injection looks like:
version: '3.7'
secrets:
aws_creds:
external: true
services:
app:
image: your_image
secrets:
- source: aws_creds
target: /home/user/.aws/credentials
uid: '1000'
gid: '1000'
mode: 0700
You turn on swarm mode with docker swarm init for a single node, then follow the directions for adding additional nodes. You can create the secret externally with docker secret create aws_creds $HOME/.aws/credentials. And you deploy the compose file with docker stack deploy -c docker-compose.yml stack_name.
I often version my secrets using a script from: https://github.com/sudo-bmitch/docker-config-update
Option E: Other tools exist to manage secrets, and my favorite is Vault because it gives the ability to create time limited secrets that automatically expire. Every application then gets its own set of tokens to request secrets, and those tokens give them the ability to request those time limited secrets for as long as they can reach the vault server. That reduces the risk if a secret is ever taken out of your network since it will either not work or be quick to expire. The functionality specific to AWS for Vault is documented at https://www.vaultproject.io/docs/secrets/aws/index.html
The best way is to use IAM Role and do not deal with credentials at all. (see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html )
Credentials could be retrieved from http://169.254.169.254..... Since this is a private ip address, it could be accessible only from EC2 instances.
All modern AWS client libraries "know" how to fetch, refresh and use credentials from there. So in most cases you don't even need to know about it. Just run ec2 with correct IAM role and you good to go.
As an option you can pass them at the runtime as environment variables ( i.e docker run -e AWS_ACCESS_KEY_ID=xyz -e AWS_SECRET_ACCESS_KEY=aaa myimage)
You can access these environment variables by running printenv at the terminal.
Yet another approach is to create temporary read-only volume in docker-compose.yaml. AWS CLI and SDK (like boto3 or AWS SDK for Java etc.) are looking for default profile in ~/.aws/credentials file.
If you want to use other profiles, you just need also to export AWS_PROFILE variable before running docker-compose command.
export AWS_PROFILE=some_other_profile_name
version: '3'
services:
service-name:
image: docker-image-name:latest
environment:
- AWS_PROFILE=${AWS_PROFILE}
volumes:
- ~/.aws/:/root/.aws:ro
In this example, I used root user on docker. If you are using other user, just change /root/.aws to user home directory.
:ro - stands for read-only docker volume
It is very helpful when you have multiple profiles in ~/.aws/credentials file and you are also using MFA. Also helpful when you want to locally test docker-container before deploying it on ECS on which you have IAM Roles, but locally you don't.
Another approach is to pass the keys from the host machine to the docker container. You may add the following lines to the docker-compose file.
services:
web:
build: .
environment:
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
- AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
The following one-liner works for me even when my credentials are set up by aws-okta or saml2aws:
$ docker run -v$HOME/.aws:/root/.aws:ro \
-e AWS_ACCESS_KEY_ID \
-e AWS_CA_BUNDLE \
-e AWS_CLI_FILE_ENCODING \
-e AWS_CONFIG_FILE \
-e AWS_DEFAULT_OUTPUT \
-e AWS_DEFAULT_REGION \
-e AWS_PAGER \
-e AWS_PROFILE \
-e AWS_ROLE_SESSION_NAME \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-e AWS_SHARED_CREDENTIALS_FILE \
-e AWS_STS_REGIONAL_ENDPOINTS \
amazon/aws-cli s3 ls
Please note that for advanced use cases you might need to allow rw (read-write) permissions, so omit the ro (read-only) limitation when mounting the .aws volume in -v$HOME/.aws:/root/.aws:ro
Volume mounting is noted in this thread but as of docker-compose v3.2 + you can Bind Mount.
For example, if you have a file named .aws_creds in the root of your project:
In your service for the compose file do this for volumes:
volumes:
# normal volume mount, already shown in thread
- ./.aws_creds:/root/.aws/credentials
# way 2, note this requires docker-compose v 3.2+
- type: bind
source: .aws_creds # from local
target: /root/.aws/credentials # to the container location
Using this idea, you can publicly store your docker images on docker-hub because your aws credentials will not physically be in the image...to have them associated, you must have the correct directory structure locally where the container is started (i.e. pulling from Git)
You could create ~/aws_env_creds containing:
touch ~/aws_env_creds
chmod 777 ~/aws_env_creds
vi ~/aws_env_creds
Add these value (replace the key of yours):
AWS_ACCESS_KEY_ID=AK_FAKE_KEY_88RD3PNY
AWS_SECRET_ACCESS_KEY=BividQsWW_FAKE_KEY_MuB5VAAsQNJtSxQQyDY2C
Press "esc" to save the file.
Run and test the container:
my_service:
build: .
image: my_image
env_file:
- ~/aws_env_creds
If someone still face the same issue after following the instructions mentioned in accepted answer then make sure that you are not passing environment variables from two different sources. In my case I was passing environment variables to docker run via a file and as parameters which was causing the variables passed as parameters show no effect.
So the following command did not work for me:
docker run --env-file ./env.list -e AWS_ACCESS_KEY_ID=ABCD -e AWS_SECRET_ACCESS_KEY=PQRST IMAGE_NAME:v1.0.1
Moving the aws credentials into the mentioned env.list file helped.
for php apache docker the following command works
docker run --rm -d -p 80:80 --name my-apache-php-app -v "$PWD":/var/www/html -v ~/.aws:/.aws --env AWS_PROFILE=mfa php:7.2-apache
Based on some of previous answers, I built my own as follows.
My project structure:
├── Dockerfile
├── code
│ └── main.py
├── credentials
├── docker-compose.yml
└── requirements.txt
My docker-compose.yml file:
version: "3"
services:
app:
build:
context: .
volumes:
- ./credentials:/root/.aws/credentials
- ./code:/home/app
My Docker file:
FROM python:3.8-alpine
RUN pip3 --no-cache-dir install --upgrade awscli
RUN mkdir /app
WORKDIR /home/app
CMD python main.py