I have a git repository with git sub-module, which linked to another git repository.
main-repo
-> file1.txt
-> submodule-repo
-> file2.txt
I created a Google Cloud Build trigger that has permissions to main-repo.
In order to load the submodule-repo repository, I added this command to the build instructions:
steps:
- name: gcr.io/cloud-builders/git
args: ['submodule', 'update', '--init', '--recursive']
...
And it fail in this stage. Why? permissions problem:
Submodule 'XXX' (XXX) registered for path '***' Cloning into
'/workspace/XXX'... ssh: Could not resolve hostname c: Name or service
not known fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository
exists.
The read permission I gave Google is for the main-repo git repository. Since I can give access only for one repository, I can't give another permission for the submodule-repo repsoitory.
How I can use Google Cloud Build to build an git repository with git sub-module?
I did the following and it's working for me:
I followed these instructions to access my private repo from google cloud:
Create an SSH key
Store the private SSH key in Secret Manager
Add the public SSH key to your private repository's deploy keys (if you need to access more than one repo, create a user or use an existing user who has access to these repos and put the deploy key in this account, not in the repo itself > from GitHub account > settings > SSH keys)
Grant permissions to Cloud Build service account to access Secret Manager
Add the public SSH key to known hosts (I stored the public key as a variable in cloud build and you can use GitHub secret to store it)
*Use this command to get the public key and don't copy it from .pub file
ssh-keyscan -t rsa github.com > known_hosts.github
Then these steps in the cloud build file:
- name: 'gcr.io/cloud-builders/git'
secretEnv: ['SSH_KEY']
entrypoint: 'bash'
args:
- -c
- |
echo "$$SSH_KEY" >> /root/.ssh/id_rsa
chmod 400 /root/.ssh/id_rsa
echo ${_SSH_PUBLIC_KEY} >> /root/.ssh/known_hosts
volumes:
- name: 'ssh'
path: /root/.ssh
- name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
args:
- '-c'
- |
git submodule init
git submodule update
volumes:
- name: 'ssh'
path: /root/.ssh
availableSecrets:
secretManager:
- versionName: projects/[GCP_Project]/secrets/[SECRET_NAME]/versions/latest
env: 'SSH_KEY'
I am trying to configure a job on my local circleci (using docker executor, image: google/cloud-sdk:latest), and that job requires a sops gcp kms encrypted file to be decrypted. I have setup a google service account for the gcp kms decrypt service (I can run the script, to be run via the circleci job, successfully locally by decrypting the sops file via the service account, so I know the service account setup is valid). Here is how I am running my job.
1- I base64 encode the google service account json file: base64 path/to/service_aacount_file.json
2- I run circleci job, setting GCLOUD_SERVICE_KEY environment variable on circleci, with the base64 encoded content from the previous step: circleci local execute --env GCLOUD_SERVICE_KEY='<Base64EncodedServiceAccountJsonFileContent>' --job '<MyJob>'
3- Here is my circleci config:
- run:
name: <MyJob>
command: |
apt-get install -y docker
apt-get install -y sudo
cd $(pwd)/path/to/jobcode
echo $GCLOUD_SERVICE_KEY | base64 -d > ${HOME}/<MyGoogleServiceAccountJsonFile.json>
export GOOGLE_APPLICATION_CREDENTIALS="${HOME}/<MyGoogleServiceAccountJsonFile.json>"
gcloud auth activate-service-account --key-file ${HOME}/<MyGoogleServiceAccountJsonFile.json>
echo $GOOGLE_APPLICATION_CREDENTIALS
ls -halt $GOOGLE_APPLICATION_CREDENTIALS
cat $GOOGLE_APPLICATION_CREDENTIALS
sudo ./<RunJob.sh>
4- I get following error when I execute the job:
Failed to get the data key required to decrypt the SOPS file.
Group 0: FAILED
projects/<MyProject>/locations/<MyLocation>/keyRings/<MySopsKeyring>/cryptoKeys/<MyKey>: FAILED
- | Cannot create GCP KMS service: google: could not find
| default credentials. See
| https://developers.google.com/accounts/docs/application-default-credentials
| for more information.
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.
5- Further, from the console output:
a- I can see that the service account was successfully activated: Activated service account credentials for: [<MyServiceAccount>#<MyProject>.iam.gserviceaccount.com]
b- The GOOGLE_APPLICATION_CREDENTIALS environment variable is set to the service account json's path: /path/to/service_account.json
c- The above file has been correctly base64 decoded and contains valid json:
{
"client_x509_cert_url": "<MyUrl>",
"auth_uri": "<MyAuthUri>",
"private_key": "<MyPrivateKey>",
"client_email": "<ClientEmail>",
"private_key_id": "<PrivateKeyId>",
"client_id": "<ClientId>",
"token_uri": "<TokenUri>",
"project_id": "<ProjectId>",
"type": "<ServiceAccount>",
"auth_provider_x509_cert_url": "<AuthProviderCertUrl>"
}
6- Some other things I have tried:
a- Tried setting google project name in environment variables, but still same error.
b- Tried setting GOOGLE_APPLICATION_CREDENTIALS to file's content, instead of file path, but again same result.
c- Tried setting GOOGLE_APPLICATION_CREDENTIALS by providing file path without quotes or single quotes, but still no difference.
d- Tried setting $BASH_ENV by doing echo 'export GOOGLE_APPLICATION_CREDENTIALS=path/to/service_account.json' >> $BASH_ENV, but same error
Please help.
Five options that could work:
Try to run the following command: gcloud auth application-default login
Try this command to set the env var: echo 'export GOOGLE_APPLICATION_CREDENTIALS=/tmp/service-account.json' >> $BASH_ENV
The other thing is that I see that runjob.sh is running under root. It could be that the gcp credentials are not visible under sudo per default. Either run the script without sudo or run the preceding commands with sudo.
As a last resort (those options worked for me, could be different in your scenario): { echo 1; echo 1; echo n; } | gcloud init
gcloud components update This sometimes works when the sdk is outdated.
config set project [PROJECT_NAME]
You can also check active accounts with: gcloud auth list
I'm stuck with this problem since 2 days.
Tried with id_rsa.pub and id_rsa from my production server, still the same error...
SSH_PRIVATE_KEY is a variable I created in the CI/CD Settings on GitLab.
edit : not protected, not masked.
# This file is a template, and might need editing before it works on your project.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/node/tags/
image: node:alpine
stages:
- deploy
deploy:
stage: deploy
before_script:
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apk add --update openssh )'
# Add bash
- apk add --update bash
# Add git
- apk add --update git
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- echo "$SSH_PRIVATE_KEY"
- echo "$SSH_PRIVATE_KEY" | ssh-add -
# For Docker builds disable host key checking. Be aware that by adding that
# you are suspectible to man-in-the-middle attacks.
# WARNING: Use this only with the Docker executor, if you use it with shell
# you will overwrite your user's SSH config.
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
# In order to properly check the server's host key, assuming you created the
# SSH_SERVER_HOSTKEYS variable previously, uncomment the following two lines
# instead.
# - mkdir -p ~/.ssh
# - '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" > ~/.ssh/known_hosts'
script:
- npm i -g pm2
- pm2 deploy ecosystem.config.js production
only:
- master
And when I run the pipeline, I still get this error...
$ echo "$SSH_PRIVATE_KEY" | ssh-add -
Error loading key "(stdin)": invalid format
Could you please help ? I'm helpless, clueless, hopeless loading...
Thanks very much !
SSH_PRIVATE_KEY is a variable I created in the CI/CD Settings on GitLab.
This is documented here
in the Value field paste the content of your private key that you created earlier.
So make sure you have pasted the id_rsa full content, including -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- (with 5 final -)
(And, as MrDuk comments, a final newline)
Stephane Paquet adds in the comments:
cat ~/.ssh/id_rsa | pbcopy
to make sure you copy all the required information.
Just as an FYI for anyone else doing this, I had the same problem but had missed the final dash off the END RSA PRIVATE KEY section. It must have 5 dashes as the dividers, apparently.
Also just as an FYI, my issue was that my SSH key was an OpenSSH format key (ex. -----BEGIN OPENSSH PRIVATE KEY-----) instead of a PEM format key (-----BEGIN RSA PRIVATE KEY-----), if you want instructions on how to convert an OpenSSH key to a PEM key you can find the answer here: Openssh Private Key to RSA Private Key
My solution was to change CI/CD Variable type from Variable to File.
And instead of sourcing from the variable, did the sourcing from the file where SSH_PRIVATE_KEY is pointing
chmod 600 $SSH_PRIVATE_KEY
ssh-add $SSH_PRIVATE_KEY
In the newly created google container builder I am unable to fetch git tags during a build. During the build process the default cloning does not seem to fetch git tags. I added a custom build process which calls git fetch --tags but this results in the error:
Fetching origin
git: 'credential-gcloud.sh' is not a git command. See 'git --help'.
fatal: could not read Username for 'https://source.developers.google.com': No such device or address
# cloudbuild.yaml
#!/bin/bash
openssl aes-256-cbc -k "$ENC_TOKEN" -in gcr_env_vars.sh.enc -out gcr_env_vars.sh -
source gcr_env_vars.sh
env
git config --global url.https://${CI_USER_TOKEN}#github.com/.insteadOf git#github.com:
pushd vendor
git submodule update --init --recursive
popd
docker build -t gcr.io/project-compute/continuous-deploy/project-ui:$COMMIT_SHA -f /workspace/installer/docker/ui/Dockerfile .
docker build -t gcr.io/project-compute/continuous-deploy/project-auth:$COMMIT_SHA -f /workspace/installer/docker/auth/Dockerfile .
This worked for me, as the first build step:
- name: gcr.io/cloud-builders/git
args: [fetch, --depth=100]
To be clear, you want all tags to be available in the Git repo, not just to trigger on tag changes? In the latter, the triggering tag should be available IIUC.
I'll defer to someone on the Container Builder team for a more detailed explanation, but that error tells me that they used gcloud to clone the Google Cloud Source Repository (GCSR), which configures a Git credential helper named as such. They likely did this in another container before invoking yours, or on the host. Since gcloud and/or the gcloud credential helper aren't available in your container, you can't authenticate properly with GCSR.
You can learn a bit more about the credential helper here.
I am running docker-container on Amazon EC2. Currently I have added AWS Credentials to Dockerfile. Could you please let me know the best way to do this?
A lot has changed in Docker since this question was asked, so here's an attempt at an updated answer.
First, specifically with AWS credentials on containers already running inside of the cloud, using IAM roles as Vor suggests is a really good option. If you can do that, then add one more plus one to his answer and skip the rest of this.
Once you start running things outside of the cloud, or have a different type of secret, there are two key places that I recommend against storing secrets:
Environment variables: when these are defined on a container, every process inside the container has access to them, they are visible via /proc, apps may dump their environment to stdout where it gets stored in the logs, and most importantly, they appear in clear text when you inspect the container.
In the image itself: images often get pushed to registries where many users have pull access, sometimes without any credentials required to pull the image. Even if you delete the secret from one layer, the image can be disassembled with common Linux utilities like tar and the secret can be found from the step where it was first added to the image.
So what other options are there for secrets in Docker containers?
Option A: If you need this secret only during the build of your image, cannot use the secret before the build starts, and do not have access to BuildKit yet, then a multi-stage build is a best of the bad options. You would add the secret to the initial stages of the build, use it there, and then copy the output of that stage without the secret to your release stage, and only push that release stage to the registry servers. This secret is still in the image cache on the build server, so I tend to use this only as a last resort.
Option B: Also during build time, if you can use BuildKit which was released in 18.09, there are currently experimental features to allow the injection of secrets as a volume mount for a single RUN line. That mount does not get written to the image layers, so you can access the secret during build without worrying it will be pushed to a public registry server. The resulting Dockerfile looks like:
# syntax = docker/dockerfile:experimental
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
And you build it with a command in 18.09 or newer like:
DOCKER_BUILDKIT=1 docker build -t your_image --secret id=aws,src=$HOME/.aws/credentials .
Option C: At runtime on a single node, without Swarm Mode or other orchestration, you can mount the credentials as a read only volume. Access to this credential requires the same access that you would have outside of docker to the same credentials file, so it's no better or worse than the scenario without docker. Most importantly, the contents of this file should not be visible when you inspect the container, view the logs, or push the image to a registry server, since the volume is outside of that in every scenario. This does require that you copy your credentials on the docker host, separate from the deploy of the container. (Note, anyone with the ability to run containers on that host can view your credential since access to the docker API is root on the host and root can view the files of any user. If you don't trust users with root on the host, then don't give them docker API access.)
For a docker run, this looks like:
docker run -v $HOME/.aws/credentials:/home/app/.aws/credentials:ro your_image
Or for a compose file, you'd have:
version: '3'
services:
app:
image: your_image
volumes:
- $HOME/.aws/credentials:/home/app/.aws/credentials:ro
Option D: With orchestration tools like Swarm Mode and Kubernetes, we now have secrets support that's better than a volume. With Swarm Mode, the file is encrypted on the manager filesystem (though the decryption key is often there too, allowing the manager to be restarted without an admin entering a decrypt key). More importantly, the secret is only sent to the workers that need the secret (running a container with that secret), it is only stored in memory on the worker, never disk, and it is injected as a file into the container with a tmpfs mount. Users on the host outside of swarm cannot mount that secret directly into their own container, however, with open access to the docker API, they could extract the secret from a running container on the node, so again, limit who has this access to the API. From compose, this secret injection looks like:
version: '3.7'
secrets:
aws_creds:
external: true
services:
app:
image: your_image
secrets:
- source: aws_creds
target: /home/user/.aws/credentials
uid: '1000'
gid: '1000'
mode: 0700
You turn on swarm mode with docker swarm init for a single node, then follow the directions for adding additional nodes. You can create the secret externally with docker secret create aws_creds $HOME/.aws/credentials. And you deploy the compose file with docker stack deploy -c docker-compose.yml stack_name.
I often version my secrets using a script from: https://github.com/sudo-bmitch/docker-config-update
Option E: Other tools exist to manage secrets, and my favorite is Vault because it gives the ability to create time limited secrets that automatically expire. Every application then gets its own set of tokens to request secrets, and those tokens give them the ability to request those time limited secrets for as long as they can reach the vault server. That reduces the risk if a secret is ever taken out of your network since it will either not work or be quick to expire. The functionality specific to AWS for Vault is documented at https://www.vaultproject.io/docs/secrets/aws/index.html
The best way is to use IAM Role and do not deal with credentials at all. (see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html )
Credentials could be retrieved from http://169.254.169.254..... Since this is a private ip address, it could be accessible only from EC2 instances.
All modern AWS client libraries "know" how to fetch, refresh and use credentials from there. So in most cases you don't even need to know about it. Just run ec2 with correct IAM role and you good to go.
As an option you can pass them at the runtime as environment variables ( i.e docker run -e AWS_ACCESS_KEY_ID=xyz -e AWS_SECRET_ACCESS_KEY=aaa myimage)
You can access these environment variables by running printenv at the terminal.
Yet another approach is to create temporary read-only volume in docker-compose.yaml. AWS CLI and SDK (like boto3 or AWS SDK for Java etc.) are looking for default profile in ~/.aws/credentials file.
If you want to use other profiles, you just need also to export AWS_PROFILE variable before running docker-compose command.
export AWS_PROFILE=some_other_profile_name
version: '3'
services:
service-name:
image: docker-image-name:latest
environment:
- AWS_PROFILE=${AWS_PROFILE}
volumes:
- ~/.aws/:/root/.aws:ro
In this example, I used root user on docker. If you are using other user, just change /root/.aws to user home directory.
:ro - stands for read-only docker volume
It is very helpful when you have multiple profiles in ~/.aws/credentials file and you are also using MFA. Also helpful when you want to locally test docker-container before deploying it on ECS on which you have IAM Roles, but locally you don't.
Another approach is to pass the keys from the host machine to the docker container. You may add the following lines to the docker-compose file.
services:
web:
build: .
environment:
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
- AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
The following one-liner works for me even when my credentials are set up by aws-okta or saml2aws:
$ docker run -v$HOME/.aws:/root/.aws:ro \
-e AWS_ACCESS_KEY_ID \
-e AWS_CA_BUNDLE \
-e AWS_CLI_FILE_ENCODING \
-e AWS_CONFIG_FILE \
-e AWS_DEFAULT_OUTPUT \
-e AWS_DEFAULT_REGION \
-e AWS_PAGER \
-e AWS_PROFILE \
-e AWS_ROLE_SESSION_NAME \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-e AWS_SHARED_CREDENTIALS_FILE \
-e AWS_STS_REGIONAL_ENDPOINTS \
amazon/aws-cli s3 ls
Please note that for advanced use cases you might need to allow rw (read-write) permissions, so omit the ro (read-only) limitation when mounting the .aws volume in -v$HOME/.aws:/root/.aws:ro
Volume mounting is noted in this thread but as of docker-compose v3.2 + you can Bind Mount.
For example, if you have a file named .aws_creds in the root of your project:
In your service for the compose file do this for volumes:
volumes:
# normal volume mount, already shown in thread
- ./.aws_creds:/root/.aws/credentials
# way 2, note this requires docker-compose v 3.2+
- type: bind
source: .aws_creds # from local
target: /root/.aws/credentials # to the container location
Using this idea, you can publicly store your docker images on docker-hub because your aws credentials will not physically be in the image...to have them associated, you must have the correct directory structure locally where the container is started (i.e. pulling from Git)
You could create ~/aws_env_creds containing:
touch ~/aws_env_creds
chmod 777 ~/aws_env_creds
vi ~/aws_env_creds
Add these value (replace the key of yours):
AWS_ACCESS_KEY_ID=AK_FAKE_KEY_88RD3PNY
AWS_SECRET_ACCESS_KEY=BividQsWW_FAKE_KEY_MuB5VAAsQNJtSxQQyDY2C
Press "esc" to save the file.
Run and test the container:
my_service:
build: .
image: my_image
env_file:
- ~/aws_env_creds
If someone still face the same issue after following the instructions mentioned in accepted answer then make sure that you are not passing environment variables from two different sources. In my case I was passing environment variables to docker run via a file and as parameters which was causing the variables passed as parameters show no effect.
So the following command did not work for me:
docker run --env-file ./env.list -e AWS_ACCESS_KEY_ID=ABCD -e AWS_SECRET_ACCESS_KEY=PQRST IMAGE_NAME:v1.0.1
Moving the aws credentials into the mentioned env.list file helped.
for php apache docker the following command works
docker run --rm -d -p 80:80 --name my-apache-php-app -v "$PWD":/var/www/html -v ~/.aws:/.aws --env AWS_PROFILE=mfa php:7.2-apache
Based on some of previous answers, I built my own as follows.
My project structure:
├── Dockerfile
├── code
│ └── main.py
├── credentials
├── docker-compose.yml
└── requirements.txt
My docker-compose.yml file:
version: "3"
services:
app:
build:
context: .
volumes:
- ./credentials:/root/.aws/credentials
- ./code:/home/app
My Docker file:
FROM python:3.8-alpine
RUN pip3 --no-cache-dir install --upgrade awscli
RUN mkdir /app
WORKDIR /home/app
CMD python main.py