I'm trying to deploy a docker application onto Elastic Beanstalk from Circle CI.
The deployment section of my circle.yml is
deployment:
hub:
branch: [internal, production]
commands:
- pip install awscli
- docker push company/web:$CIRCLE_SHA1
- sudo bash deploy.sh $CIRCLE_SHA1 $CIRCLE_BRANCH $CIRCLE_BUILD_NUM
and my deploy.sh calls aws cli as follows
aws --version
aws configure set aws_access_key_id $AWSKEY
aws configure set aws_secret_access_key $AWSSECRETKEY
aws configure set default.region us-west-2
aws configure set default.output json
echo "SAVING NEW DOCKERRUNFILE: $DOCKERRUN_FILE"
aws s3 cp $DOCKERRUN_FILE s3://$EB_BUCKET/$DOCKERRUN_FILE
But I get the error
--version: mispelled meta parameter?
sanity-check: "/root/.awssecret": file is missing. (Format: AccessKeyID\nSecretAccessKey\n)
configure: unknown command Usage: aws ACTION [--help]
The script works completely fine locally on mac os using the exact same key and secret.
Both versions (on circle and my mac) of awscli are 1.7.14
I'm Kevin from CircleCI. It looks like the issue here is related to the fact that when you install Python dependencies CircleCI installs them into a virtualenv. This is usually a great thing, as it isolates your python environment from the default system Python and supports our dependency cacheing. The problem here is that you're running your deploy.sh script with sudo, which clobbers the virtualenv environment and runs the default system version (which in this case is actually an older alternative AWS CLI). Dropping the sudo should fix things for you. (You would also be better off running pip install awscli==x.x.x in the "dependencies" phase, as it would be cached then.)
PS: Please contact sayhi#circleci.com for a timely response to questions in general.
Related
I am attempting to upload a rails app that was recently updated from rails 5.2 to 6 to AWS Elastic Beanstalk. We had someone else working on this, but with the pandemic he had to step away - and now our site is kind of in limbo and I have not been able to update it. I have searched many different variations of my problem but no solutions have worked yet.
The app was working on EB with rails 5.2. I have the app running in 6.0 locally. When I eb deploy I get this error:
MacBook-Pro:app $ eb deploy
Starting environment deployment via CodeCommit
--- Waiting for Application Versions to be pre-processed ---
Finished processing application version app-0e294-200420_110159
2020-04-21 00:22:24 INFO Environment update is starting.
2020-04-21 00:23:07 INFO Deploying new version to instance(s).
2020-04-21 00:27:59 ERROR [Instance: i-0e613ac1fe175f3f6] Command failed on instance. Return code: 1 Output: (TRUNCATED)...-- : Writing /var/app/ondeck/public/assets/application-06fe3df6175ba0def3d0e732489f883d0c09de.css.gz
Webpacker requires Node.js ">=10.13.0" and you are using v6.17.1
Please upgrade Node.js https://nodejs.org/en/download/
Exiting!.
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/11_asset_compilation.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-04-21 00:27:59 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-04-21 00:28:00 ERROR Unsuccessful command execution on instance id(s) 'i-0e613ac1fe175f3f6'. Aborting the operation.
2020-04-21 00:28:00 ERROR Failed to deploy application.
ERROR: ServiceError - Failed to deploy application.
It was giving me a bundler error before this, that I was able to fix by adding a file into .ebextensions that installs the correct version of bundler. I figured the solution to this would be similar.
This post was close to my problem:
Deploy rails react app with webpacker gem on AWS elastic beanstalk
So I added this file to my .ebextensions based off the selected answer of that:
01_update_note.config
commands:
01_download_nodejs:
command: curl --silent --location https://rpm.nodesource.com/setup_10.x | sudo bash -
02_install_nodejs:
command: yum -y install nodejs
However, it did not appear to do anything, I still get the same error. I tried a couple variations of the file based off a few other blog posts about the issue, but the error remains. Is anyone able to point me in the right direction or offer any insight into the problem? I apologize for not being very familiar with AWS or EB yet, but I will do my best to answer additional questions.
Maybe it is caused by yarn install later.
I try the following scripts and remove yarn install then set RAILS_SKIP_ASSET_COMPILATION=false and it works for me.
commands:
01_install_yarn:
command: "sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo && curl --silent --location https://rpm.nodesource.com/setup_12.x | sudo bash - && sudo yum install yarn -y"
02_download_nodejs:
command: curl --silent --location https://rpm.nodesource.com/setup_12.x | sudo bash -
03_install_nodejs:
command: yum -y install nodejs
04_install_packages:
command: sudo yum install -y yarn
This is how I did it on Amazon Linux 2:
Create this file in .platform/hooks/prebuild/yarn_config.sh:
#!/usr/bin/env bash
curl --silent --location https://rpm.nodesource.com/setup_16.x | sudo bash -
sudo yum -y install nodejs
sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo
sudo yum -y install yarn
yarn install
Give it the right permission: chmod +x .platform/hooks/prebuild/yarn_config.sh
And the error is gone, while you assets still compile (unlike with accepted answer)
When running a bash script during CodeBuild, I get this error:
./scripts/test.sh: line 95: docker: command not found
However, I've made sure to install docker at the start of the script using:
curl -sSL https://get.docker.com/ | sh
apt-get install -y docker-ce docker-compose
But this results in the following error:
Package docker-ce is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'docker-ce' has no installation candidate
Any ideas on how to get docker working during CodeBuild?
There are a few different options for this in CodeBuild:
You can use CodeBuild provided images, which will already have docker installed on them. To use any one of these images select the privilege mode when creating the CodeBuild project.
You can enable Docker in custom image (images not managed by CodeBuild. e.g.: hosted in your ECR repo or public DockerHub) when configuring CodeBuild project. Select the privileged mode for your project settings. Instructions here: https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html
This question spun out of this one. Now that I have a better understanding of what was going wrong there, and a workable, if imperfect, solution, I'm submitting a more focused follow-up (I'm still something of a novice at StackOverflow - please let me know if this contravenes etiquette, and I should follow-up on the original).
This page suggests that "You use AWS CodeBuild to build, locally test, and package your serverless application". However, when I include a sam build command in my buildspec.yml, I get the following log output, suggesting that sam is not installed on CodeBuild images:
[Container] 2018/12/31 11:41:49 Running command sam build --use-container
sh: 1: sam: not found
[Container] 2018/12/31 11:41:49 Command did not exit successfully sam build --use-container exit status 127
[Container] 2018/12/31 11:41:49 Phase complete: BUILD Success: false
[Container] 2018/12/31 11:41:49 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: sam build --use-container. Reason: exit status 127
Furthermore, if I install SAM with pip install aws-sam-cli, running sam build --use-container in CodeBuild gives an error. sam build itself succeeds, but since it doesn't install test dependencies, I'd still need to use pip install -r tests/requirements-test.txt -t . to be able to run tests in CodeBuild. Moreover, this suggests that --use-container is required for "packages that have natively compiled programs").
This makes me wonder whether I'm trying to do something wrong. What's the recommended way of building SAM services in a CI/CD workflow on AWS?
2019_10_18 - Update (confirming #Spiff answer above):
Apparently Codebuild now work seamlessly with SAM, that's all I needed in buildspec.yml for a lambda using pandas and psycopg2-binary:
version: 0.2
phases:
install:
runtime-versions:
python: 3.7
pre_build:
commands:
- python -m unittest discover tests
build:
commands:
- sam build
post_build:
commands:
- sam package --output-template-file packaged.yaml --s3-bucket my-code-pipeline-bucketz
artifacts:
type: zip
files:
- packaged.yaml
Cheers
Please see below for buildspec.yaml that works for me when using AWS SAM with AWS CodeBuild, with cloudformation.yml
phases:
build:
commands:
- pip install --user aws-sam-cli
- USER_BASE_PATH=$(python -m site --user-base)
- export PATH=$PATH:$USER_BASE_PATH/bin
- sam build -t cloudformation.yml
- aws cloudformation package --template-file .aws-sam/build/template.yaml --s3-bucket <TARGET_S3_BUCKET> --output-template-file cloudformation-packaged.yaml
- aws s3 cp ./cloudformation-packaged.yaml <TARGET_S3_BUCKET>/cloudformation-packaged.yaml
In the result I get a deployment package and packaged cloudformation template in the TARGET_S3_BUCKET.
For each function in the ./src folder, I have a requirements.txt file that includes all the dependencies, but I dont run pip install -r requirements.txt manually.
If you want to run sam build command in CodeBuild, you must install aws-sam-cli first (probably in the install phase of buildspec.yml file) i.e. by running pip install aws-sam-cli command or alike.
--use-container option in the sam build command will cause the command to pull in the Docker image resembling the AWS Lambda execution environment, then run the container from this Docker image to pip install (if your lambda is written in Python) your function dependencies for creating your lambda deployment package. This will ensure that the lambda function will use native compiled libraries that are compatible with the actual runtime environment of AWS Lambda.
Therefore, if you specify --use-container option for sam build command running in CodeBuild, you also need to make sure that a Docker image used by your CodeBuild build project must support Docker runtime.
The most easiest way is to use CodeBuild build environment named aws/codebuild/standard:2.0 Docker image. Enabling Docker runtime in runtime-versions property in the install phases of your buildspec.yml. Also you might need to enable PrevilegedMode of your CodeBuild project in order to connect with Docker daemon from your build environment.
As of October 2019 I had no issues whatsoever deploying a serverless application with codebuild using sam build,
First of all --user is not needed for pip install aws-sam-cli. In fact including --user appears to be the only reason that sam is not in the path.
In addition the --use-container is not needed either as long as no native libraries are built, like psycopg
I am trying to install AWS SAM Cli on my Mac because I am trying to learn the AWS services. But I have installed the AWS cli successful using bundle. But when I tried to install the AWS SAM Cli as well. But it is not working. This is what I have done so far.
Run this command
pip install --user aws-sam-cli
Everything went fine.
Then I opened and edited the ~/.bash_profile. This is the content of the .bash_profile
export PATH=/Applications/MAMP/bin/php/php7.2.7/bin:$PATH
# Find your Python User Base path (where Python --user will install packages/scripts)
$ USER_BASE_PATH=$(python -m site --user-base)
# Update your preferred shell configuration
-- Standard bash --> ~/.bash_profile
-- ZSH --> ~/.zshrc
export PATH=$PATH:$USER_BASE_PATH/bin
Then I closed the terminal and run sam --version.
It is saying command not found. What is wrong with my installation?
The now recommended way to install SAM CLI is to use brew and honestly it's way better, saves you a lot of headaches, like the one you're facing now. See these instructions for details.
I'm not sure if this is Docker, the Elastick Beanstalk, or docker image issue related, but my problem is that I'm running the command eb local run to start the local environment alongside with docker.
Expected behavior
The command runs seamlessly
Actual behavior
ERROR: DockerVersionError - Your local host has the 'docker-py' version 1.10.6 Python package installed on it.
When you run a Multicontainer Docker application locally, the EB CLI requires the 'docker' Python package.
To fix this error:
Be sure that no applications on your local host require 'docker-py', and then run this command:
pip uninstall docker-py
The EB CLI will install 'docker' the next time you run it.
$ eb --version : EB CLI 3.12.2 (Python 2.7.1)
$ docker -v : Docker version 17.12.0-ce, build c97c6d6
If you want to launch multi-container Dockers using eb local run, you need to have uninstalled docker-py, and installed docker
As the error message indicates:
perform pip uninstall docker-py ** if you don't need it **.
run pip install "docker>=2.6.0,<2.7" immediately after
docker and docker-py cannot coexist. These release notes highlight the change in the package name. These release notes allude to the breakage the change in package name caused.
Not to be confused with Docker, the engine/client, docker-py/docker is a Python wrapper around the Docker client which the EBCLI relies on.