With Snap-CI going away I've been trying to get our builds working on AWS CodeBuild. I have my buildspec.yml built out, but changing directories doesn't seem to work.
version: 0.1
phases:
install:
commands:
- apt-get update -y
- apt-get install -y node
- apt-get install -y npm
build:
commands:
- cd MyDir //Expect to be in MyDir now
- echo `pwd` //Shows /tmp/blablabla/ instead of /tmp/blablabla/MyDir
- npm install //Fails because I'm not in the right directory
- bower install
- npm run ci
post_build:
commands:
- echo Build completed on `date`
artifacts:
files:
- MyDir/MyFile.war
discard-paths: yes
It seems like this should be fairly simple, but so far I haven't had any luck getting this to work.
If you change the buildspec.yml version to 0.2 then the shell keeps its settings.
In version: 0.1 you get a clean shell for each command.
Each command in CodeBuild runs in a separate shell against the root of your source (access root of your source from CODEBUILD_SRC_DIR environment variable).
Your possible options are
Short circuit the commands to run under the same shell: Works when you have relatively simple buildspec (like yours).
commands:
- cd MyDir && npm install && bower install
- cd MyDir && npm run ci
Move your commands from buildspec to a script and have more control (useful for more complicated build logic).
commands:
- ./mybuildscipt.sh
Let me know if any of these work for you.
-- EDIT --
CodeBuild has since launched buildspec v0.2 where this work around is no longer required.
Related
I am new to create a pipeline in aws. I want to create a ci-cd pipeline for my nuxt project.
I create a yml file in which I want to install nvm and then install node version 12.18.3
The problem is I am getting the nvm not found error.
Can you please check and let me know if there is any error in my yml file:
version: 0.2
phases:
install:
commands:
- echo Installing nvm...
- curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
- export NVM_DIR="$HOME/.nvm"
- '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"'
- '[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"'
pre_build:
commands:
#install dependencies
- echo Installing node...
- nvm install 12.18.3
- echo Installing npm...
- npm install
build:
commands:
#build
- echo building...
- npm run generate
artifacts:
files:
- '**/*'
base-directory: dist
cache:
paths:
- node_modules/**/*
Thank you.
You don't actually need to use nvm to install specific node.js version on AWS CodeBuild.
You can use runtime-versions option which would install some version, but you don't have much control over this.
phases:
install:
runtime-versions:
nodejs: 12.x
But AWS standard 5 image comes with n preinstalled (haven't checked 4 but it should be there as well), so you can use it like:
phases:
install:
commands:
- n 12.18.3
and it would install that version same as nvm.
Working Solution
For some reason, It doesn't recognize nvm in the next line. I did not get a chance to investigate it further. The following configuration works. The idea is to set the nvm configurations and install node in the same line.
version: 0.2
phases:
install:
commands:
- echo Installing nvm...
- curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
- export NVM_DIR="$HOME/.nvm"
- '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"'
- '[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"'
pre_build:
commands:
#install dependencies
- . "$NVM_DIR/nvm.sh" && nvm install 12.18.3 && echo "node installed by arun"
- echo Installing node...
#- nvm install 12.18.3
- echo Installing npm...
- npm install
build:
commands:
#build
- echo building...
- npm run generate
cache:
paths:
- node_modules/**/*
Troubleshoot in local environment.
In order to troubleshoot the buildspec.yaml, you can run the build locally. here is how to run the buildspec locally.
I am trying to install SonarQube using AWS CodeBuild. I am using a Nodejs: 10 as the run time environment. I am getting the below error when I run the below script as the build spec? As I understood, the issue is the NodeJS env does not contain Maven inbuilt. If that is the case, How can I proceed with Maven with in the Node JS Env. Thanks in advance.
[Container] 2020/07/26 18:16:43 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: mvn test. Reason: exit status 1
Issue occurs when it starts to execute -mvn test
buildspec.yml
version: 0.2
env:
secrets-manager:
LOGIN: SonarCloud:sonartoken
HOST: SonarCloud:HOST
Organization: SonarCloud:Organization
Project: prod/sonar:Project
phases:
install:
runtime-versions:
nodejs: 10
pre_build:
commands:
- npm install
- apt-get update
- apt-get install -y jq
- wget http://www-eu.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
- tar xzf apache-maven-3.5.4-bin.tar.gz
- ln -s apache-maven-3.5.4 maven
- wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-3.3.0.1492-linux.zip
- unzip ./sonar-scanner-cli-3.3.0.1492-linux.zip
- export PATH=$PATH:/sonar-scanner-3.3.0.1492-linux/bin/
build:
commands:
- mvn test
- mvn sonar:sonar -Dsonar.login=$LOGIN -Dsonar.host.url=$HOST -Dsonar.projectKey=$Project -Dsonar.organization=$Organization
- sleep 5
- curl https://sonarcloud.io/api/qualitygates/project_status?projectKey=$Project >result.json
- cat result.json
- if [ $(jq -r '.projectStatus.status' result.json) = ERROR ] ; then $CODEBUILD_BUILD_SUCCEEDING -eq 0 ;fi
- echo Build started on `date`
- echo Compiling the Node.js code
post_build:
commands:
- echo Build completed on `date`
artifacts:
files:
- server.js
- package.json
- controller/*
Maven is available in java : openjdk8.
You need to add the same to your yml.
Sample format :
phases:
install:
runtime-versions:
java: openjdk8
build:
commands:
- mvn test
Add either java: corretto11 or java: openjdk8 or java: openjdk11 under runtime-versions: and maven will start executing.
Probably you might need to use your project specific settings.xml for maven build, which you can easily provide in a S3 bucket and then refer it under build commands of buildspec.yml
I'm using corretto rather than openjdk in my aws configuration, as aws provide LTS for corretto. Reference - https://aws.amazon.com/corretto/faqs/
Trying to use CodeBuild for the first time, pulling data from CodeCommit. But I'm having issues with my buildspec. This is the code I have on it so far:
version: 0.2
phases:
INSTALL:
runtime-versions:
nodejs: 10
commands:
- npm install
PRE_BUILD:
commands:
- npm install --quiet --global expo-cli
- >
if [ -f yarn.lock ]; then
yarn
elif [ -f package-lock.json ] || [ -f npm-shrinkwrap.json ]; then
npm ci
else
npm install
fi
BUILD:
commands:
- expo build:web
artifacts:
baseDirectory: web-build
files:
- '**/*'
name:
myname-$(date +%Y-%m-%d)
cache:
paths:
- node_modules/**/*
- $(npm root --global)/**/*
I have already added the runtime for nodejs 10, it had stopped to trigger this error, but now it kicked again. Does anyone know how to properly tweak it for React-Native web projects?
I believe the phase names are case sensitive, so change them to install, pre_build and build.
I'm using zappa to deploy on aws. And I wanted to implement CI/CD on AWS.
So, I created a pipeline and successfully did Aws COMMIT and AWS BUILD.
I'm unable to deploy the same using AWS CODE DEPLOY.
The Buildspec.yaml looks like this:
version: 0.2
phases:
install:
commands:
- echo Setting up virtualenv
- python -m venv venv
- source venv/bin/activate
- echo Installing requirements from file
- pip install -r requirements.txt
build:
commands:
- echo Build started on `date`
- echo Building and running tests
- python tests.py
- flask db upgrade
post_build:
commands:
- echo Build completed on `date`
- echo Starting deployment
- zappa update dev
- echo Deployment completed
How should I execute zappa deploy or zappa update on AWS?
I'm not sure how to add create appspec.yaml file.
Please HELP! Stuck!!
Here's a buildspec.yml file that I use. You could adjust this to suit your needs (for example, including the DB upgrade command).
version: 0.2
phases:
install:
commands:
- mkdir /tmp/src/
- mv $CODEBUILD_SRC_DIR/* /tmp/src/
- cd /tmp/src/
- python3 -m venv docker_env && source docker_env/bin/activate && pip install --upgrade pip==9.0.3 && pip install -r requirements.txt && zappa update production && deactivate && rm -rf docker_env
post_build:
commands:
- cd $CODEBUILD_SRC_DIR
- rm -rf /tmp/src/
- echo Build completed on `date`
Note that this is using the Docker image danielwhatmuff/zappa:python3.6 in CodeBuild. I use this image as it's based on AWS Lambda and has been tuned for Zappa.
Zappa update to Code Deploy:
Your Buildspec.yaml looks fair good but there is one important point to consider.
Postbuild will always run regardless of success/failure. Debug information can be pulled from a failed build.
Either check the reason for failure from build log, or modify your yml to look like below (caution: this is only draft change, test before using in systems):
version: 0.2
phases:
install:
commands:
- yum -y groupinstall development
- yum -y install zlib-devel
- yum -y install openssl-devel
- wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz
- tar xJf Python-3.6.0.tar.xz
- cd Python-3.6.0
- ./configure
- make
- make install
- ln -s /usr/local/bin/python3.6 /usr/bin/python3
- curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
- python3 get-pip.py
- pip3 install virtualenv
- virtualenv -p /usr/bin/python3 venv
- source venv/bin/activate
- pip3 install -r requirements.txt
build:
commands:
- echo Build started on `date`
- echo Building and running tests
- python3 tests.py
- flask db upgrade
post_build:
commands:
- if [ $CODEBUILD_BUILD_SUCCEEDING = 1 ]; then echo Build completed on `date`; echo Starting deployment; zappa update dev; else echo Build failed ignoring deployment; fi
- echo Deployment completed
Hope it answers.
Zappa update to AWS
Below are the steps to do Zappa update on AWS
Configure AWS with IAM user
Configure AWS cli in the local host using command
a. pip install awscli
b. aws configure
Call "Zappa init", it will generate zappa_settings.json based on details provided
Zappa deploy <name provided for environment in step3>
Now your application will be deployed to AWS. Whenever you need to update call
Zappa update <name provided for environment in step3>
Is there a way to drop root user on AWS CodeBuild?
We are building a Yocto project that fails on CodeBuild if we're root (Bitbake sanity check).
Our desperate approach doesn't work either:
...
build:
commands:
- chmod -R 777 $(pwd)/ && chown -R builder $(pwd)/ && su -c "$(pwd)/make.sh" -s /bin/bash builder
...
Fails with:
bash: /codebuild/output/src624711770/src/.../make.sh: Permission denied
Any idea how we could run this a non-root?
I am succeeded in using non-root user in AWS CodeBuild.
It takes much more than knowing some CodeBuild options to come up with a practical solution.
Everyone should spot run-as option quite easily.
The next question is "which user?"; you cannot just put any word as a username.
In order to find out which users are available, the next clue is at Docker images provided by CodeBuild section. There, you'll find a link to each image definition.
For me, the link leads me to this page on GitHub
After inspecting the source code of Dockerfile, we'll know that there is a user called codebuild-user available. And we can use this codebuild-user for our run-as in the buildspec.
Then we'll face with a whole lot of other problems because the standard image only installs runtime of each language for root only.
This is as far as generic explanations can go.
For me, I wanted to use the Ruby runtime, so my only concern is the Ruby runtime.
If you use CodeBuild for something else, you are on your own now.
In order to utilize Ruby runtime as codebuild-user, we have to expose them from the root user. To do that, I change the required permissions and owner of .rbenv used by the CodeBuild image with the following command.
chmod +x ~
chown -R codebuild-user:codebuild-user ~/.rbenv
The bundler (Ruby's dependency management tool) still wants to access the home directory, which is not writable. We have to set up an environment variable to make it use other writable location as the home directory.
The environment variable is BUNDLE_USER_HOME.
Put everything together; my buildspec looks like:
version: 0.2
env:
variables:
RAILS_ENV: test
BUNDLE_USER_HOME: /tmp/bundle-user
BUNDLE_SILENCE_ROOT_WARNING: true
run-as: codebuild-user
phases:
install:
runtime-versions:
ruby: 2.x
run-as: root
commands:
- chmod +x ~
- chown -R codebuild-user:codebuild-user ~/.rbenv
- bundle config set path 'vendor/bundle'
- bundle install
build:
commands:
- bundle exec rails spec
cache:
paths:
- vendor/bundle/**/*
My points are:
It is, indeed, possible.
Show how I did it for my use case.
Thank you for this feature request. Currently you cannot run as a non-root user in CodeBuild, I have passed it to the team for further review. Your feedback is very much appreciated.
To run CodeBuild as non root you need to specify a Linux username using the run-as tag in your buildspec.yaml as shown in the docs
version: 0.2
run-as: Linux-user-name
env:
variables:
key: "value"
key: "value"
parameter-store:
key: "value"
key: "value"
phases:
install:
run-as: Linux-user-name
runtime-versions:
runtime: version
What we ended up doing was the following:
Create a Dockerfile which contains all the stuff to build a Yocto / Bitbake project in which we ADD the required sources and create an user builder which we use to build our project.
FROM ubuntu:16.04
RUN apt-get update && apt-get -y upgrade
# Required Packages for the Host Development System
RUN apt-get install -y gawk wget git-core diffstat unzip texinfo gcc-multilib \
build-essential chrpath socat cpio python python3 python3-pip python3-pexpect \
xz-utils debianutils iputils-ping vim
# Additional host packages required by poky/scripts/wic
RUN apt-get install -y curl dosfstools mtools parted syslinux tree
# Create a non-root user that will perform the actual build
RUN id builder 2>/dev/null || useradd --uid 30000 --create-home builder
RUN apt-get install -y sudo
RUN echo "builder ALL=(ALL) NOPASSWD: ALL" | tee -a /etc/sudoers
# Fix error "Please use a locale setting which supports utf-8."
# See https://wiki.yoctoproject.org/wiki/TipsAndTricks/ResolvingLocaleIssues
RUN apt-get install -y locales
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
echo 'LANG="en_US.UTF-8"'>/etc/default/locale && \
dpkg-reconfigure --frontend=noninteractive locales && \
update-locale LANG=en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANG US.UTF-8
ENV LANGUAGE en_US.UTF-8
WORKDIR /home/builder/
ADD ./ ./
USER builder
ENTRYPOINT ["/bin/bash", "-c", "./make.sh"]
We build this docker during the Codebuild pre_build step and run the actual build in the ENTRYPOINT (in make.sh) when we run the image. After the container has been excited, we copy the artifacts to the Codebuild host and put them on S3:
version: 0.2
phases:
pre_build:
commands:
- mkdir ./images
- docker build -t bob .
build:
commands:
- docker run bob:latest
post_build:
commands:
# copy the last excited container's images into host as build artifact
- docker cp $(docker container ls -a | head -2 | tail -1 | awk '{ print $1 }'):/home/builder/yocto-env/build/tmp/deploy/images ./images
- tar -cvzf artifacts.tar.gz ./images/*
artifacts:
files:
- artifacts.tar.gz
The only drawback this approach has, is the fact that we can't (easily) use Codebuild's caching functionality. But the build is sufficiently fast for us, since we do local builds during the day and basically one rebuild from scratch at night, which takes about 90 minutes (on the most powerful Codebuild instance).
Sigh, so I came across this question and I am disappointed that there is no good or simple answer to this problem. There are many, many processes that strongly discourage running as root like composer and others that will flat-out refuse like wp-cli. If you are using the Ubuntu "standard image" provided by AWS, then there appears to be an existing user in the /etc/passwd file, dockremap:x:1000:1000::/home/dockremap:/bin/sh. I think this user is for userns-remap in docker and I am not sure about it's availability. The other option that astonishingly hasn't been mentioned is running useradd -N -G users develop to create a new user in the container. It is far simpler than spinning up a custom container for something so trivial.