I am new to AWS codeBuild and code pipeline, what I am wondering is if there is a way I can fail a pipeline or codeBuild project if the coverage of my test cases is below a certain threshold, i.e 80%?
I have an angular project configured with Karma & Jasmine.
Upon research I have found that I can use this command aws codebuild stop-build --id ${CODEBUILD_BUILD_ID}, not sure if this will help me in any way?
Buildspec.yml
version: 0.2
phases:
install:
commands:
# Install the Angular CLI
- npm install -g #angular/cli
# Install puppeteer as a dev dependency
- npm i -D puppeteer
- npm i -D #angular-devkit/build-angular
# Print out missing libs
- echo "Missing Libs" || ldd ./node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome | grep not
# Upgrade apt
- apt-get upgrade
# Update libs
- apt-get update
# Install apt-transport-https
- apt-get install -y apt-transport-https
# Use apt to install the Chrome dependencies
- apt-get install -y libxcursor1
- apt-get install -y libgtk-3-dev
- apt-get install -y libxss1
- apt-get install -y libasound2
- apt-get install -y libnspr4
- apt-get install -y libnss3
- apt-get install -y libx11-xcb1
# Print out missing libs
- echo "Missing Libs" || ldd ./node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome | grep not
# Install project dependencies
- npm install
# - usermod -g 0 -o codebuild-user
build:
# run-as: codebuild-user
commands:
# Run headless Chrome tests
- ng test --browsers=ChromeHeadlessNoSandbox --watch=false --code-coverage
- printenv
I wish to condition the ng tests output here to fail the pipeline.
- aws codebuild stop-build --id ${CODEBUILD_BUILD_ID}
artifacts:
files: imagedefinitions.json
Related
It is for a while that I am working on a project and I want to build it on travis ci. My project has a dependency to boost-geometry, so I tried the following travis.yml
language: cpp
dist: Xenial
sudo: true
matrix:
include:
- os: linux
env:
- PYTHON=3.6
- CXX=g++-5
- CC=gcc-5
- CXXFLAGS = "$CXXFLAGS -std=c++14"
# - os: linux
# env: CONDA=3.6 CXX=g++ CC=gcc CXXFLAGS = "$CXXFLAGS -std=c++14"
# Boost
addons:
apt:
update: true
packages:
- libboost-all-dev
- g++-5
before_install:
- sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
- sudo apt-get update -qq
- sudo apt-get install gcc-5 -y
- sudo apt-get install g++-5 -y
- sudo apt-get install libboost-all-dev -y
- sudo apt-get install python-pip python-dev build-essential
- sudo apt-get install python3-setuptools
- sudo apt-get install python3-pip
- sudo pip install --user --upgrade pip virtualenv
- sudo pip install --user setuptools
- sudo virtualenv -p python3 venv
- source venv/bin/activate
- sudo pip install pybind11
- sudo python3 -m pip install pybind11
- sudo python3 -m pip install --upgrade pip
- cd src
- ls
- cd ../
install:
- ls $TRAVIS_BUILD_DIR
- sudo python3 setup.py install
- sudo python3 setup.py sdist
- sudo pip install --verbose dist/*.tar.gz
script:
- python tests/test.py
In this file it gives me a error. which indicates that this file is not installed as part of boost geometry library.
could not find boost/geometry/core/assert.hpp
So I decided to build boost from source. I used the following file
language: cpp
dist: Xenial
sudo: true
matrix:
include:
- os: linux
env:
- PYTHON=3.6
- CXX=g++-5
- CC=gcc-5
- CXXFLAGS = "$CXXFLAGS -std=c++14"
# - os: linux
# env: CONDA=3.6 CXX=g++ CC=gcc CXXFLAGS = "$CXXFLAGS -std=c++14"
# Boost
addons:
apt:
update: true
sources:
-boost-latest
packages:
- libboost-all-dev
- g++-5
before_install:
- sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
- sudo apt-get update -qq
- sudo apt-get install gcc-5 -y
- sudo apt-get install g++-5 -y
- sudo apt-get install libboost-all-dev -y
- sudo apt-get install python-pip python-dev build-essential
- sudo apt-get install python3-setuptools
- sudo apt-get install python3-pip
- wget https://sourceforge.net/projects/boost/files/boost/1.70.0/boost_1_70_0.tar.gz
- tar -xzvf boost_1_70_0.tar.gz
- cd boost_1_70_0
- sudo ./bootstrap.sh --prefix=/usr/local --with-libraries=all
- sudo ./b2 install
- sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/boost.conf'
- sudo ldconfig
- cd ../../
- sudo pip install --user --upgrade pip virtualenv
- sudo pip install --user setuptools
- sudo virtualenv -p python3 venv
- source venv/bin/activate
- sudo pip install pybind11
- sudo python3 -m pip install pybind11
- sudo python3 -m pip install --upgrade pip
- cd src
- ls
- cd ../
# Inform user that Boost 1.55 was successfully installed
# echo "Boost 1.55 was successfully installed."
install:
#- |
# if [ -n "$PYTHON" ]; then
# sudo apt-get install -qq g++-6
# sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-6 90
- ls $TRAVIS_BUILD_DIR
- sudo python3 setup.py install
- sudo python3 setup.py sdist
- sudo pip install --verbose dist/*.tar.gz
# elif [ -n "$CONDA" ]; then
# conda build conda.recipe
# conda install --use-local pydggrid
# fi
script:
- python tests/test.py
It starts to run but as boost writes logs travis raises the following error
The job exceeded the maximum log length, and has been terminated.
Well, what can I do to fix this issue? I looked for a flag to avoid boost generating that amount of logs but could not find any. I also tried to find a why to use apt-get to install boost-geometry but did not get any chance to find proper package. Can some one help me with this issue?
thanks
well I ended up with this method
- wget https://sourceforge.net/projects/boost/files/boost/1.70.0/boost_1_70_0.tar.gz -P $HOME
- cd $HOME
- tar -xzvf boost_1_70_0.tar.gz >/dev/null
and as boost is header only library I did not compile it, I just added its directory in include section of my make script
I'm trying to deploy a geodjango application on AWS Elastic Beanstalk. The configuration is 64bit Amazon Linux 2017.09 v2.6.6 running Python 3.6. I am getting this error when trying to deploy:
Requires: libpoppler.so.5()(64bit) Error: Package: gdal-java-1.9.2-8.rhel6.x86_64 (pgdg93) Requires: libpoppler.so.5()(64bit)
How do I install the required package? I read through Setting up Django with GeoDjango Support in AWS Beanstalk or EC2 Instance but I am still getting problems. My ebextensions currently looks like:
commands:
01_yum_update:
command: sudo yum -y update
02_epel_repo:
command: sudo yum-config-manager -y --enable epel
03_install_gdal_packages:
command: sudo yum -y install gdal gdal-devel
packages:
yum:
git: []
postgresql95-devel: []
gettext: []
libjpeg-turbo-devel: []
libffi-devel: []
I'm going to answer my own question for the sake my future projects and anyone else trying to get started with geodjango. Updating this answer as of July 2020
Create an ebextensions file to install GDAL on the EC2 instance at deployment:
01_gdal.config
commands:
01_install_gdal:
test: "[ ! -d /usr/local/gdal ]"
command: "/tmp/gdal_install.sh"
files:
"/tmp/gdal_install.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel
# Geos
cd /
sudo mkdir -p /usr/local/geos
cd usr/local/geos/geos-3.7.2
sudo wget geos-3.7.2.tar.bz2 http://download.osgeo.org/geos/geos-3.7.2.tar.bz2
sudo tar -xvf geos-3.7.2.tar.bz2
cd geos-3.7.2
sudo ./configure
sudo make
sudo make install
sudo ldconfig
# Proj4
cd /
sudo mkdir -p /usr/local/proj
cd usr/local/proj
sudo wget -O proj-5.2.0.tar.gz http://download.osgeo.org/proj/proj-5.2.0.tar.gz
sudo wget -O proj-datumgrid-1.8.tar.gz http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz
sudo tar xvf proj-5.2.0.tar.gz
sudo tar xvf proj-datumgrid-1.8.tar.gz
cd proj-5.2.0
sudo ./configure
sudo make
sudo make install
sudo ldconfig
# GDAL
cd /
sudo mkdir -p /usr/local/gdal
cd usr/local/gdal
sudo wget -O gdal-2.4.4.tar.gz http://download.osgeo.org/gdal/2.4.4/gdal-2.4.4.tar.gz
sudo tar xvf gdal-2.4.4.tar.gz
cd gdal-2.4.4
sudo ./configure
sudo make
sudo make install
sudo ldconfig
As shown, the script checks whether gdal already exists using the test function. It then downloads the Geos, Proj, and GDAL libraries and installs them in the usr/local directory. At the time of writing this, geodjango (Django 3.0) supports up to Geos 3.7, Proj 5.2 (which also requires projdatum. Current releases do not require it), and GDAL 2.4 Warning: this installation process can take a long time. Also I am not a Linux professional so some of those commands may be redundant, but it works.
Lastly I add the following two environment variables to my Elastic Beanstalk configuration:
LD_LIBRARY_PATH: /usr/local/lib:$LD_LIBRARY_PATH
PROJ_LIB: usr/local/proj
If you still have troubles I recommend checking the logs and ssh-ing in the EC2 instance to check that installation took place. Original credit to this post
There's a GitLab.com update rolling out today, and I'm seeing issues connecting to a particular AWS region with Ansible: us-gov-west-1.
This is odd, since in my CI job I'm able to use the AWS CLI just fine:
CI build step:
$ aws ec2 describe-instances
Output (truncated):
{
"Reservations": [
{
"Instances": [
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": "ec2-...
The very build step is as follows, notice that it fails to connect to the region:
CI build step:
$ ansible-playbook -vvv -i inventory/ec2.py -e ansible_ssh_private_key_file=aws-keypairs/gitlab_keypair.pem playbooks/deploy.yml
Output (truncated)
Using /builds/me/my-project/ansible.cfg as config file
ERROR! Attempted to execute "inventory/ec2.py" as inventory script: Inventory script (inventory/ec2.py) had an execution error: region name: us-gov-west-1 likely not supported, or AWS is down. connection to region failed.
ERROR: Job failed: exit code 1
Is anyone else seeing this?
It was working this morning. Any idea why this might be failing now?
I wrote a small Python script to dive deeper into boto. When I googled how to list the regions,I was reminded of the differences in boto 2 vs boto 3. Then, I reviewed the mechanism I was using to install boto. It looks like the boto installation was the problem.
Here’s the buggy version of my .gitlab-ci.yml file:
image: ansible/ansible:ubuntu1604
test_aws:
stage: deploy
before_script:
- apt-get update
- apt-get -y install python
- apt-get -y install python-boto python-pip
- pip install awscli
script:
- 'aws ec2 describe-instances'
deploy_app:
stage: deploy
before_script:
- apt-get update
- apt-get -y install python
- apt-get -y install python-boto python-pip
- pip install awscli
script:
- 'chmod 400 aws-keypairs/gitlab_keypair.pem'
- 'ansible-playbook -vvv -i inventory/ec2.py -e ansible_ssh_private_key_file=aws-keypairs/gitlab_keypair.pem playbooks/deploy.yml'
And here’s the fixed version:
image: ansible/ansible:ubuntu1604
all_in_one:
stage: deploy
before_script:
- rm -rf /var/lib/apt/lists/*
- apt-get update
- apt-get -y install python python-pip
- pip install boto==2.48.0
- pip install awscli
- pip install ansible==2.2.2.0
script:
- 'chmod 400 aws-keypairs/gitlab_keypair.pem'
- 'aws ec2 describe-instances'
- 'python ./boto_debug.py'
- 'ansible-playbook -vvv -i inventory/ec2.py -e ansible_ssh_private_key_file=aws-keypairs/gitlab_keypair.pem playbooks/deploy.yml'
Notice that I switched from using apt-get install to using pip install. Hopefully others will come across this post in the future and avoid installing boto with apt-get -y install python-boto!
I am currently using Tika to extract text from files uploaded to my Rails app running on AWS Elastic Beanstalk (64bit Amazon Linux 2016.03 v2.1.2 running Ruby 2.2). I'd like to index scanned images as well, so I need to install Tesseract.
I was able to get it to work by installing it from source like so, but it added 10 minutes to my deploys to a fresh instance. Is there a faster way to do this?
.ebextensions/02-tesseract.config
packages:
yum:
autoconf: []
automake: []
libtool: []
libpng-devel: []
libtiff-devel: []
zlib-devel: []
container_commands:
01-command:
command: mkdir -p install
cwd: /home/ec2-user
02-command:
command: cp .ebextensions/scripts/install_tesseract.sh /home/ec2-user/install/
03-command:
command: bash install/install_tesseract.sh
cwd: /home/ec2-user
.ebextensions/scripts/install_tesseract.sh
#!/usr/bin/env bash
cd_to_install () {
cd /home/ec2-user/install
}
cd_to () {
cd /home/ec2-user/install/$1
}
if ! [ -x "$(command -v tesseract)" ]; then
# Add `usr/local/bin` to PATH
echo 'pathmunge /usr/local/bin' > /etc/profile.d/usr_local.sh
chmod +x /etc/profile.d/usr_local.sh
# Install leptonica
cd_to_install
wget http://www.leptonica.org/source/leptonica-1.73.tar.gz
tar -zxvf leptonica-1.73.tar.gz
cd_to leptonica-1.73
./configure
make
make install
rm -rf /home/ec2-user/install/leptonica-1.73.tar.gz
rm -rf /home/ec2-user/install/leptonica-1.73
# Install tesseract ~ the jewel of Odin's treasure room
cd_to_install
wget https://github.com/tesseract-ocr/tesseract/archive/3.04.01.tar.gz
tar -zxvf 3.04.01.tar.gz
cd_to tesseract-3.04.01
./autogen.sh
./configure
make
make install
ldconfig
rm -rf /home/ec2-user/install/3.04.01.tar.gz
rm -rf /home/ec2-user/install/tesseract-3.04.01
# Install tessdata
cd_to_install
wget https://github.com/tesseract-ocr/tessdata/archive/3.04.00.tar.gz
tar -zxvf 3.04.00.tar.gz
cp /home/ec2-user/install/tessdata-3.04.00/eng.* /usr/local/share/tessdata/
rm -rf /home/ec2-user/install/3.04.00.tar.gz
rm -rf /home/ec2-user/install/tessdata-3.04.00
fi
Short answer
.ebextensions/02-tesseract.config
commands:
01-libwebp:
command: "yum --enablerepo=epel --disablerepo=amzn-main -y install libwebp"
02-tesseract:
command: "yum --enablerepo=epel -y install tesseract"
Long answer
I'm not familiar with non-Ubuntu package managers or ebextensions, so after some digging, I found that there are precompiled binaries that can be installed on Amazon Linux in the stable EPEL repo.
The first obstacle was figuring out how to use the EPEL repo. The easiest way is to use the enablerepo option on the yum command.
That gets us here:
yum --enablerepo=epel install tesseract
Next, I had to resolve this dependency error:
[root#ip-10-0-1-193 ec2-user]# yum install --enablerepo=epel tesseract
Loaded plugins: priorities, update-motd, upgrade-helper
951 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package tesseract.x86_64 0:3.04.00-3.el6 will be installed
--> Processing Dependency: liblept.so.4()(64bit) for package: tesseract-3.04.00-3.el6.x86_64
--> Running transaction check
---> Package leptonica.x86_64 0:1.72-2.el6 will be installed
--> Processing Dependency: libwebp.so.5()(64bit) for package: leptonica-1.72-2.el6.x86_64
--> Finished Dependency Resolution
Error: Package: leptonica-1.72-2.el6.x86_64 (epel)
Requires: libwebp.so.5()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
I found the solution here
Just adding the epel repo doesn't solve it, as the packages in the
amzn-main repository seem to overrule those in the epel repository. If
the libwebp package in the amzn-main repo are excluded it should work
The Tesseract install has some dependencies found in the amzn-main repo. This is why I first install libwebp with --disablerepo=amzn-main.
yum --enablerepo=epel --disablerepo=amzn-main install libwebp
yum --enablerepo=epel install tesseract
Finally, here's how you can install yum packages on Elastic Beanstalk with options:
.ebextensions/02-tesseract.config
commands:
01-libwebp:
command: "yum --enablerepo=epel --disablerepo=amzn-main -y install libwebp"
02-tesseract:
command: "yum --enablerepo=epel -y install tesseract"
Fortunately, this is also the easiest way to install Tesseract on Elastic Beanstalk!
Here is the portion of my .travis.yml that handles the dependency installation for my wxWidgets/CxxTest project:
install:
- echo | sudo apt-add-repository ppa:dhart/ppa
- sudo apt-get update -qq
- sudo apt-get install -qq libwxgtk2.8 cxxtest
env: CXXTEST=/usr/include
Everything installs fine, and the project builds successfully, but when it comes time to call cxxtestgen, it goes kaput.
cxxtestgen: Command not found
How do I install CxxTest onto Travis-CI while making the cxxtestgen command accessible?
Here is my github project: https://github.com/gbchaosmaster/nds-toolkit
$ sudo apt-get install -qq cxxtest
Unable to locate package cxxtest
cxxtest is not in trusted archive in ubuntu 12.04 (precise), which is used in travis-ci now (2012.11), it can't be installed there directly using command apt-get install
See https://launchpad.net/ubuntu/+source/cxxtest
Using either Ubuntu PPA or installation of cxxtest itself (like build from source)
Beside the answer, below is the way to debug in general
# .travis-ci.yml
- sudo apt-get install -qq cxxtest
env: CXXTEST=/usr/include
# Test build success and unit test passing.
script:
- dpkg -L cxxtest
- echo $PATH
- cxxtestgen