I want to integrate my C++ project with Jenkins using Docker.
First I created a setup for Jenkins using:
Dockerfile:
FROM jenkins/jenkins:latest
USER root
RUN apt-get update && apt-get install git && apt-get install make g++ -y
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
ENV CASC_JENKINS_CONFIG /var/jenkins_home/casc.yaml
ENV JENKINS_ADMIN_ID=admin
ENV JENKINS_ADMIN_PASSWORD=password
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
COPY casc.yaml /var/jenkins_home/casc.yaml
plugins.txt
ant:latest
antisamy-markup-formatter:latest
authorize-project:latest
build-timeout:latest
cloudbees-folder:latest
configuration-as-code:latest
credentials-binding:latest
email-ext:latest
git:latest
github-branch-source:latest
gradle:latest
ldap:latest
mailer:latest
matrix-auth:latest
pam-auth:latest
pipeline-github-lib:latest
pipeline-stage-view:latest
ssh-slaves:latest
timestamper:latest
workflow-aggregator:latest
ws-cleanup:latest
casc.yaml
jenkins:
securityRealm:
local:
allowsSignup: false
users:
- id: ${JENKINS_ADMIN_ID}
password: ${JENKINS_ADMIN_PASSWORD}
authorizationStrategy:
globalMatrix:
permissions:
- "USER:Overall/Administer:admin"
- "GROUP:Overall/Read:authenticated"
remotingSecurity:
enabled: true
security:
queueItemAuthenticator:
authenticators:
- global:
strategy: triggeringUsersAuthorizationStrategy
unclassified:
location:
url: http://127.0.0.1:8080/
So far, so good. I used docker build -t jenkins:jcasc . and docker run --name jenkins --rm -p 8080:8080 jenkins:jcasc and Jenkins starts normally. I entered the user name (admin) and the password and it worked.
There are although 2 issues that Jenkins displays:
-> It appears that your reverse proxy set up is broken.
-> Building on the built-in node can be a security issue. You should set up distributed builds. See the documentation.
I ignored them so far, and continued with my project and installed Blue Ocean.
After installing Blue Ocean I connected it to my GitHub repository and for this I created a token and the pipeline was created, but it is not giving any result.
The log says:
Branch indexing
Connecting to https://api.github.com with no credentials, anonymous access
Jenkins-Imposed API Limiter: Current quota for Github API usage has 37 remaining (1 over budget). Next quota of 60 in 36 min. Sleeping for 4 min 43 sec.
Jenkins is attempting to evenly distribute GitHub API requests. To configure a different rate limiting strategy, such as having Jenkins restrict GitHub API requests only when near or above the GitHub rate limit, go to "GitHub API usage" under "Configure System" in the Jenkins settings.
My C++ project looks like this:
-->HelloWorld
|
-->HelloWorld.cpp
-->scripts
|
-->Linux-Build.sh
|
-->Linux-Run.sh
-->Jenkinsfile
HelloWorld.cpp:
#include <iostream>
int main(){
std::cout << "Hello World!" << std::endl;
return 0;
}
Linux-Build.sh
#!/bin/bash
vendor/bin/premake/premake5 gmake2
make
Linux-Run.sh
#!/bin/bash
bin/Debug/HelloWorld
Jenkinsfile
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'echo "Building..."'
sh 'chmod +x scripts/Linux-Build.sh'
sh 'scripts/Linux-Build.sh'
archiveArtifacts artifacts: 'bin/Debug/*', fingerprint: true
}
}
stage('Test') {
steps {
sh 'echo "Running..."'
sh 'chmod +x scripts/Linux-Run.sh'
sh 'scripts/Linux-Run.sh'
}
}
}
}
So, where might be the problem? I don't know if the problem is in my C++ code, or in the Docker configuration.
I am also a big fan of cmake, instead of make and I would prefer to use cmake for this, but I am not sure how to integrate this and make it work.
EDIT:
In the tutorial that I followed there was a "vendor" directory but the instructor didn't show what was in that directory.
EDIT:
The program took 4 minutes but it eventually did something and the error is:
+ scripts/Linux-Build.sh
scripts/Linux-Build.sh: line 3: vendor/bin/premake/premake5: No such file or directory
make: *** No targets specified and no makefile found. Stop.
script returned exit code 2
So, the problem is in the C++ project, not Docker.
Related
I have a simple project but i have no experiences with GitLab-ci. I use ROS and cmake to build my project on my local machine(ubuntu18-04).
now I want to build my project on GitLab but this doesn't look easy for me.
Steps:
1-) installed binary runners from here
2-) registered runners from here for Linux
- used docker as an executor (like gitlabci; i have no experience with docker)
- selected ruby:2.6 default image
3-) Now' i can see my runner under Settings > CI/CD -> Runners
4-) Creating example .yml provided from gitlab
build-job:
stage: build
script:
- echo "Hello, $GITLAB_USER_LOGIN!"
test-job1:
stage: test
script:
- echo "This job tests something"
test-job2:
stage: test
script:
- echo "This job tests something, but takes more time than test-job1."
- echo "After the echo commands complete, it runs the sleep command for 20 seconds"
- echo "which simulates a test that runs 20 seconds longer than test-job1"
- sleep 20
deploy-prod:
stage: deploy
script:
- echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
And its working!
But now I want to build my code on gitlab-ci. File structure:
scripts
->build.sh
src
->Cmakelist.txt
->codes.cpp
binaries
->outputs ll be here.
.gitlab-ci.yml
build.sh doing everything that I want:
...
mkdir build
cd build
cmake .. -GNinja
ninja
So I need just run it. But I don't know how to install prerequisites.
Which system exactly am I working on right now and how can I install prerequisites? (ubuntu 18.04 - docker - runner.. I just got mixed up )
Which system exactly am I working on right now
You clicked selected ruby:2.6 default image so you are on ruby:2.6. You may then browse docker hub : https://hub.docker.com/_/ruby and dockerfile https://github.com/docker-library/ruby/blob/8e49e25b591d4cfa6324b6dada4f16629a1e51ce/2.6/buster/Dockerfile - I see it has "buster" which is the name of one of debian releases, so I guess it's debian.
how can I install prerequisites?
That depends on the image your are using, different linux distributions use different package mangers. I usually take a look at wiki package manager.
You could just something like:
build:
image: ubuntu
script:
- apt-get install -y cmake gcc whatever-else-like-you-have-on-your-machine
- ./scripts/build.sh
I am using normal jenkins installation (NOT THE DOCKER IMAGE) on a normal AWS ec2 instance, with docker engine installed along side jenkins.
I have a simple jenkins pipeline like this:
pipeline {
agent none
stages {
stage('Example Build') {
agent { docker {
image 'cypress/base:latest'
args '--privileged --env CYPRESS_CACHE_FOLDER=~/.cache'
} }
steps {
sh 'ls'
sh 'node --version'
sh 'yarn install'
sh 'make e2e-test'
}
}
}
}
this will make the pipeline fail in the yarn install step while installing cypress although all it's dependenices is satisfied from the cypress image.
ERROR LOG FROM JENKINS
error /var/lib/jenkins/workspace/Devops-Capstone-Project_master/node_modules/cypress: Command failed.
Exit code: 1
Command: node index.js --exec install
Arguments:
Directory: /var/lib/jenkins/workspace/Devops-Capstone-Project_master/node_modules/cypress
Output:
Cypress cannot write to the cache directory due to file permissions
See discussion and possible solutions at
https://github.com/cypress-io/cypress/issues/1281
----------
Failed to access /.cache:
EACCES: permission denied, mkdir '/.cache'
After some investigation i found that although i have provided the environment variable "CYPRESS_CACHE_FOLDER=~/.cache" to override the default location in the root directory, and also provided the "--privileged". it fails because for some reason jenkins and docker is forcing their args and user mapping from the jenkins host.
I have also tried providing "-u 1000:1000" to override the user mapping but it didn't work.
What could possibly be wrong? and any recommendations or work arounds about this issue?
Thanks ,,
I have found a work around by creating a docker file to build the image and pass the jenkins user id and group to it as build arguments, as described here on this thread .
But this is not guaranteed to work on multiple nodes (master->slaves) jenkins installations as the jenkins user id and group may differ.
I have a container builder step
steps:
- id: dockerbuild
name: gcr.io/cloud-builders/docker
entrypoint: 'bash'
args:
- -c
- |
docker build . -t test
images: ['gcr.io/project/test']
The Dockerfile used to create this test image has gsutil specific commands like
FROM gcr.io/cloud-builders/gcloud
RUN gsutil ls
When I submit a docker build to container builder service using
gcloud container builds submit --config cloudbuild.yml
I see the following error
You are attempting to perform an operation that requires a project id, with none configured. Please re-run gsutil config and make sure to follow the instructions for finding and entering your default project id.
The command '/bin/sh -c gsutil ls' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1
My question is, how do we use gcloud/gsutil commands inside the DockerFile so that I can run inside a docker build step ?
To invoke "gcloud commands." using the tool builder, you need Container Builder service account, because it executes your builds on your behalf.
Here in this GitHub there is an example for cloud-builders using the gcloud command:
Note : you have to specify $PROJECT_ID it's mandatory for your builder to work.
To do this, your Dockerfile either needs to start from a base image that has the cloud SDK installed already (like FROM gcr.io/cloud-builders/gcloud) or you would need to install it. Here's a Dockerfile that installs it: https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile.slim
What I am trying to do is to enable Continuous delivery from GitLab to my compute engine on Google Cloude. I have Ubuntu 16.04 TSL running over there. I did install all components needed to run my project like: Swift, vapor, nginx.
I have manage to install Gitlab runner as well and created a runner whcihc is accessible from my gitlab repo. Everytime I do push on master the runner triggers. What happen is a failure due to:
could not create leading directories of '/home/gitlab-runner/builds/2bbbbbd/0/Server/Packages/vapor.git': Permission denied
If I change the permissions to chmod -R 777 It will hange on running for build stage visible on gitlab pipeline.
I did something like:
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/builds
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/cache
but this haven't help, the error is same Permission denied
Below you have my .gitlab-ci.yml
before_script:
- swift --version
stages:
- build
- deploy
job_build:
stage: build
before_script:
- vapor clean
script:
- vapor build --release
only:
- master
job_run_app:
stage: deploy
script:
- echo "Deploy a API"
- vapor run --name=App --env=production
environment:
name: production
job_run_frontend:
stage: deploy
script:
- echo "Deploy a Frontend"
- vapor run --name=Frontend --env=production
environment:
name: production
But that haven't pass to next stage eg. deploy. I had waited more then 14h for that but with out result.
And... I have few more questions:
Gitlab runner creates builds under location /home/gitlab-runner/builds/ in this location every new job have own folder. for eg. /home/gitlab-runner/builds/2bbbbbd/ in which is my project and the commands are executed. So what happens when the first one is running and I do deploy new version? the ports are blocked by the first instance and so on?
If I want to enable supervisor how do I do that with this when every time I deploy folder is different?
Can anyone explain or show me or point me to tutorial how do Continuous deployment with out docker?
How to start a service using GitLab runner
Thanks to long deep search I finally found an answer! The full article can be found above.
Briefly GitLab CI documentation recommends using dpl for deployment. Gitlab runner run test and process should end. The runner is designed to kill all created processes after finishing each build. The GitLab runner is unable to perform operations outside the catalogue.
I try to support the use of the webp format with EB, however it's not working as expected...
I created a .config file in .ebextensions with this:
commands:
01-command:
command: wget https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-0.5.0.tar.gz
02-command:
command: tar xvzf libwebp-0.5.0.tar.gz
03-command:
command: cd libwebp-0.5.0
04-command:
command: ./configure
05-command:
command: make
06-command:
command: sudo make install
But when deploying I got this error:
ERROR: Command failed on instance. Return code: 127 Output: /bin/sh: ./configure: No such file or directory.
Am I doing something wrong?
(environment: 64bit Amazon Linux 2015.09 v2.0.6 running PHP 5.6)
You need to execute the install post deployment. AWS hasn't really documented how to execute commands post deployment, so I'll do so here.
commands:
create_post_dir:
command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/99_install_libwebp.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
. /opt/elasticbeanstalk/support/envvars
cd $EB_CONFIG_APP_CURRENT
wget https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-0.5.0.tar.gz
tar xvzf libwebp-0.5.0.tar.gz
cd libwebp-0.5.0
sudo ./configure
sudo make
sudo make install
As I mentioned, AWS has not really documented that you can actually execute scripts on ElasticBeanstalk post deployment. If you talk a look in the eb-commandprocessor.log file you will see that eb looks for AppDeployPreHook (4 of 6) and AppDeployPostHook (1 of 2). It will look something like this:
[2016-04-13T14:15:22.955Z] DEBUG [8851] : Loaded 6 actions for stage 0.<br>
[2016-04-13T14:15:22.955Z] INFO [8851] : Running 1 of 6 actions: InfraWriteConfig...<br>
[2016-04-13T14:15:22.962Z] INFO [8851] : Running 2 of 6 actions: DownloadSourceBundle...<br>
[2016-04-13T14:15:23.606Z] INFO [8851] : Running 3 of 6 actions: EbExtensionPreBuild...<br>
[2016-04-13T14:15:24.229Z] INFO [8851] : Running 4 of 6 actions: AppDeployPreHook...<br>
[2016-04-13T14:15:28.469Z] INFO [8851] : Running 5 of 6 actions: EbExtensionPostBuild...<br>
[2016-04-13T14:15:28.970Z] INFO [8851] : Running 6 of 6 actions: InfraCleanEbextension...<br>
[2016-04-13T14:15:28.974Z] INFO [8851] : Running stage 1 of command CMD-AppDeploy...<br>
[2016-04-13T14:15:28.974Z] DEBUG [8851] : Loaded 2 actions for stage 1.<br>
[2016-04-13T14:15:28.974Z] INFO [8851] : Running 1 of 2 actions: AppDeployEnactHook...<br>
[2016-04-13T14:15:29.600Z] INFO [8851] : Running 2 of 2 actions: AppDeployPostHook...<br>
[2016-04-13T14:16:42.048Z] INFO [8851] : Running AddonsAfter for command CMD-AppDeploy... <br>
That little "AppDeployPostHook" tells us that it is searching for scripts to run post deployment. You can find the eb deployment scripts in the /opt/elasticbeanstalk directory on the server, and if you ssh in and ls on that directory you'll find hooks, which is what we're looking for, and if you cd hooks you'll find the appdeploy directory, cd appdeploy and then ls and you'll get two directories pre and enact. This seems mundane but is really great, because now we know where eb is looking for scripts it's running. Since the AppDeployPreHook scripts are executing from the "pre" directory we know that we'll need a "post" directory to execute a command post deployment with that AppDeployPostHook that eb is running. Now that we know what to do, we can start writing our commands.
create_post_dir First step is to actually going to create the "post" directory on the server using the mkdir command. mkdir "/opt/elasticbeanstalk/hooks/appdeploy/post" will do that for us, so we'll create that as the command.
files The files config allows us to create a file in a directory via ElasticBeanstalk. Pretty convenient for our purposes! The first line of the files action gives us the name of the file to create. We'll create a shell script to execute out commands, and you can call it whatever you want, but I'd start with 99 and go onwards. We'll call this shell script that we're creating "99_install_libwebp.sh".
File settings The next three lines set the file settings. Make sure root owns them and that there 000755'd.
File Contents This is the content of the file we're creating. Straight forward. Put your shell script in there and you're good to go.
Load environment vars We opted to load the eb environment variables so our script can know where the current version of the app is. It's usually in /var/app/current but it could be elsewhere depending on a variety of factors. We'll use the environment variables to make life a bit easier for us.
Change to our current app directory We're going to cd to our current app directory so we can do what we we're here to do.
Get the package we want use wget to get the libwebp we want
Unpack the package self explanatory
Change to the package directory Now that we've unpacked the package we can change to the package directory.
Do what we need to do We can now run our ./configure, make, and make install.
That's it. You can use the stealthy AppDeployPostHook to run pretty much any post deployment command that you need. Super useful if you need to install packages, restart services, or do anything else post deployment.
I added the code I deployed to Github, for easy reference too. https://github.com/hephalump/testphp
Note: I did this successfully running a slightly different environment. I used ElasticBeanstalk to deploy a new PHP application using the latest environment version which is PHP 5.6 on 64bit Amazon Linux 2016.03 v2.1.0; the environment type that you are using was not available as an option to me... Actually, this was the only version with PHP 5.6 that was available to me so I went with that.