How to set variable inline in gitlab-ci.yaml based on regex matching? - regex

I am trying to create a variable in gitlab-ci.yaml based on the name of the branch.
Suppose I am pushing to a branch named 3.2.7
Here is the situation:
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
variables:
PRODUCTION_BRANCH: "master"
STAGING_BRANCH: (\d)\.(\d)\.(\d)
.deploy_rules:
rules:
- if: '$CI_COMMIT_BRANCH =~ /$STAGING_BRANCH/'
variables:
SERVER_PORT: 3007 # TODO: should be 300d ; d is the second digit
I want to generate 3002 inline using regex matching.
How can I do this?
I have done some research and seems I have to use sed but I am not sure if it is the best way to do it and how to do it.
TO MAKE THE PROBLEM SIMPLER
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
variables:
TEST_VAR: sed -E 's/(\d)\.(\d)\.(\d)/300\2/gm;t;d' <<< $CI_COMMIT_BRANCH
stages:
- temp
temp:
stage: temp
script:
- echo $TEST_VAR
Should be echoing 3002 but it is echoing sed -E 's/(\d)\.(\d)\.(\d)/300\2/gm;t;d' <<< 3.2.7

You can't use variables in the regex pattern. You just have to write the regex verbatim, it cannot be directly parameterized. You also cannot use sed or other Linux utilities in variables: or other parts of your yaml. You're bound to the limitations of YAML specification and features provided by GitLab.
However, there is an option available to you that will fit your stated use case.
Dynamic variables
TEST_VAR: sed -E 's/(\d).(\d).(\d)/300\2/gm;t;d' <<< $CI_COMMIT_BRANCH
While you can't use sed or other utilities directly in variables: declarations, you can use dotenv artifacts via artifacts:reports:dotenv to set variables dynamically.
For example, a job can use sed or whatever other utilities you like to create variables which will be used by the rest of the pipeline.
stages:
- temp
create_variables:
stage: .pre
script:
- TEST_VAR="$(sed -E 's/(\d)\.(\d)\.(\d)/300\2/gm;t;d' <<< ${CI_COMMIT_BRANCH})"
- echo "TEST_VAR=${TEST_VAR}" >> dotenv.txt
artifacts:
reports:
dotenv: dotenv.txt
temp:
stage: temp
script:
- echo $TEST_VAR
Here, the .pre stage is used, which is a special stage that is always ordered before every other stage. The dotenv artifact from the create_variables job will dynamically create variables for the jobs in subsequent stages that receive the artifact.

Related

How to grep lines of an indented paragraph using regex?

I've got a docker-compose file, from which I want to extract it's environment variables within the section "environment:".
image: non-important
environment:
DATABASE_URL: postgresql://localhost:5234
REDIS_URL: 'redis://redis:6379'
PRODUCTION_URL: 'http://localhost:4000'
CLIENT_URL: 'http://localhost:3000'
TOKEN_SEND_MODE: 'cookie|query_param'
other_service:
...
Is there a way to grep only those variables, between "environment:" and "other_service:"?
The only big difference from the other lines is that they are more indented than others, but I don't know to differentiate them from the rest using a regex expression.
Using perl :
perl -0 -pe 's/.*(\n\s*DATABASE_URL:.*?)\bother_service:/$1/s' file
Output
DATABASE_URL: postgresql://localhost:5234
REDIS_URL: 'redis://redis:6379'
PRODUCTION_URL: 'http://localhost:4000'
CLIENT_URL: 'http://localhost:3000'
TOKEN_SEND_MODE: 'cookie|query_param'

How to use if-else condition on gitlabci

How to use if else condition inside the gitlab-CI.
I have below code:
deploy-dev:
image: testimage
environment: dev
tags:
- kubectl
script:
- kubectl apply -f demo1 --record=true
- kubectl apply -f demo2 --record=true
Now I want to add a condition something like this
script:
- (if [ "$flag" == "true" ]; then kubectl apply -f demo1 --record=true; else kubectl apply -f demo2 --record=true);
Could someone provide the correct syntax for the same? Is there any documentation for the conditions (if-else, for loop) in gitlabci?
Hereunder three syntax options for that kind of statement. From gitlab-ci documentation :
Using shell variable
deploy-dev:
image: testimage
environment: dev
tags:
- kubectl
script:
- if [ "$flag" == "true" ]; then MODULE="demo1"; else MODULE="demo2"; fi
- kubectl apply -f ${MODULE} --record=true
Using shell variable with yaml multiline block
deploy-dev:
image: testimage
environment: dev
tags:
- kubectl
script:
- >
if [ "$flag" == "true" ]; then
kubectl apply -f demo1 --record=true
else
kubectl apply -f demo2 --record=true
fi
Using gitlab rules
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
when: never
- if: '$CI_PIPELINE_SOURCE == "push"'
when: never
- when: always
Using gitlab templates and variables
demo1-deploy-dev:
extends: .deploy-dev
only:
variables: [ $flag == "true" ]
variables:
MODULE: demo1
demo2-deploy-dev:
extends: .deploy-dev
only:
variables: [ $flag == "false" ]
variables:
MODULE: demo2
.deploy-dev:
image: testimage
environment: dev
tags:
- kubectl
script:
- kubectl apply -f ${MODULE} --record=true
Note that with GitLab 13.3 (August 2020), there is an improvement to the if-else rule syntax:
CI/CD rules:if support logical expressions with parentheses
If you use the rules keyword with if clauses, it’s now even more powerful, with support for bracketed expressions evaluated by the pipeline processor.
You can use more complex and efficient AND (&&) / OR (||) expressions, making your pipelines rules more logical, powerful, and easier to manage.
See Documentation and Issue.
And, with GitLab 13.8 (January 2021)
Support variables for pipeline rules
Previously, the rules keyword was limited in scope and only determined if a job should be included or excluded from pipelines. In this release, you can now decide if certain conditions are met and subsequently override variables in jobs, providing you with more flexibility when configuring your pipelines.
See Documentation and Issue.
With GitLab 13.12 (May 2021):
Support variables in CI/CD pipeline 'workflow:rules'
Previously, the rules keyword was limited in scope and only determined if a job should be included or excluded from pipelines. In 13.8, we added the ability to use the variables keyword with rules to set variable values in a job based on which rule matched.
In this release we’ve extended this ability to workflow: rules, so you can set variable values for the whole pipeline if certain conditions match.
This helps you make your pipelines even more flexible.
See Documentation and Issue.
I think you need to just add a semicolon and closing "fi" at the end.
I couldn't find a link to documentation.
script:
- (if [ "$flag" == "true" ]; then kubectl apply -f demo1 --record=true; else kubectl apply -f demo2 --record=true; fi);
In addition, in the case of a multiline block if you want or need to preserve line breaks you can use the pipe character:
script: |
if [ "$flag" == "true" ]; then
kubectl apply -f demo1 --record=true
else
kubectl apply -f demo2 --record=true
fi
To go deeper, visit https://yaml-multiline.info/
You may consider checking rules
It allows for a list of individual rule objects to be evaluated in order, until one matches and dynamically provides attributes to the job.
Available rule clauses include:
if (similar to only:variables)
changes (same as only:changes)
exists
Example:
job:
script: "echo Hello, Rules!"
rules:
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
when: always
- if: '$VAR =~ /pattern/'
when: manual
- when: on_success
This worked for me when using powershell based gitlab scripts:
script:
- 'if ($flag -eq "true") { kubectl apply -f demo1 --record=true; } else { kubectl apply -f demo2 --record=true; }'

Cloudbuild.yaml command with nested quotes

I am trying to run the following command found at http://blog.wrouesnel.com/articles/Totally%20static%20Go%20builds/:
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' .
The two inner layers of quotes are tripping me up. How to deal with this in a cloudbuild.yaml file?
Escaping quotes don't seem to work:
steps:
- name: 'gcr.io/cloud-builders/go'
args: ['build', '-o', 'main', '-ldflags', "'-extldflags \"-static\"'", '.']
env:
- 'GOOS=linux'
Update:
There is no need for such quotes. See comment in Github here: https://github.com/GoogleCloudPlatform/cloud-builders/issues/146#issuecomment-337890587
===
Original Answer
Well, to quote ' within '-quoted strings, use '' as per YAML specification:
http://yaml.org/spec/current.html#id2534365
e.g. 'here''s to a toast!'
For the above args, it would be:
['build', '-o', 'main', '-ldflags', '''-extldflags "-static"''', '.']
Whether or not the command works within Cloud Builder is beyond the scope of this question.

Is it possible to set a top-level only/except in a .gitlab-ci.yml file?

I have three stages in my CI file, they all have only/except like this:
test:
only:
- tags
except:
- branches
script:
- npm run test
Seems redundant to have the only/except in three places. Is there a way to set this at the top level of the script config? Don't see anything like that in the docs.
You can use the map merging feature: https://docs.gitlab.com/ee/ci/yaml/#special-yaml-features
.job_template: &job_definition
only:
- tags
except:
- branches
test1:
<<: *job_definition
script:
- npm run test
test2:
<<: *job_definition
script:
- # ...

Drone YAML if condition is not working

I have below conditional logic in the Drone YAML, but I saw that the control is not going inside that, even though drone branch is "develop". How to fix this, did I do anything wrong?
commands:
- "./gradlew clean build"
- echo "${DRONE_BRANCH}"
- echo "${DRONE_BRANCH}" = "develop"
- >
if [ "${DRONE_BRANCH}" = "develop" ]; then
export CLOUD_USER_KEY=$STAGE_CLOUD_USER_KEY
export HOST_NAME="11.22.111.111"
fi
- >
if [ "${DRONE_BRANCH}" = "master" ]; then
export CLOUD_USER_KEY=$PROD_CLOUD_USER_KEY
export HOST_NAME="11.22.111.112"
fi
- echo "CLOUD_USER_KEY "${CLOUD_USER_KEY}
- echo "HOST NAME "${HOST_NAME}
> is the YAML indicator for a folded block scalar. Line folding in YAML means that a line break between two non-empty consecutive lines is changed into a space. This is not what you want when writing bash commands into YAML!
For example, this simple folded block scalar (using only parts of your original YAML file):
- >
export CLOUD_USER_KEY=$STAGE_CLOUD_USER_KEY
export HOST_NAME="11.22.111.111"
will be parsed as:
- "export CLOUD_USER_KEY=$STAGE_CLOUD_USER_KEY export HOST_NAME=\"11.22.111.111\"\n"
Which won't work. Use a literal block scalar instead which preserves line endings:
- |
export CLOUD_USER_KEY=$STAGE_CLOUD_USER_KEY
export HOST_NAME="11.22.111.111"