I'm trying to replace a character in a variable within a GITHUB actions step
- name: Set Up DB Name
run: |
DB_NAME="${GITHUB_REF_SLUG/-/_}"
echo $DB_NAME
I'm getting a bad request error
What am I doing wrong?
I successfully made the character replace works (with GITHUB_REPOSITORY) using this implementation:
job1:
runs-on: ubuntu-latest
steps:
- name: character-replacement-test
run: |
REPO=$GITHUB_REPOSITORY
DB_NAME="${REPO//-/_}"
echo $DB_NAME
I couldn't get to the same result with 2 lines.
(But someone more experienced with bash might help us get there as well).
Evidence:
So in your case, it should work using this code if you substitute the GITHUB_REPOSITORY by GITHUB_REF_SLUG in your workflow.
I used this post as reference.
Related
I'm trying to run a dataflow job using cloud build
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args:
dataflow
jobs
run
google-template-job
--gcs-location=gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery
--parameters=inputSubscription='projects/$PROJECT_ID/subscriptions/messages'
--parameters=outputTableSpec="$PROJECT_ID:beam_samples.streaming_beam"
--staging-location=gs://cloudbuild-dataflow-testproject123456789-313307/tmp'
--region=us-central1
Every time I trigger the build I get the following error
ERROR: (gcloud.dataflow.jobs.run) INVALID_ARGUMENT: The template parameters are invalid.
- '#type': type.googleapis.com/google.dataflow.v1beta3.InvalidTemplateParameters
parameterViolations:
- description: 'Unmatched regex: ^projects\/[^\n\r\/]+\/subscriptions\/[^\n\r\/]+$'
parameter: inputSubscription
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcloud" failed: step exited with non-zero status: 1
My project id has a ' - ' in it so if I replace the $PROJECT_ID with the value of project id, I still get the same error, is there any workaround for this. I don't have any control over stopping the regex check since it's a google provided template.
How do I get past this
Got it. It's only a command interpreter issue. If you put single quote, you prevent any evaluation of the inside string.
In you case
--parameters=inputSubscription='projects/$PROJECT_ID/subscriptions/messages'
the value 'projects/$PROJECT_ID/subscriptions/messages' is taken as is and therefore the project ID contain uppercase and underscore, that violate the regex pattern.
Change for double quote and it should work great!
--parameters=inputSubscription="projects/$PROJECT_ID/subscriptions/messages"
I am trying to run a for loop to traverse multiple folders in the cloned code using the following method
commands:
- folders=`ls`
- for value in ${folders}
- do
- some_code_here
- done
Also, I've tried different ways like
- for value in ${folders}; do
- some_code_here
- done
But none of them works.
You should write for-loops as one-liner. As CodeBuild merges all lines in one command together, you can write for-loops in a readable format as follows:
- folders=`ls`
- for value in $folders;
do
echo $value;
done
- echo "run the next command"
I think you can use YAML multiline string.
- |
for value in ${folders}; do
some_code_here
done
I would like to setup my project_dev CI only for 3 branches and specific kind of tags like: dev_1.0, dev_1.1, dev_1.2.
How can I achieve that?
This is what I have now:
project_dev:
stage: dev
script:
- export
- bundle exec pod repo update
- bundle exec pod install
- bundle exec fastlane crashlytics_project_dev
after_script:
- rm -rf ~/Library/Developer/Xcode/Archives || true
when: manual
only:
- develop
- release
- master
- //here I need to add condition to fire that stage additionally only for specific tags. How can I setup regexp here?
tags:
- iOS
When I type it like:
only:
- branches
- /^dev_[0-9.]*$/
It also runs the CI for tags like: dev1.2 but it should not. Why? Is there a regexp for tags at all?
Sounds like a regular expression question. I just created a project on gitlab.com for the regular expression.
File: .gitlab-ci.yml
project_dev:
# Irrelevant keys is skipped
script:
- echo "Hello World"
only:
- develop
- release
- master
- /^dev_[0-9]+(?:.[0-9]+)+$/ # regular expression
I was pushed all of tags you mentioned to test this regular expression.
As you can see , It will match tags like dev_1.0, dev_1.1, but the job project_dev will not be triggered by tag dev1.2, You can check the result on pipeline pages
Instead of using only/except you can use rules which are more powerful.
Rules support regex pattern matching.
Your rule for excepting only specific kind of branches/tags like dev_1.0, dev_1.1, dev_1.2 should look like:
rules:
- if: '$CI_COMMIT_BRANCH =~ /^dev_[0-9]+\.[0-9]+$/ || $CI_COMMIT_TAG =~ /^dev_[0-9]+\.[0-9]+$/'
Predefined environment variables like CI_COMMIT_BRANCH and CI_COMMIT_TAG are described here.
Gitlab.com ?
You could try a combination of except and only.
Something like
only:
- tags
- branches
except:
- /^(?!(branch1|branch2|branch3|dev_[0-9.]*$)$).*$/
The idea being, allowing only branches and tags to trigger a job, with the exception of everything different from branch[1-3] and dev_ branches/tags
And here is the official documentation for this:
GitLab CI/CD pipeline configuration reference
There you find the section for only/except with the supported regex syntax, although it states that:
only and except are not being actively developed. rules is the preferred keyword to control when to add jobs to pipelines.
We want to migrate from Bitbucket Pipelines to Google Cloud Build to test, build and push Docker images.
How can we use environment variables without a CryptoKey? For example:
- printf "https://registry.npmjs.org/:_authToken=${NPM_TOKEN}\nregistry=https://registry.npmjs.org" > ~/.npmrc
To use environment variables in the args portion of your build steps you need:
"a shell to resolve environment variables with $$" (as mentioned in the example code here)
and you also need to be careful with your usage of quotes (use single quotes)
See below the break for a more detailed explanation of these two points.
While the Using encrypted resources docs that David Bendory also linked to (and which you probably based your assumption on) show how to do this using an encrypted environment variable specified via secretEnv, this is not a requirement and it works with normal environment variables too.
In your specific case you'll need to modify your build step to look something like this:
# you didn't show us which builder you're using - this is just one example of
# how you can get a shell using one of the supported builder images
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: ['-c', 'printf "https://registry.npmjs.org/:_authToken=%s\nregistry=https://registry.npmjs.org" $$NPM_TOKEN > ~/.npmrc']
Note the usage of %s in the string to be formatted and how the environment variable is passed as an argument to printf. I'm not aware of a way that you can include an environment variable value directly in the format string.
Alternatively you could use echo as follows:
args: ['-c', 'echo "https://registry.npmjs.org/:_authToken=$${NPM_TOKEN}\nregistry=https://registry.npmjs.org" > ~/.npmrc']
Detailed explanation:
My first point at the top can actually be split in two:
you need a shell to resolve environment variables, and
you need to escape the $ character so that Cloud Build doesn't try to perform a substitution here
If you don't do 2. your build will fail with an error like: Error merging substitutions and validating build: Error validating build: key in the template "NPM_TOKEN" is not a valid built-in substitution
You should read through the Substituting variable values docs and make sure that you understand how that works. Then you need to realise that you are not performing a substitution here, at least not a Cloud Build substitution. You're asking the shell to perform a substitution.
In that context, 2. is actually the only useful piece of information that you'll get from the Substituting variable values docs (that $$ evaluates to the literal character $).
My second point at the top may be obvious if you're used to working with the shell a lot. The reason for needing to use single quotes is well explained by these two questions. Basically: "You need to use single quotes to prevent interpolation happening in your calling shell."
That sounds like you want to use Encrypted Secrets: https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials
I'm trying to remove sensitive data like passwords from my Git history. Instead of deleting whole files I just want to substitute the passwords with removedSensitiveInfo. This is what I came up with after browsing through numerous StackOverflow topics and other sites.
git filter-branch --tree-filter "find . -type f -exec sed -Ei '' -e 's/(aSecretPassword1|aSecretPassword2|aSecretPassword3)/removedSensitiveInfo/g' {} \;"
When I run this command it seems to be rewriting the history (it shows the commits it's rewriting and takes a few minutes). However, when I check to see if all sensitive data has indeed been removed it turns out it's still there.
For reference this is how I do the check
git grep aSecretPassword1 $(git rev-list --all)
Which shows me all the hundreds of commits that match the search query. Nothing has been substituted.
Any idea what's going on here?
I double checked the regular expression I'm using which seems to be correct. I'm not sure what else to check for or how to properly debug this as my Git knowledge quite rudimentary. For example I don't know how to test whether 1) my regular expression isn't matching anything, 2) sed isn't being run on all files, 3) the file changes are not being saved, or 4) something else.
Any help is very much appreciated.
P.S.
I'm aware of several StackOverflow threads about this topic. However, I couldn't find one that is about substituting words (rather than deleting files) in all (ASCII) files (rather than specifying a specific file or file type). Not sure whether that should make a difference, but all suggested solutions haven't worked for me.
git-filter-branch is a powerful but difficult to use tool - there are several obscure things you need to know to use it correctly for your task, and each one is a possible cause for the problems you're seeing. So rather than immediately trying to debug them, let's take a step back and look at the original problem:
Substitute given strings (ie passwords) within all text files (without specifying a specific file/file-type)
Ensure that the updated Git history does not contain the old password text
Do the above as simply as possible
There is a tailor-made solution to this problem:
Use The BFG... not git-filter-branch
The BFG Repo-Cleaner is a simpler alternative to git-filter-branch specifically designed for removing passwords and other unwanted data from Git repository history.
Ways in which the BFG helps you in this situation:
The BFG is 10-720x faster
It automatically runs on all tags and references, unlike git-filter-branch - which only does that if you add the extraordinary --tag-name-filter cat -- --all command-line option (Note that the example command you gave in the Question DOES NOT have this, a possible cause of your problems)
The BFG doesn't generate any refs/original/ refs - so no need for you to perform an extra step to remove them
You can express you passwords as simple literal strings, without having to worry about getting regex-escaping right. The BFG can handle regex too, if you really need it.
Using the BFG
Carefully follow the usage steps - the core bit is just this command:
$ java -jar bfg.jar --replace-text replacements.txt my-repo.git
The replacements.txt file should contain all the substitutions you want to do, in a format like this (one entry per line - note the comments shouldn't be included):
PASSWORD1 # Replace literal string 'PASSWORD1' with '***REMOVED***' (default)
PASSWORD2==>examplePass # replace with 'examplePass' instead
PASSWORD3==> # replace with the empty string
regex:password=\w+==>password= # Replace, using a regex
Your entire repository history will be scanned, and all text files (under 1MB in size) will have the substitutions performed: any matching string (that isn't in your latest commit) will be replaced.
Full disclosure: I'm the author of the BFG Repo-Cleaner.
Looks OK. Remember that filter-branch retains the original commits under refs/original/, e.g.:
$ git commit -m 'add secret password, oops!'
[master edaf467] add secret password, oops!
1 file changed, 4 insertions(+)
create mode 100644 secret
$ git filter-branch --tree-filter "find . -type f -exec sed -Ei '' -e 's/(aSecretPassword1|aSecretPassword2|aSecretPassword3)/removedSensitiveInfo/g' {} \;"
Rewrite edaf467960ade97ea03162ec89f11cae7c256e3d (2/2)
Ref 'refs/heads/master' was rewritten
Then:
$ git grep aSecretPassword `git rev-list --all`
edaf467960ade97ea03162ec89f11cae7c256e3d:secret:aSecretPassword2
but:
$ git lola
* e530e69 (HEAD, master) add secret password, oops!
| * edaf467 (refs/original/refs/heads/master) add secret password, oops!
|/
* 7624023 Initial
(git lola is my alias for git log --graph --oneline --decorate --all). Yes, it's in there, but under the refs/original name space. Clear that out:
$ rm -rf .git/refs/original
$ git reflog expire --expire=now --all
$ git gc
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), done.
Total 6 (delta 0), reused 0 (delta 0)
and then:
$ git grep aSecretPassword `git rev-list --all`
$
(as always, run filter-branch on a copy of the repo Just In Case; and then removing original refs, expiring the reflog "now", and gc'ing, means stuff is Really Gone).