I'm trying to run a dataflow job using cloud build
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args:
dataflow
jobs
run
google-template-job
--gcs-location=gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery
--parameters=inputSubscription='projects/$PROJECT_ID/subscriptions/messages'
--parameters=outputTableSpec="$PROJECT_ID:beam_samples.streaming_beam"
--staging-location=gs://cloudbuild-dataflow-testproject123456789-313307/tmp'
--region=us-central1
Every time I trigger the build I get the following error
ERROR: (gcloud.dataflow.jobs.run) INVALID_ARGUMENT: The template parameters are invalid.
- '#type': type.googleapis.com/google.dataflow.v1beta3.InvalidTemplateParameters
parameterViolations:
- description: 'Unmatched regex: ^projects\/[^\n\r\/]+\/subscriptions\/[^\n\r\/]+$'
parameter: inputSubscription
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcloud" failed: step exited with non-zero status: 1
My project id has a ' - ' in it so if I replace the $PROJECT_ID with the value of project id, I still get the same error, is there any workaround for this. I don't have any control over stopping the regex check since it's a google provided template.
How do I get past this
Got it. It's only a command interpreter issue. If you put single quote, you prevent any evaluation of the inside string.
In you case
--parameters=inputSubscription='projects/$PROJECT_ID/subscriptions/messages'
the value 'projects/$PROJECT_ID/subscriptions/messages' is taken as is and therefore the project ID contain uppercase and underscore, that violate the regex pattern.
Change for double quote and it should work great!
--parameters=inputSubscription="projects/$PROJECT_ID/subscriptions/messages"
Related
I have the below step function and I'm trying to pass the ARN to the StartProjectVersion step in the "test" variable using "ProjectVersionArn": "$.test". Every time I execute the step functions, StartProjectVersion fails with:
2 validation errors detected: Value '$.test' at 'projectVersionArn'
failed to satisfy constraint: Member must satisfy regular expression
pattern: (^arn:[a-z\d-]+:rekognition
How can I set the ARN using $.test? Thank you.
Parameters: For key-value pairs where the value is selected using a path, the key name must end in .$.
"ProjectVersionArn.$": "$.test"
I'm trying to replace a character in a variable within a GITHUB actions step
- name: Set Up DB Name
run: |
DB_NAME="${GITHUB_REF_SLUG/-/_}"
echo $DB_NAME
I'm getting a bad request error
What am I doing wrong?
I successfully made the character replace works (with GITHUB_REPOSITORY) using this implementation:
job1:
runs-on: ubuntu-latest
steps:
- name: character-replacement-test
run: |
REPO=$GITHUB_REPOSITORY
DB_NAME="${REPO//-/_}"
echo $DB_NAME
I couldn't get to the same result with 2 lines.
(But someone more experienced with bash might help us get there as well).
Evidence:
So in your case, it should work using this code if you substitute the GITHUB_REPOSITORY by GITHUB_REF_SLUG in your workflow.
I used this post as reference.
I would like to setup my project_dev CI only for 3 branches and specific kind of tags like: dev_1.0, dev_1.1, dev_1.2.
How can I achieve that?
This is what I have now:
project_dev:
stage: dev
script:
- export
- bundle exec pod repo update
- bundle exec pod install
- bundle exec fastlane crashlytics_project_dev
after_script:
- rm -rf ~/Library/Developer/Xcode/Archives || true
when: manual
only:
- develop
- release
- master
- //here I need to add condition to fire that stage additionally only for specific tags. How can I setup regexp here?
tags:
- iOS
When I type it like:
only:
- branches
- /^dev_[0-9.]*$/
It also runs the CI for tags like: dev1.2 but it should not. Why? Is there a regexp for tags at all?
Sounds like a regular expression question. I just created a project on gitlab.com for the regular expression.
File: .gitlab-ci.yml
project_dev:
# Irrelevant keys is skipped
script:
- echo "Hello World"
only:
- develop
- release
- master
- /^dev_[0-9]+(?:.[0-9]+)+$/ # regular expression
I was pushed all of tags you mentioned to test this regular expression.
As you can see , It will match tags like dev_1.0, dev_1.1, but the job project_dev will not be triggered by tag dev1.2, You can check the result on pipeline pages
Instead of using only/except you can use rules which are more powerful.
Rules support regex pattern matching.
Your rule for excepting only specific kind of branches/tags like dev_1.0, dev_1.1, dev_1.2 should look like:
rules:
- if: '$CI_COMMIT_BRANCH =~ /^dev_[0-9]+\.[0-9]+$/ || $CI_COMMIT_TAG =~ /^dev_[0-9]+\.[0-9]+$/'
Predefined environment variables like CI_COMMIT_BRANCH and CI_COMMIT_TAG are described here.
Gitlab.com ?
You could try a combination of except and only.
Something like
only:
- tags
- branches
except:
- /^(?!(branch1|branch2|branch3|dev_[0-9.]*$)$).*$/
The idea being, allowing only branches and tags to trigger a job, with the exception of everything different from branch[1-3] and dev_ branches/tags
And here is the official documentation for this:
GitLab CI/CD pipeline configuration reference
There you find the section for only/except with the supported regex syntax, although it states that:
only and except are not being actively developed. rules is the preferred keyword to control when to add jobs to pipelines.
We want to migrate from Bitbucket Pipelines to Google Cloud Build to test, build and push Docker images.
How can we use environment variables without a CryptoKey? For example:
- printf "https://registry.npmjs.org/:_authToken=${NPM_TOKEN}\nregistry=https://registry.npmjs.org" > ~/.npmrc
To use environment variables in the args portion of your build steps you need:
"a shell to resolve environment variables with $$" (as mentioned in the example code here)
and you also need to be careful with your usage of quotes (use single quotes)
See below the break for a more detailed explanation of these two points.
While the Using encrypted resources docs that David Bendory also linked to (and which you probably based your assumption on) show how to do this using an encrypted environment variable specified via secretEnv, this is not a requirement and it works with normal environment variables too.
In your specific case you'll need to modify your build step to look something like this:
# you didn't show us which builder you're using - this is just one example of
# how you can get a shell using one of the supported builder images
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: ['-c', 'printf "https://registry.npmjs.org/:_authToken=%s\nregistry=https://registry.npmjs.org" $$NPM_TOKEN > ~/.npmrc']
Note the usage of %s in the string to be formatted and how the environment variable is passed as an argument to printf. I'm not aware of a way that you can include an environment variable value directly in the format string.
Alternatively you could use echo as follows:
args: ['-c', 'echo "https://registry.npmjs.org/:_authToken=$${NPM_TOKEN}\nregistry=https://registry.npmjs.org" > ~/.npmrc']
Detailed explanation:
My first point at the top can actually be split in two:
you need a shell to resolve environment variables, and
you need to escape the $ character so that Cloud Build doesn't try to perform a substitution here
If you don't do 2. your build will fail with an error like: Error merging substitutions and validating build: Error validating build: key in the template "NPM_TOKEN" is not a valid built-in substitution
You should read through the Substituting variable values docs and make sure that you understand how that works. Then you need to realise that you are not performing a substitution here, at least not a Cloud Build substitution. You're asking the shell to perform a substitution.
In that context, 2. is actually the only useful piece of information that you'll get from the Substituting variable values docs (that $$ evaluates to the literal character $).
My second point at the top may be obvious if you're used to working with the shell a lot. The reason for needing to use single quotes is well explained by these two questions. Basically: "You need to use single quotes to prevent interpolation happening in your calling shell."
That sounds like you want to use Encrypted Secrets: https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials
I'm trying to run a Jenkins build step based on a conditional regular expression match and I can't figure out why it doesn't match.
Here's how it looks in Jenkins:
I'm using this regular expression: .*iOS. against a build parameter, pmt_content.
Here's the pmt_content value:
<body><pre>^1234567890^abc123def132afd1213afas^iOS^Test User^test#abc.com^iPad 3^</pre><img src='https://cirrus.app47.com/notifications/5626a0bc9ac25b6ea7003f2f/img' alt=''/></body>
Here's the console log telling me it did not match.
[PollMailboxTrigger] An email matching the filter criteria was found. (log)
Building in workspace /Users/jenkins/.jenkins/jobs/New Device Listener/workspace
Run condition [Regular expression match] enabling prebuild for step [Execute shell]
Regular expression run condition: Expression=[.*iOS.*], Label=[
<body><pre>^1234567890^abc123def132afd1213afas^iOS^Test User^test#abc.com^iPad 3^</pre><img src='https://cirrus.app47.com/notifications/5626a0bc9ac25b6ea7003f2f/img' alt=''/></body>]
Run condition [Regular expression match] preventing perform for step [Execute shell]
Finished: SUCCESS
Clearly pmt_content contains "iOS".
I tested the same values at www.regexplanet.com.
What am I doing wrong?
I experienced similar frustrations due to a newline character in my label data which prevented a match. Hard to tell from your dump if there might be a newline in there at the start. In any event you might try (?is).*iOS.*. Reference.