I am using Azure devops yml pipeline at code base.
I have created Variable Group at pipeline (pipeline > Library > variable group > called 'MY_VG')
In my pipeline (yml) file i want to send this variable group MY_VG to template my_template.yml as parameter.
But this parameter MY_VG is not being expanded when i use it under 'Variable' (though while printing it gives me value)
How to access the value of this MY_VG in the template here group: ${{parameters.variable_group}} shown below?
(I am calling a template file my_template_iterator.yml which iterates the environments and call the my_template.yml)
azure-pipelines.yml
resources:
repositories:
- repository: templates
type: git
name: MY_PROJECT/GIT_REPO_FOR_TEMPLATE
stages:
- stage: "CheckOut"
displayName: Checkout
jobs:
- job: Checkout
displayName: Checkout Application
pool:
name: $(my_pool_name)
workspace:
clean: all
steps:
- checkout: self
- template: folder_name/my_template_iterator.yml#templates
parameters:
agent_pool_name: $(my_pool)
db_resource_path: $(System.DefaultWorkingDirectory)/src/main/resources/db
envs:
- env:
env_name: 'dev'
variable_group: MY_VG
pipeline_environment_name: DEV_ENV
is_master: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
## Iterator: my_template_iterator.yml
parameters:
agent_pool_name: ''
db_resource_path: ''
envs: {}
stages:
- ${{ each env in parameters.envs }}:
- template: my_template.yml
parameters:
agent_pool_name: ${{ parameters.agent_pool_name }}
db_resource_path: ${{ parameters.db_resource_path }}
env_name: ${{ env.env_name }}
variable_group: ${{ env.variable_group }}
pipeline_environment_name: ${{ env.pipeline_environment_name }}
is_master: ${{ env.is_master }}
## my_template.yml
parameters:
- name: 'variable_group'
type: string
default: 'default_variable_group'
stages:
- stage:
displayName: Read Parameters
jobs:
- job: READ
displayName: Reading Parameters
steps:
- script: |
echo variable_group: ${{parameters.variable_group}}
- stage:
displayName: Deployment
variables:
group: ${{parameters.variable_group}}
condition: ${{parameters.is_master}}
I tested with above your yaml files. It seems fine. However i found a small mistake that you missed a - when you define a variable group in my_template.yml of yours. Maybe that is the reason the variable group didnot expand for you.
variables:
# a variable group
- group: myvariablegroup
I modified your my_template.yml file a little bit. And i got it working. See below:
parameters:
- name: 'variable_group'
type: string
default: 'default_variable_group'
- name: agent_pool_name
default: ""
- name: env_name
default: ""
- name: db_resource_path
default: ""
- name: pipeline_environment_name
default: ""
- name: is_master
default: ""
stages:
- stage:
displayName: ${{parameters.env_name}}
## i changed here add '-' to group
variables:
- group: ${{parameters.variable_group}}
jobs:
- job: READ
displayName: Reading Parameters
steps:
- script: |
echo variable_group: ${{parameters.variable_group}}
- powershell: echo "$(variableName-in-variablegroup)"
Ok. I'm not sure if this doable in a way how you are trying. But I tried another approach which seems to be working.
First let's define template for simple stage:
parameters:
- name: env_name
type: string
default: 'dev'
- name: variable_group
type: string
default: 'MY_VG'
- name: pipeline_environment_name
type: string
default: 'DEV_ENV'
stages:
- stage: ${{ parameters.env_name }}
jobs:
- job: angularinstall
steps:
- script: echo "${{ parameters.env_name }}"
- script: echo "${{ parameters.variable_group }}"
- script: echo "${{ parameters.pipeline_environment_name }}"
then template for main build:
parameters:
- name: stages # the name of the parameter is stageList
type: stageList # data type is stageList
default: [] # default value of stageList
stages: ${{ parameters.stages }}
and then we can combine them together in main build file:
extends:
template: template.yaml
parameters:
stages:
- template: stageTemplate.yaml
parameters:
env_name: 'dev'
variable_group: 'MY_VG'
pipeline_environment_name: 'DEV_ENV'
- template: stageTemplate.yaml
parameters:
env_name: 'stage'
variable_group: 'MY_STAGE'
pipeline_environment_name: 'STAGE_ENV'
And for that I got:
Does it sth what bring you closer to your solution?
Related
Is it possible to add some conditions to this way of using templates?
I would like to use conditions based on branches.
If branch = main or development > deploy to dev
Only if branch > main, deploy to staging
The complete Pipeline:
trigger:
batch: true
branches:
include:
- main
pool: 'On-Prem Pool'
variables:
- group: ContainerRegistry
- group:xyz
- name: appName
value: 'xyz'
- name: dockerfilePath
value: '**/Dockerfile'
stages:
# Build App and push to container registry
- stage: Build
displayName: Build and push to container registry
jobs:
- job: Build
displayName: Build job
pool: 'On-Prem Pool'
steps:
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: $(registryServiceConnection)
- task: Docker#2
displayName: Build container image
inputs:
command: build
repository: $(appName)
dockerfile: $(dockerfilePath)
containerRegistry: $(registryServiceConnection)
tags: $(Build.BuildId)
- task: Docker#2
displayName: Push container image
inputs:
command: push
repository: $(appName)
containerRegistry: $(registryServiceConnection)
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest')) # DonĀ“t push to ACR on Pull Requests
# Infrastructure
# Lint the Bicep file.
- stage: Lint
jobs:
- template: pipeline-templates/lint.yml
# Deploy to the dev environment.
- template: pipeline-templates/deploy.yml
parameters:
environmentType: Development
resourceGroupName: rg-xx
appSuffix: xyz
costCenter: 'xyz'
serviceConnectionName: 'SVCxx'
dockerRegistryPassword: $(dockerRegistryPassword)
dockerRegistryUserName: $(dockerRegistryUserName)
containerTag: $(Build.BuildId)
registryUrl: $(registryUrl)
cpuCores: 0.5
memory: 1
minReplicas: 0
maxReplicas: 1
# Deploy to the staging environment.
- template: pipeline-templates/deploy.yml
parameters:
environmentType: Staging
resourceGroupName: rg-xxx-001
appSuffix: xyz
costCenter: 'xyz'
serviceConnectionName: 'SVCxxx'
dockerRegistryPassword: $(dockerRegistryPassword)
dockerRegistryUserName: $(dockerRegistryUserName)
containerTag: $(Build.BuildId)
registryUrl: $(registryUrl)
cpuCores: 0.5
memory: 1
minReplicas: 1
maxReplicas: 2
)
And it is the last two -templates I would like to apply some conditions on.
Many thanks!
During my test, I edited my build.yaml as below. And when I ran it, the second template could evaluate correctly but the third one not, and thus, the third template was skipped.
pool:
vmImage: windows-latest
stages:
- template: template.yml
parameters:
env: dev
- ${{ if eq(variables['Build.QueuedBy'], 'Ceeno') }}:
- template: template.yml
parameters:
env: qa
- ${{ if eq(variables['Build.QueuedBy'], 'Vacee') }}:
- template: template.yml
parameters:
env: prod
And if you are looking for the evaluation for other method or other scenario, you could share with me for further investigations.
Yes I solved it in a similar way;
- stage: BuildDeployProd
displayName: Build and deploy Prod env
jobs:
- template: pipeline-templates/deploy.yml
parameters:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
I want to include a template, but it won't work. I know this is somehow a duplicate question, but the other answers don't solve my problem.
Template-File (base.yml)
trigger:
- master
pool: 'Etiscan Windows'
variables:
solution: '**/*.sln'
buildPlatform: 'x86'
buildConfiguration: 'Release'
steps:
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
command: 'restore'
restoreSolution: '**/*.sln'
feedsToUse: 'select'
vstsFeed: '76b8ae4e-2263-4849-bfb3-e1f621bf5bd7'
...to be continued....
Azure-Pipelines.yml
resources:
repositories:
- repository: templates
type: git
name: YML-Templates/NugetCfBuilderTemplate
stages:
- stage: Job1
- template: base.yml#templates
Errors
/base.yml#templates (Line: 6, Col: 1): Unexpected value 'trigger'
/base.yml#templates (Line: 9, Col: 1): Unexpected value 'pool'
/base.yml#templates (Line: 11, Col: 1): Unexpected value 'variables'
/base.yml#templates (Line: 16, Col: 1): Unexpected value 'steps'
In the resource, we can also add the trigger, refer doc. Help you modified the yaml, hope this will help:
resources:
repositories:
- repository: templates
type: git
name: YML-Templates/NugetCfBuilderTemplate
trigger:
branches:
include:
- master
stages:
- stage: Job1
- template: base.yml#templates
Note: please remove the trigger in your base.yml file.
Update:
I help you modified your base yaml and here I use the Powershell task to do the test:
stages:
- stage: job1
pool: 'default'
variables:
solution: '**/*.sln'
buildPlatform: 'x86'
buildConfiguration: 'Release'
jobs:
- job: A1
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World"
# - task: NuGetToolInstaller#1
# - task: NuGetCommand#2
# inputs:
# command: 'restore'
# restoreSolution: '**/*.sln'
# feedsToUse: 'select'
# vstsFeed: '76b8ae4e-2263-4849-bfb3-e1f621bf5bd7'
Note: Please remove the ' - stage: Job1' from your main yaml file.
Trigger can be defined only at the pipeline level, you cannot set it at the stage level. The stage can only have these children (see YAML schema reference):
stages:
- stage: string # name of the stage (A-Z, a-z, 0-9, and underscore)
displayName: string # friendly name to display in the UI
dependsOn: string | [ string ]
condition: string
variables: # several syntaxes, see specific section
jobs: [ job | templateReference]
You also need to insert a Job containing the steps and move the pool stanza down to Job level.
In my main pipeline in one stage, I call the same (deployment) template twice with just a bit different data:
//pipeline.yml
- stage: dev
condition: and(succeeded(), eq('${{ parameters.environment }}', 'dev'))
variables:
getCommitDate: $[ stageDependencies.prepare_date.set_date.outputs['setCommitDate.rollbackDate'] ]
jobs:
- template: mssql/jobs/liquibase.yml#templates
parameters:
command: update
username: $(username_dev)
password: $(password_dev)
environment: exampleEnv
databaseName: exampleDB
databaseIP: 123456789
context: dev
checkoutStep:
bash: git checkout ${{parameters.commitHash}} -- ./src/main/resources/objects
- template: mssql/jobs/liquibase.yml#templates
parameters:
command: rollbackToDate $(getCommitDate)
username: $(username_dev)
password: $(password_dev)
environment: exampleEnv
databaseName: exampleDB
databaseIP: 123456789
context: dev
//template.yml
parameters:
- name: command
type: string
- name: environment
type: string
- name: username
type: string
- name: password
type: string
- name: databaseName
type: string
- name: databaseIP
type: string
- name: context
type: string
- name: checkoutStep
type: step
default:
checkout: self
jobs:
- deployment: !MY PROBLEM!
pool:
name: exampleName
demands:
- agent.name -equals example
environment: ${{ parameters.environment }}
container: exampleContainer
strategy:
runOnce:
deploy:
steps:
...
My problem is that the deployment cannot have the same name twice.
It is not possible to use the ${{parameters.command}} to distinguish between deployments names, because it contains forbidden characters. Only ${{parameters.command}} differs between two calls.
My question is whether it is possible to distinguish the name of a deployment other way than passing another parameter (e.g. jobName: ). I have tried various conditions and predefined variables but without success.
Additionally, I should add DependsOn so that the second template is called for sure after the first.
It is not possible because getCommitDate and thus command parameter in your second stage contains runtime expression and job name needs compile time expression. So if you use command as job name at compile you have there rollbackToDate $(getCommitDate).
To solve this issue, the job identifier should be empty in a template:
- job: # Empty identifier
More informations available HERE
I am building a pipeline using a template (and nextly will be more pipelines using the same template). My problem is when I change something in the template the pipeline is executed, that means when there be more pipelines using the template, all will be executed.
Is there anyway to indicate that when template changes I don't need to execute pipeline automatically?
this is my template:
parameters:
- name: repoName # name of the parameter; required
type: string # data type of the parameter; required
default: ''
jobs:
- job: BuildImage
displayName: Build
pool:
name: Azure Pipelines
demands: java
vmImage: ubuntu-20.04
steps:
- checkout: ${{ parameters.repoName }}
clean: true
fetchDepth: 1
- task: Gradle#2
displayName: 'gradlew clean'
inputs:
workingDirectory: ./
tasks: clean
publishJUnitResults: false
jdkVersionOption: 1.11
- task: Gradle#2
displayName: 'gradlew build'
inputs:
options: '-x test'
publishJUnitResults: false
jdkVersionOption: 1.11
##Fix Jacoco
- task: Gradle#2
displayName: 'gradlew test'
inputs:
tasks: test
publishJUnitResults: false
codeCoverageToolOption: JaCoCo
jdkVersionOption: 1.11
## Add Sonar
and here the template is using it:
resources:
repositories:
- repository: repository
type: git
name: CORE_tpaas-utilities
ref: 'refs/heads/development'
extends:
template: ../templates/ci.yml
parameters:
repoName: repository
You can use the path filter for your pipeline trigger. For example add below to your pipeline yaml file.
resources:
....
trigger:
paths:
exclude:
- path/to/template.yml
extends:
....
After you specify above path filter to exclude the template yaml file for the pipeline trigger. The pipeline will not be executed when changes are made to the template file.
See here for more information.
Is it possible to nest yaml templates inside another yaml template ?
I've multiple NuGet projects in different Git repository, and I'm trying to template the process of publishing NuGets on nuget.org.
So I created a git repository named "devops-templates", did a first yaml template, be sure that it works, then divide it 4 yaml templates (build solution, generate packages, run unit tests, publish), and reference them into the global yaml template.
The problem is that when I tried to use this global template into my pipelines, I obtained errors
/Net/Utilities/BuildSolution.yml#templates (Line: 33, Col: 18): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 36, Col: 21): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 48, Col: 24): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 53, Col: 28): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 54, Col: 26): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 59, Col: 21): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 60, Col: 22): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 61, Col: 32): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 63, Col: 21): A template expression is not allowed in this context,/Net/Utilities/BuildSolution.yml#templates (Line: 64, Col: 26): A template expression is not allowed in this context
I searched inside Microsoft documentation : https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops but didn't find any information about it.
Here some parts of my code:
azure-pipelines.yml (main repository):
resources:
repositories:
- repository: templates
type: github
name: (...)/devops-templates
ref: refs/tags/v1.1.0
endpoint: (...)
stages:
- template: Net/Pipeline/NuGetsPipeline.yml#templates
parameters:
solution: $(solution)
nuGetsArtifactName: $(nuGetsArtifactName)
buildArtifactName : $(buildArtifactName)
(...)
NuGetsPipeline.yml (devops-templates repository):
parameters:
nuGetsArtifactName: 'NuGets'
buildArtifactName : 'Build'
nuGetSource: https://api.nuget.org/v3/index.json
solution: ''
(...)
stages:
- stage: Build
jobs:
- template: ${{variables['System.DefaultWorkingDirectory']}}/Net/Utilities/BuildSolution.yml
parameters:
buildArtifactName : ${{ parameters.buildArtifactName }}
(...)
- template: ${{variables['System.DefaultWorkingDirectory']}}/Net/Utilities/GenerateNuGets.yml
parameters:
nuGetsArtifactName: ${{ parameters.nuGetsArtifactName }}
buildArtifactName : ${{ parameters.buildArtifactName }}
(...)
- stage: 'UnitTests'
jobs:
- template: ${{variables['System.DefaultWorkingDirectory']}}/Net/Utilities/RunUnitTests.yml
parameters:
buildArtifactName : ${{ parameters.buildArtifactName }}
(...)
- stage: 'Publish'
jobs:
- template: ${{variables['System.DefaultWorkingDirectory']}}/Net/Utilities/PublishNuGets.yml
parameters:
nuGetsArtifactName: ${{ parameters.nuGetsArtifactName }}
(...)
BuildSolution.yml (devops-template repository):
parameters:
buildArtifactName: 'Build'
solution: ''
(...)
jobs:
- job: 'BuildSolution'
pool:
vmImage: ${{ parameters.vmImage }}
continueOnError: false
variables:
artifactName: ${{ parameters.buildArtifactName }}
steps:
- task: NuGetCommand#2
displayName: 'Restore NuGet packages'
inputs:
restoreSolution: ${{ parameters.solutionDir }}/${{ parameters.solution }}
configuration: ${{ parameters.buildConfiguration}}
- task: VSBuild#1
(...)
Edit : I added some parts of my code.
It seems like there is an indentation mistake in your BuildSolution.yml.
parameters and jobs should have the same indentation. refer to below:
parameters:
buildArtifactName: "build"
solution: ""
jobs:
- job: 'BuildSolution'
pool:
vmImage: ${{parameters.vmImage}}
continueOnError: false
variables:
artifactName: ${{ parameters.buildArtifactName}}
steps:
- task: NuGetCommand#2
displayName: 'Restore NuGet packages'
inputs:
restoreSolution: ${{ parameters.solutionDir }}/${{ parameters.solution }}
configuration: ${{parameters.buildConfiguration}}
You can reference four kinds of templates within Azure Pipelines: Stage, Job, Step and Variable.
An example (with slightly modified comments by me) taken from the "Template References" documentation is as follows:
# File: azure-pipelines.yml which references another YAML file (test.yml)
stages:
- template: stages/test.yml # Template reference
parameters:
name: Mini
testFile: tests/miniSuite.js
- template: stages/test.yml # Template reference
parameters:
name: Full
testFile: tests/fullSuite.js
And the test.yml file is as follows:
# File: test.yml file to be referenced by the azure-pipelines.yml file
parameters:
name: ''
testFile: ''
stages:
- stage: Test_${{ parameters.name }}
jobs:
- job: ${{ parameters.name }}_Windows
pool:
vmImage: vs2017-win2016
steps:
- script: npm install
- script: npm test -- --file=${{ parameters.testFile }}
- job: ${{ parameters.name }}_Mac
pool:
vmImage: macos-10.13
steps:
- script: npm install
- script: npm test -- --file=${{ parameters.testFile }}