I have used the following yml command for my .Net 5 API project and xUnit Test Project but it throws error and my pipeline is not getting succeeded. Where did I go wrong?
Note: The pipeline is not getting succeded even if the task executed the test cases and showing 15 test cases passed and 2 test cases are filed.
- task: DotNetCoreCLI#2
inputs:
command: 'restore'
projects: '**/GeniusData.Test/GeniusData.Test.csproj'
displayName: 'Restore Projects'
- task: DotNetCoreCLI#2
inputs:
command: test
projects: '**/*Test/*.csproj'
arguments: '--configuration $(buildConfiguration) --collect "Code coverage"'
displayName: 'Test Project'
You're using the DotNetCoreCLI#2 task which will always fail when tests fail. That's by design: failing tests should break the build.
Related
We have a git repository that contains .Net code(backend) and since recently typescript code(angular, front-end).
When we added the angular test execution, it appears that the initial .net tests are not correctly published. From my test, it seems that only the last publish is conserved.
How can we keep both of them? It's important because it's used in a PR and we don't want ton miss anything.
Here is how we publish the .Net tests:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'NUnit'
testResultsFiles: '**/TEST-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/testResults'
failTaskOnFailedTests: true
testRunTitle: '.Net tests'
buildPlatform: 'Any CPU'
buildConfiguration: 'Debug'
and here how we publish the Angular tests:
- task: PublishTestResults#2
displayName: 'Publish Angular test results'
condition: succeededOrFailed()
inputs:
searchFolder: $(System.DefaultWorkingDirectory)/angular-test/results
testRunTitle: Angular
testResultsFormat: JUnit
testResultsFiles: '**/TESTS*.xml'
How can we ensure both tests results are considered in Azure Devop:
(here you see only the angular tests and its (few) tests. You also see there has been two tests run).
I taught about doing only once the PublishTestResults task, but since they have different format(.net is NUnit, while angular is JUnit), it will not work.
Gitlab Version: v14.1.1
Gitlab pipeline is succeeding even though there is a failed test cases in unit test.
Gitlab.yaml code:
unit-test:
stage: Test
script:
- npm run test
needs:
- lint
artifacts:
when: always
paths:
- coverage
reports:
junit:
- junit.xml
cobertura:
- coverage/cobertura-coverage.xml
expire_in: 4 days
only:
- test-case-testing
- merge_requests
Test Results:
Update: test command used in package.json
"test": "node ./node_modules/nyc/bin/nyc.js --reporter=cobertura --reporter=html node_modules/cucumber/bin/cucumber-js src/use-cases --parallel 5 --format=json --fail-fast --require \"src/use-cases/**/!(index).js\" | cucumber-junit > junit.xml",
How can I abort the Gitlab pipeline when there are any failed test cases? I read this but couldn't figure out what exact changes should I do?
Following change in script managed the failure.
stage: Test
script:
- npm run test
- test -f junit.xml && grep -L "<failure" junit.xml
and this resolved the issue.
I have been really getting frustrated with getting unit tests to run in a Azure Devops Pipeline
below is my YAML file contents
trigger:
- main
pool:
vmImage: 'windows-2019'
variables:
solution: '**/FestWise.stock*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
imageRepository: 'festwisestock'
dotNetCoreVrs: '3.1.x'
steps:
- task: UseDotNet#2
inputs:
version: '$(dotNetCoreVrs)'
packageType: 'sdk'
- task: NuGetCommand#2
inputs:
command: 'restore'
restoreSolution: '**/FestWise.stock*.sln'
feedsToUse: 'select'
- task: DotNetCoreCLI#2
displayName: 'Restore project dependencies'
inputs:
command: 'restore'
projects: '**/FestWise.Stock/FestWise.Stock.API/FestWise.StockService*.csproj'
- task: DotNetCoreCLI#2
displayName: 'Build the project - $(buildConfiguration)'
inputs:
command: 'build'
arguments: '--no-restore --configuration $(buildConfiguration)'
projects: '**/FestWise.Stock/FestWise.Stock.API/FestWise.StockService*.csproj'
- task: DotNetCoreCLI#2
displayName: 'Run unit tests - $(buildConfiguration)'
inputs:
command: 'test'
arguments: '--no-build --configuration $(buildConfiguration)'
publishTestResults: true
projects: '**/*StockService.UnitTests.csproj'
the result is as follows:
Starting: Run unit tests - Release
==============================================================================
Task : .NET Core
Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command
Version : 2.187.0
Author : Microsoft Corporation
Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
==============================================================================
C:\Windows\system32\chcp.com 65001
Active code page: 65001
Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
C:\hostedtoolcache\windows\dotnet\dotnet.exe test D:\a\1\s\FestWise.Stock\FestWise.StockService.UnitTests\FestWise.StockService.UnitTests.csproj --logger trx --results-directory D:\a\_temp --no-build --configuration Release
##[warning]No test result files were found.
Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://learn.microsoft.com/en-us/dotnet/core/tools/ and https://learn.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
Finishing: Run unit tests - Release
This is the furthest that i have come, it states it found the correct project but subsequently doesnt run any tests
So this was a drama to get figured out,
for some reason it didnt want to find the build sln from the previous step and thus the --no-build argument made it not work
removing that argument solved it for me
new complete yaml (with some improvements to make it more variable dependent)
trigger:
- main
pool:
vmImage: 'windows-2019'
variables:
solution: '**/FestWise.stock*.sln'
projectPath: '**/FestWise.Stock/FestWise.Stock.API/FestWise.StockService*.csproj'
testProjectPath: '**/*/FestWise.StockService.UnitTests.csproj'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
dotNetCoreVrs: '3.1.x'
steps:
- task: UseDotNet#2
inputs:
version: '$(dotNetCoreVrs)'
packageType: 'sdk'
- task: NuGetCommand#2
inputs:
command: 'restore'
restoreSolution: $(solution)
feedsToUse: 'select'
- task: DotNetCoreCLI#2
displayName: 'Restore project dependencies'
inputs:
command: 'restore'
projects: $(projectPath)
- task: DotNetCoreCLI#2
displayName: 'Build the project - $(buildConfiguration)'
inputs:
command: 'build'
arguments: '--no-restore --configuration $(buildConfiguration)'
projects: $(projectPath)
- task: DotNetCoreCLI#2
inputs:
command: "test"
includeNuGetOrg: true
projects: $(testProjectPath)
publishTestResults: true
displayName: Run the server-side tests
I use Azure DevOps Server 2020 with self hosted agents and created a CI pipeline which executes all tests in parallel on one agent. The ~5000 tests (no UI tests) need around 7min to complete. That's way to slow for our needs, so to speed things up I added 3 agents, put the VsTest task into another job in the same pipeline with parallel: 4. All 4 agents first download the build artifacts and run a slice of the tests. Unfortunately this actually made it worse. Now the test run needs around 8 min on each agent.
My vstest yaml for 1 agent
- task: VSTest#2
displayName: 'Run tests'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2:
**\*test*.dll
!**\*TestAdapter.dll
!**\*TestFramework.dll
!**\obj\**
searchFolder: '$(System.ArtifactsDirectory)'
runInParallel: true
codeCoverageEnabled: false
rerunFailedTests: false
My vstest yaml for 4 agents
- task: VSTest#2
displayName: 'Run tests'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2:
**\*test*.dll
!**\*TestAdapter.dll
!**\*TestFramework.dll
!**\obj\**
searchFolder: '$(System.ArtifactsDirectory)'
runInParallel: true
codeCoverageEnabled: false
distributionBatchType: 'basedOnExecutionTime'
rerunFailedTests: false
I even tried batching by assembly and based on number of tests + number of agents, test run time still sits at ~8min.
Comparing this to our old UI based CI pipeline, with multi-config and multiplier on a variable with 4 TestCategories, which runs even more tests ~10000 (including the 5000 of the new pipeline) but these are distributed by TestCategory on the same 4 agents (Cat1 on agent1, Cat2 on agent2 and so on), the agents average on ~5min each.
The yaml of the UI based one looks like this:
steps:
- task: VSTest#2
displayName: 'Run tests'
inputs:
searchFolder: '$(Build.BinariesDirectory)'
testFiltercriteria: 'TestCategory=$(Tests)'
runInParallel: true
codeCoverageEnabled: false
I think I have to be missing something obvious.
Thanks in advance!
Edit 1:
I connected to my agents with RDP and inside the task manager there are multiple instances of testhost.x86 running, up to 8 simultaneously but not constantly. If I start my tests locally the 8+ instances of testhost.x86 are up almost all the time and rarely vanish at all. If that's any help.
We have an application that has a number of projects isolated in their own solutions, each with their own UnitTest and IntegrationTest projects within those solutions. What happens on our locally hosted Azure DevOps applications is that the following code forces Azure DevOps to build each project in the solution before running tests. What I'd like to do is to run all tests sequentially on an initial build or at least cut the build time down because on the build server each build takes about a minute or 2 which is the bulk of the time. Since we have XUnit running the tests in say Rider it processes all tests across a solution from multiple projects well within a minute.
Is there a way to cut the build time or is this as good as it gets?
- task: DotNetCoreCLI#2
displayName: Unit Tests
inputs:
command: test
projects: '**/*UnitTest*/*.csproj'
arguments: '--configuration $(BuildConfiguration)'
# run integration tests
- task: DotNetCoreCLI#2
displayName: Integration Tests
inputs:
command: test
projects: '**/*IntegrationTest*/*.csproj'
arguments: '--configuration $(BuildConfiguration)'
What happens on our locally hosted azure devops application is that
the following code below will cause Azure Devops to build each project
in the solution before running tests.
For this issue , you can add --no-build argument to skip the project build on test run.
--no-build:
Doesn't build the test project before running it. This is listed in the Options part of document.
- task: DotNetCoreCLI#2
displayName: 'dotnet test'
inputs:
command: test
projects: '**/*UnitTest*/*.csproj'
arguments: '--configuration $(BuildConfiguration) --no-build'
Here is a case with similar issue , you can refer to it.