We have a git repository that contains .Net code(backend) and since recently typescript code(angular, front-end).
When we added the angular test execution, it appears that the initial .net tests are not correctly published. From my test, it seems that only the last publish is conserved.
How can we keep both of them? It's important because it's used in a PR and we don't want ton miss anything.
Here is how we publish the .Net tests:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'NUnit'
testResultsFiles: '**/TEST-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/testResults'
failTaskOnFailedTests: true
testRunTitle: '.Net tests'
buildPlatform: 'Any CPU'
buildConfiguration: 'Debug'
and here how we publish the Angular tests:
- task: PublishTestResults#2
displayName: 'Publish Angular test results'
condition: succeededOrFailed()
inputs:
searchFolder: $(System.DefaultWorkingDirectory)/angular-test/results
testRunTitle: Angular
testResultsFormat: JUnit
testResultsFiles: '**/TESTS*.xml'
How can we ensure both tests results are considered in Azure Devop:
(here you see only the angular tests and its (few) tests. You also see there has been two tests run).
I taught about doing only once the PublishTestResults task, but since they have different format(.net is NUnit, while angular is JUnit), it will not work.
Related
My team has a gitlab repo. It has two parts: an NPM package under projects folder and an angular application under src folder. So there are 2 projects in the angular.json file.
We currently have unit tests with coverage setup in our gitlab pipes. The issue is, since we have 2 projects in this repo, we really need to show the coverage for each project.
I noticed in demo image of the gitlab badges documentation (https://docs.gitlab.com/ee/user/project/badges.html), they have a 'JS Coverage' badge. This seems to be a custom badge (I can't find a list of given badges, but I'm not finding anything for 'JS coverage', so I'm assuming it's custom).
So I think I can do something like that to create 2 custom badges that has the code coverage of each project (1 for 'Pkg Coverage' and 1 for 'App Coverage'). But (TBH) the documentation around creating custom badges isn't great. I need to know how to store this custom value to use in the badge, and how to update in the gitlab pipe.
Does anyone know how to achieve this? If I could just figure out how that example is using 'JS Coverage' (and how to update the value in the pipe), then I could figure out what I need to do for my 2 custom badges. Any tips?
Some details, right now we have a gitlab job like this (it runs unit tests and updates the coverage values. Since 'ng test' runs the tests of both projects 1 by 1, the code coverage of the 1st project is saved to the 'coverage' value):
unit-tests:
stage: test
rules:
# Run unit tests, including when merge requests are merged to default branch (so coverage % is updated)
- when: on_success
image: trion/ng-cli-karma:$ANGULAR_VERSION
before_script:
- *angular-env-setup-script
coverage: '/Statements \W+: (\d+\.\d+)%.*/'
script:
- npm run build:ds-prod
- npm install dist/ds
- ng test --code-coverage --progress false --watch false
artifacts:
expose_as: "Coverage Report"
paths:
- coverage/
tags:
- kubernetes-runner
I have used the following yml command for my .Net 5 API project and xUnit Test Project but it throws error and my pipeline is not getting succeeded. Where did I go wrong?
Note: The pipeline is not getting succeded even if the task executed the test cases and showing 15 test cases passed and 2 test cases are filed.
- task: DotNetCoreCLI#2
inputs:
command: 'restore'
projects: '**/GeniusData.Test/GeniusData.Test.csproj'
displayName: 'Restore Projects'
- task: DotNetCoreCLI#2
inputs:
command: test
projects: '**/*Test/*.csproj'
arguments: '--configuration $(buildConfiguration) --collect "Code coverage"'
displayName: 'Test Project'
You're using the DotNetCoreCLI#2 task which will always fail when tests fail. That's by design: failing tests should break the build.
I use Azure DevOps Server 2020 with self hosted agents and created a CI pipeline which executes all tests in parallel on one agent. The ~5000 tests (no UI tests) need around 7min to complete. That's way to slow for our needs, so to speed things up I added 3 agents, put the VsTest task into another job in the same pipeline with parallel: 4. All 4 agents first download the build artifacts and run a slice of the tests. Unfortunately this actually made it worse. Now the test run needs around 8 min on each agent.
My vstest yaml for 1 agent
- task: VSTest#2
displayName: 'Run tests'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2:
**\*test*.dll
!**\*TestAdapter.dll
!**\*TestFramework.dll
!**\obj\**
searchFolder: '$(System.ArtifactsDirectory)'
runInParallel: true
codeCoverageEnabled: false
rerunFailedTests: false
My vstest yaml for 4 agents
- task: VSTest#2
displayName: 'Run tests'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2:
**\*test*.dll
!**\*TestAdapter.dll
!**\*TestFramework.dll
!**\obj\**
searchFolder: '$(System.ArtifactsDirectory)'
runInParallel: true
codeCoverageEnabled: false
distributionBatchType: 'basedOnExecutionTime'
rerunFailedTests: false
I even tried batching by assembly and based on number of tests + number of agents, test run time still sits at ~8min.
Comparing this to our old UI based CI pipeline, with multi-config and multiplier on a variable with 4 TestCategories, which runs even more tests ~10000 (including the 5000 of the new pipeline) but these are distributed by TestCategory on the same 4 agents (Cat1 on agent1, Cat2 on agent2 and so on), the agents average on ~5min each.
The yaml of the UI based one looks like this:
steps:
- task: VSTest#2
displayName: 'Run tests'
inputs:
searchFolder: '$(Build.BinariesDirectory)'
testFiltercriteria: 'TestCategory=$(Tests)'
runInParallel: true
codeCoverageEnabled: false
I think I have to be missing something obvious.
Thanks in advance!
Edit 1:
I connected to my agents with RDP and inside the task manager there are multiple instances of testhost.x86 running, up to 8 simultaneously but not constantly. If I start my tests locally the 8+ instances of testhost.x86 are up almost all the time and rarely vanish at all. If that's any help.
We have an application that has a number of projects isolated in their own solutions, each with their own UnitTest and IntegrationTest projects within those solutions. What happens on our locally hosted Azure DevOps applications is that the following code forces Azure DevOps to build each project in the solution before running tests. What I'd like to do is to run all tests sequentially on an initial build or at least cut the build time down because on the build server each build takes about a minute or 2 which is the bulk of the time. Since we have XUnit running the tests in say Rider it processes all tests across a solution from multiple projects well within a minute.
Is there a way to cut the build time or is this as good as it gets?
- task: DotNetCoreCLI#2
displayName: Unit Tests
inputs:
command: test
projects: '**/*UnitTest*/*.csproj'
arguments: '--configuration $(BuildConfiguration)'
# run integration tests
- task: DotNetCoreCLI#2
displayName: Integration Tests
inputs:
command: test
projects: '**/*IntegrationTest*/*.csproj'
arguments: '--configuration $(BuildConfiguration)'
What happens on our locally hosted azure devops application is that
the following code below will cause Azure Devops to build each project
in the solution before running tests.
For this issue , you can add --no-build argument to skip the project build on test run.
--no-build:
Doesn't build the test project before running it. This is listed in the Options part of document.
- task: DotNetCoreCLI#2
displayName: 'dotnet test'
inputs:
command: test
projects: '**/*UnitTest*/*.csproj'
arguments: '--configuration $(BuildConfiguration) --no-build'
Here is a case with similar issue , you can refer to it.
I'm trying to setup code coverage for a Java application project.
Project name : NewApp
Project structure:
src/java/** (source code)
src/java-test (unit tests - Jnuit)
test/it-test (integration test)
test/at-tests (acceptance tests)
tomcat/* (contain tomcat start/stop scripts)
xx/.. etc folders which are required for a usual application.
Gradle version : 1.6
Environment : Linux
I have a running gradle build script that fetches application (NewApp) dependencies (i.e. service jars used by the app for build process) from a build artifact repository (artifactory/maven for ex), and builds the app.
Now at this point, I wanted to get code coverage using JaCoCo plugin for my NewApp application project.
I followed the documentation per Gradle/Jacoco but it doesn't seems to create any reports/... folder for jacoco etc where I can find what Jacoco coverage report did.
My questions:
1. For getting code coverage using Unit tests (Junit), I assume all I have to do is the following and it will NOT require me to start/stop the tomcat before running unit test (test task i.e. "gradle test") to get code coverage for/via using unit tests. Please advise/correct. The code (just for Gradle jacoco unit test part) - I'm using is:
apply plugin: 'jacoco'
test {
include 'src/java-test/**'
}
jacocoTestReport {
group = "reporting"
description = "Generate Jacoco coverage reports after running tests."
reports {
xml.enabled true
html.enabled true
csv.enabled false
}
//classDirectories = fileTree(dir: 'build/classes/main', include: 'com/thc/**')
//sourceDirectories = fileTree(dir: 'scr/java', include: 'com/thc/**')
additionalSourceDirs = files(sourceSets.main.allJava.srcDirs)
}
and for Integration tests:
task integrationTest(type: Test) {
include 'test/java/**'
}
As jacocoTestReport is depends upon test task(s), thus they will be called first and then finally jacocoTestReport will report what it found for the code coverage.
For getting code coverage for integration tests, I assume I must start tomcat first (i.e. before running / calling test target for integration tests), then call "gradle integrationTest" or "gradle test" task and then stop tomcat -- to get the code coverage report. From other blog posts I also found that one should setup JAVA_OPTS variable to assign jacoco agent before tomcat starts.
for ex: setting JAVA_OPTS variable like:
export JACOCO="-Xms256m -Xmx512m -XX:MaxPermSize=1024m -javaagent:/production/jenkinsAKS/jobs/NewApp/workspace/jacoco-0.6.3.201306030806/lib/jacocoagent.jar=destfile=/production/jenkinsAKS/jobs/NewApp/workspace/jacoco/jacoco.exec,append=true,includes=*"
export JAVA_OPTS="$JAVA_OPTS $JACOCO"
Being new to Gradle/groovy - I'm not sure what code should I write within build.gradle (build script) to get the above Integration/Unit tests working if it involves start/stop of tomcat. If someone can provide a sample script to do that, I'll try.
I'm not getting any code coverage right now, when I publish Jacoco code coverage in Jenkins (using Jenkins post build action for publishing Jacoco reports). Jenkins build dashboard shows 0% for code coverage (i.e. bars showing all red color, no green for actual code coverage).
Need your advice to get some traction on this.
Question : I assume that your unit tests doesn't depend on tomcat. In this case, you're right, you must not start tomcat upfront.
To create the coverage report you need to execute
gradle jacocoTestReport
without jacocoTestReport gradle won't trigger jacoco to generate the reports.
One additional thing, regarding to your snippet. I assume that you have changed the the default main sourceset to source/java. in this case you don't have to set the additionalSourceDirs.
Integration tests : Yes, you need to start tomcat first, or at least you have to ensure that tomcat is running. You should have a look into Gradle 1.7. It has a new task ordering rule called finalizedBy
With this you could do something like
task integrationtests(type: Test) {
dependsOn startTomcat
finalizedBy stopTomcat
}
where start/stopTomcat are custom tasks.If you have to stay on Gradle 1.6 you have to build a dependsOn chain:
stopTomcat -dependsOn-> integrationtests -dependsOn-> startTomcat
I assume that the blog article is right, I don't have any experience with that.
Starting/Stoping Tomcat : You could do it in a way like this
task startTomcat() << {
def tomcatStartScript = "${project.rootDir}/tomcat/startScript"
tomcatStartScript.execute()
}
The stop script can be written in a similiar way. (Some in from Groovy doc : Executing)
Jenkins & Jacoco : Should be fixed when executing jacocoTestReport
Got it working.
Gradle 1.7
- download the .zip which contains the binaries/src and documentation.
- Go to folder: if you unzip the above .zip at C:\gradle-1.7
C:\gradle-1.7\samples\testing\jacoco\quickstart
Run:
gradle build jacocoTestReport
You’ll see a new folder “build” after the build.
– folder jacoco gets created with classdumps and .exec if only build task is called.
– folder jacoco and jacocoHtml gets created – if both build jacocoTestReport is called
have fun.
I also saw that it’s better to include:
the following section in build.gradle
/////
tasks.withType(Compile) {
options.debug = true
options.compilerArgs = ["-g"]
}
////