Spockframework soft assertion describes only first fail - unit-testing

Spockframework offers soft assertions mechanism, but it seems it doesn't work properly (at least in my configuration).
I have created simplest test:
verifyAll {
1 == 2
2 == 3
}
I expect to see two failure message, but I see only first one:
Condition not satisfied:
1 == 2
|
false
Second one is also executed, but it appears only in gradle report:
org.example.SoftAssertionsTest > simplest test FAILED
org.spockframework.runtime.SpockComparisonFailure at SoftAssertionsTest.groovy:9
org.spockframework.runtime.SpockComparisonFailure at SoftAssertionsTest.groovy:10
Versions:
compile 'org.codehaus.groovy:groovy-all:2.5.8'
testCompile group: 'junit', name: 'junit', version: '4.12'
testImplementation("org.springframework.boot:spring-boot-starter-test:2.+")
testImplementation('org.spockframework:spock-spring:1.3-groovy-2.5')
I have pushed this example to: https://github.com/fergus-macdubh/spock-soft-assertions
Is there a way to make it show ALL messages?

Okay, I saw similar behaviour with an old 4.4.1 Gradle version installed locally. But as soon as I added gradlew[.bat] files to your project and modified your build file to be more similar to the Spock example project, it worked.
apply plugin: "groovy"
group = 'org.example'
version = '1.0-SNAPSHOT'
description = "Soft Assertions"
// Spock works with Java 1.8 and above
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
// mandatory dependencies for using Spock
compile "org.codehaus.groovy:groovy-all:2.5.8"
testCompile platform("org.spockframework:spock-bom:2.0-M1-groovy-2.5")
testCompile "org.spockframework:spock-core"
testCompile "org.spockframework:spock-junit4" // you can remove this if your code does not rely on old JUnit 4 rules
// optional dependencies for using Spock
testCompile "org.hamcrest:hamcrest-core:1.3" // only necessary if Hamcrest matchers are used
testRuntime "net.bytebuddy:byte-buddy:1.9.3" // allows mocking of classes (in addition to interfaces)
testRuntime "org.objenesis:objenesis:2.6" // allows mocking of classes without default constructor (together with CGLIB)
// dependencies used by examples in this project
testImplementation("org.springframework.boot:spring-boot-starter-test:2.+")
testImplementation('org.spockframework:spock-spring:1.3-groovy-2.5')
}
test {
useJUnitPlatform()
}
The console log now looks as expected:
Testing started at 12:57 ...
> Task :compileJava NO-SOURCE
(...)
> Task :test
Multiple Failures (2 failures)
org.spockframework.runtime.SpockComparisonFailure: Condition not satisfied:
1 == 2
|
false
org.spockframework.runtime.SpockComparisonFailure: Condition not satisfied:
2 == 3
|
false
(...)

Related

Sonarqube gives 0% coverage on spring boot gradle project

So I am working with SonarQube and I keep getting stuck. My coverage is 0% while it should not be 0% (I made sure with an simple test that always should run correctly and covers at least one method).
I am reading a lot of different gradle.build files on the internet, but I can't find one that works for me.
Also I think the problem is that sonarqube can't find certain files, but I can't find a working directory of someone which I can compare with my own.
Gradle.build
plugins {
id 'org.springframework.boot' version '2.4.2'
id 'io.spring.dependency-management' version '1.0.11.RELEASE'
id 'java'
id "org.sonarqube" version "3.0"
id 'jacoco'
}
group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '15'
configurations {
compileOnly {
extendsFrom annotationProcessor
}
}
repositories {
mavenCentral()
}
dependencies {
compile 'org.apache.httpcomponents:httpcore:4.4.1'
compile 'org.apache.httpcomponents:httpclient:4.5'
implementation('io.jsonwebtoken:jjwt:0.2')
implementation 'org.springframework.boot:spring-boot-starter-mail'
implementation 'org.springframework.boot:spring-boot-starter-security'
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
implementation 'org.springframework.boot:spring-boot-starter-web'
compile 'junit:junit:4.12'
implementation 'org.modelmapper:modelmapper:2.4.1'
compileOnly 'org.projectlombok:lombok'
runtimeOnly 'mysql:mysql-connector-java'
annotationProcessor 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
implementation 'org.eclipse.jgit:org.eclipse.jgit:5.4.2.201908231537-r'
/**
* JUnit jupiter with mockito.
*/
testCompile group: 'org.mockito', name: 'mockito-junit-jupiter', version: '2.19.0'
testCompile group: 'org.mockito', name: 'mockito-core', version: '2.19.0'
testCompile group: 'org.springframework.security', name: 'spring-security-test', version: '5.1.6.RELEASE'
}
sonarqube{
properties{
property 'sonarjava.source', '1.8'
property 'sonar.java.coveragePlugin', 'jacoco'
}
}
test {
useJUnitPlatform()
}
This is the folder where my test reports are in (I think you will need this to help me):
The build directory:
The jacoco test directory
Can anyone help me with my problem? If you want to see directories or other files, just ask and add them to the post.
Thanks in advance!
you need to make sure jacoco runs before sonarqube, so add something like:
tasks["sonarqube"].dependsOn("jacocoTestReport")
Also, side note, compile is deprecated, so use implementation and testImplementation instead.
The answer was in a directory that I had not shown. My project was missing the Jacocoreportxml file. Once I added this with the gradle build jacocoreportxml command in my terminal it worked!

Can we run Spock Testcases and Junit 5 test cases together In one project?

We are not able to run test cases written with Junit 5 and Spock framework together in one gardle project?
We tried add dependencies given in https://www.baeldung.com/junit-5-gradle to our gradle file. Gradle version is 4.10.3 and Junit 5. Below is my build.gradle file
apply plugin: 'groovy'
apply plugin: 'java'
repositories {
mavenCentral()
maven {
url "http://repo.fusesource.com/nexus/content/groups/public/"
}
maven {
url "https://repository.jboss.org/nexus/content/groups/public"
}
jcenter()
}
dependencies {
compile group: 'com.google.inject', name: 'guice', version: '4.2.2'
compile group: 'javax.servlet', name: 'javax.servlet-api', version: '3.0.1'
testCompile(
'org.codehaus.groovy:groovy-all:2.4.8',
'org.spockframework:spock-core:1.0-groovy-2.4',
'org.jmockit:jmockit:1.8',
'junit:junit:4.12'
)
testRuntime(
'cglib:cglib:2.2.2',
'com.athaydes:spock-reports:1.2.7'
)
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.3.1'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.3.1'
testCompileOnly 'junit:junit:4.12'
}
test {
useJUnitPlatform()
testLogging { showStandardStreams = true }
}
I have created two test cases, one is using spock framework and other is using junit 5. But when I do gradlew -test, it runs only test cases written with Junit 5. Below is build path.
You need the Vintage test engine to execute Spock tests since they are based on JUnit 4, and you need the Jupiter test engine to execute the JUnit Jupiter tests.
So you need dependencies on both engines.
testRuntimeOnly 'org.junit.vintage:junit-vintage-engine:5.3.1'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.3.1'
I'd also recommend that you upgrade to JUnit 5.5.1 (i.e., the latest and greatest).

TeamCity does not catch failing tests in minitest

I have TeamCity up and running in a mac environment. A trigger is also setup to to run a rake task containing a number of test cases on every git commit. The rake command successfully pulls the code and runs the test cases. The passing test cases are shown correctly in the TeamCity interface, however, the failing test cases do not show up and the build is marked as successful. When I look at the error Build Log, failing tests are shown as errors. The question is why TeamCity is interpreting failing tests as errors and not as actual failing tests?
The assertions are as simple as the following (obvious failure):
test 'simple_test' do
a = 14
ssert_equal 341, a
ends
PS: the check box to fail the build upon at least one test fail is turned on in Failure Conditions(i.e. at least one test failed)
Ruby version = 2.2.2,
Rails version = 4.2.1,
TeamCity version = 9.1.1,
Testing framework = minitest,
Here is the build log output
Teamcity is not picking up failed tests because mini-test reporter is not up and running. To fix this, following gems must be present in the Gemfile:
gem "minitest", :group => :test
gem 'minitest-reporters', :group => :test
Also minitest reporter must be called. The best place to call it is test_helper.rb
require 'minitest/reporters'
MiniTest::Reporters.use!

Gradle test task - Gradle 1.6 and with Gradle 2.3 or later

I have a Java project.
PS: In my project, I don't have any java program/source code in src/test/java. - This folder just contains a blank.txt file.
I have two different Gradle versions:
Gradle 1.6 with Java 7 (as Java 8 is NOT compatible with Gradle 1.6 or any version < less than 1.10 version if I'm correct).
The other version is: Gradle 2.3 with Java 8.
Using both of the above mentioned Gradle 1.6 + Java7 OR Gradle 2.3 + Java 8 versions my project build successfully.
Though, I noticed one thing: That while running the build, it calls "test" task automatically (as per the Gradle design, test task runs for free); I found during Gradle 1.6 + Java7 run --- I see the following output.
:jar
:assemble
:compileTestJava UP-TO-DATE
:processTestResources
:testClasses
:test
:check
As you'll notice, it says I don't have any test source code (i.e. src/test/java doesn't contain any source code OR there's nothing new for Gradle to compile this time may be nothing changed since last time gradle ran the build) and that's why compileTestJava task is showing UP-TO-DATE in front of it.
But, :test task is showing that it ran successfully. I have used jacoco (code coverage) section within test { .. } task, then it actually ran that part (as there is no UP-TO-DATE in front of test task). Jacoco section is NOT defined in my project's build.gradle but actually it's coming from a top level / GRADLE_HOME/init.d/some-common-top-level.gradle file (where test { ... has jacoco { ... } .. } section in it).
As I mentioned above, test task didn't say UP-TO-DATE, therefore, after Gradle build process was complete, I can see it created the following folder/files structure inside build/tmp/expandedArchives/org.jacoco.... folder:
$ ls -ltr build/tmp/expandedArchives/
total 4
drwxr-xr-x+ 1 e020001 Domain Users 0 Jul 7 20:45 org.jacoco.agent-0.7.2.201409121644.jar_778m6tp3jrtvcetasufl59dmau
$ ls -ltr build/tmp/expandedArchives/org.jacoco.agent-0.7.2.201409121644.jar_778m6tp3jrtvcetasufl59dmau/
total 272
drwxr-xr-x+ 1 e020001 Domain Users 0 Jul 7 20:58 META-INF
-rwxr-xr-x 1 e020001 Domain Users 2652 Jul 7 20:58 about.html
-rwxr-xr-x 1 e020001 Domain Users 272311 Jul 7 20:58 jacocoagent.jar
drwxr-xr-x+ 1 e020001 Domain Users 0 Jul 7 20:58 org
The same is NOT happening when I'm running Gradle 2.3 and Java8.
Build is successful but I'm not getting build/tmp/expandedArchives/org.jacoco.... folder containing jacocoagent.jar file.
Any idea, why Gradle 2.3 is not creating this jacoco specific .jar file.
With Gradle 2.3+Java8, the following output shows UP-TO-DATE in front of both :compileTestJava and :test tasks (which was not the case with Gradle 1.6 for test task).
I ran "gradle clean build".
:compileTestJava UP-TO-DATE
:compileTestGroovy UP-TO-DATE
:processTestResources
:testClasses
:test UP-TO-DATE
:check
I need Gradle 2.3 to generate this jacocoagent.jar under build/tmp/expandedArchives/org.jacoco..... folder so that I can use it in a downstream Jenkins job (which runs non-Unit tests) as this project does have some Integration tests and I'm fetching the jacocoagent.jar from the parent main build job (which runs gradle clean build including test task) in downstream job so that I can pass it to TOMCAT JVM while starting Tomcat (so that I can get jacocoIT.exec code coverage for IT tests). But, after I switched to Gradle 2.3, all projects where I don't have src/test/java ... now jacocoagent.jar is not getting created and the copy artifact plugin fails while trying to copy the .jar file from parent job.
One more point:
With Gradle 1.6 + Java7, if I run gradle clean build, it successfully creates that jacocoagent.jar inside build/tmp/expandedArchives/org.jacoco..... folder but it works this way, only when I run gradle clean build or "gradle clean; gradle test".
If I run gradle clean build, and then remove build/tmp folder, and now just run: gradle test, it shows me UP-TO-DATE in front of both :compileTestJava and :test tasks and doesn't create build/tmp/expandedArchives/org.jacoco.... folder containing jacocoagent.jar file.
For more info, I'm attaching the profile run (i.e. using --profile option) while running gradle test task for Gradle 1.6 + java 7.
I see that, in the profile html file that when test task is run, it first calls compileJava as per Gradle process logic and then test task and it's also calling depedencies --- :jacocoAgent (as per the dependency resolution tab):
But,
with Gradle 2.3 + Java8, the dependency Resolution / order and Task execution step is not same (or in the order as compared to Gradle 1.6) for generating or showing any reference to jacocoAgent dependency as it's not even calling it.
Running Gradle1.6 +Java7 test task with -i (or --info) option shows why it ran test task even though I had no test source code, see the reason why:
Note: Recompile with -Xlint:unchecked for details.
:processResources
Skipping task ':processResources' as it has no source files.
:processResources UP-TO-DATE
:classes
Skipping task ':classes' as it has no actions.
:compileTestJava
Skipping task ':compileTestJava' as it has no source files.
:compileTestJava UP-TO-DATE
:processTestResources
Executing task ':processTestResources' due to:
No history is available.
:testClasses
Skipping task ':testClasses' as it has no actions.
:test
file or directory '/my/workspace/project/build/classes/test', not found
Executing task ':test' due to:
No history is available.
file or directory '/my/workspace/project/build/classes/test', not found
Finished generating test XML results (0.001 secs)
Generating HTML test report...
Finished generating test html results (0.012 secs)
BUILD SUCCESSFUL
you can force the test task to be executed no matter what the status of inputs and outputs are:
test{
outputs.upToDateWhen{false}
}
for earlier gradle versions you can ensure the class directory exists by
task createTestClassesDir << {sourceSets.test.output.classesDir.mkdirs()}
test.dependsOn createTestClassesDir
Summary:
With Gradle 2.3, if there are no valid .java/.groovy (or etc) test code, then test task won't even run and thus there'll be no jacocoagent.jar created somewhere deep in build/tmp/exapandedArchives/org.jacoco.xxx.... folder.
Solution was to include the following (in top level $GRADLE_HOME/init.d/some-global-file.gradle) inside allprojects { .... } section. All we are doing is, if src/test/java (standard) or any legacy folder structure (src/java if your project structure is like this) doesn't have any valid test source code, then we can add a dummy test file (DummyTestXYZ.java or groovy) and let test task run which will generate jacocoagent.jar (which we can use / tie in Tomcat options for generating jacoco report for non-unit aka integration tests). This way, if your main build job calls a downstream/child job to run your IT tests, it won't fail as it can fetch jacocoagent.jar (from main build job's workspace) as test task will create jacocoagent.jar in build/tmp/expandedArchives/org.jacoco.xx.x.xx..x folder (that you can get using Copy Artifact plugin in Jenkins).
PS: Change the if statement logic acc. to your own folder setup i.e. in which folder you'd want to create the DummyTestXYZ.java file. In our case, all new projects were using src/test/java (standard folder structure as per Maven/Gradle standard) and during the new project creation, we are adding valid sample unit tests checked-in to the source control. Thus, in the logic below, we are actually ignoring to create this DummyTestXYZ.java in case src/test/java exists and creating this file only if src/test/java folder doesn't exist in the project (i.e. this is a project which has legacy folder structure) + test/java (legacy folder for storing JUnit unit tests) has no .java programs and/or if test/java doesn't exist then create it first and then create the dummy file. I know, we could have uploaded jacocoagent.jar at some location on Jenkins server and use that file while starting Tomcat for getting code coverage for IT tests. The dummy test file we added requires junit:junit:4.10 or 4.11 library version for the :compileTestJava task to succeed.
compileJava {
doLast {
def dirName = "${projectDir}/test/java"
if(!file( "${projectDir}/src/test/java" ).exists())
if(!file( dirName ).exists())
new File( dirName ).mkdirs()
if(file( dirName ).exists()) {
def javaCnt = new FileNameByRegexFinder().getFileNames(dirName, /.*\.java/).size()
if(javaCnt == 0) {
def f = new File( dirName , 'DummyTestXYZ.java' )
def w = f.newPrintWriter()
w.println('import org.junit.Test;')
w.println('')
w.println('public class DummyTestXYZ {')
w.println('#Test' )
w.println('public void test() {')
w.println('}')
w.println('}')
w.close()
}
}
}
}
test {
doFirst {
testResultsDirName = "test-results/UT"
testReportDirName = "tests/UT"
}
maxParallelForks = 5
forkEvery = 50
//ignoreFailures = true
// Following Jacoco section is required only in Jenkins
// But a developer can uncomment them if they want this feature to work for their
// Desktop local Gradle builds.
jacoco {
//Following vars works only with versions >= 1.7 version of Gradle
destinationFile = file("$buildDir/jacoco/UT/jacocoUT.exec")
}
doLast {
if (file("${projectDir}/test/java/DummyTestXYZ.java").exists()) {
println "++"
println "++"
println "++"
println "======================================================="
println "DEV Team – Please add valid Unit tests in this project."
println "======================================================="
println "++"
println "++"
println "++"
sleep(30 * 1000)
new File("${projectDir}/build/classes/test").deleteDir()
new File("${buildDir}/jacoco/UT").deleteDir()
new File("${buildDir}/test-results/UT").deleteDir()
delete "${projectDir}/test/java/DummyTestXYZ.java"
}
}
}
//Do the same (as above test code) for any other similar test tasks like integartionTest, acceptanceTest etc..
jacocoTestReport {
//cleaning any compile time generated (for ex: JiBx classes files) so that jacoco task won't fail for not finding the actual source files (.java/.groovy for the compile time generated .class files)
doFirst {
delete fileTree (dir: "${buildDir}/classes", include: "**/JiBX_*.class")
}
group = "Reporting"
description = "Generate Jacoco coverage reports after running tests."
//ignoreFailures = true
executionData = fileTree(dir: 'build/jacoco', include: '**/*.exec')
reports {
xml{
enabled true
//Following value is a file
destination "${buildDir}/reports/jacoco/xml/jacoco.xml"
}
csv.enabled false
html{
enabled true
//Following value is a folder
destination "${buildDir}/reports/jacoco/html"
}
}
sourceDirectories = files(['src/java','src/main/java', 'src/main/groovy'])
classDirectories = files('build/classes/main')
doLast {
if (file("${projectDir}/test/java/DummyTestXYZ.java").exists()) {
delete "${projectDir}/test/java/DummyTestXYZ.java"
}
}
}

Display test coverage using jacoco in gradle

I am using a gradle file to build my project. In this file I am using jacoco to produce a test report. What I want to do is modify the build file so that it displays a message if my test coverage isn't 100%. Or at least just display the test coverage. Is this possible?
Here is my build.gradle:
apply plugin: 'java'
apply plugin: 'jacoco'
sourceSets.main.java.srcDirs = ['src']
sourceSets.test.java.srcDirs = ['test']
dependencies {
testCompile group: 'junit', name: 'junit', version: "4.+"
}
repositories {
mavenCentral()
}
jacocoTestReport {
doFirst {
classDirectories = files('build/classes/main/draw')
}
reports {
xml.enabled false
csv.enabled false
html.destination "build/reports/coverageHtml"
}
}
task(runui, dependsOn: jar, type: JavaExec) {
main = 'gui.Main'
classpath = sourceSets.main.runtimeClasspath
}
defaultTasks 'clean', 'test', 'jacocoTestReport', 'runui'
There is a very simple Gradle plugin called gradle-jacoco-log that simply logs the Jacoco coverage data:
plugins {
id 'org.barfuin.gradle.jacocolog' version '1.0.1'
}
Then after ./gradlew jacocoTestReport, it shows:
Test Coverage:
- Class Coverage: 100%
- Method Coverage: 100%
- Branch Coverage: 81.2%
- Line Coverage: 97.8%
- Instruction Coverage: 95.3%
- Complexity Coverage: 90.2%
There are also some options to customize what is logged.
The other topic of enforcing a certain test coverage is covered by Gradle these days.
Full disclosure: I am the author of this little plugin.
At the moment this is not supported by the gradle jacoco plugin. You can vote for this issue at https://issues.gradle.org/browse/GRADLE-2783 and https://issues.gradle.org/browse/GRADLE-2854. As a workaround you could possibly parse the output file in a custom task.
You can use Gradle plugin gradle-console-reporter to report various kinds of summaries to console. JUnit, JaCoCo and Cobertura reports are supported.