First some background : I have a remote jenkins that performs some Newman tests on my web app built with PlayFramework. Results are consolidated in an xml file and displayed with the "test results analyzer" plugin in Jenkins. This is ok.
Now, I would like to add some JUnit tests to my web app and, if possible, I would like to write the new results in the same xml file so as to see all results in the same report.
As a first step, I've been trying to run JUnit tests localy and to save results in a specific file. I can run my JUnit tests, but I cannot save the results in an xml file.
Could you please help me a bit?
JUnit test:
import org.junit.Test;
import static junit.framework.TestCase.assertTrue;
public class Benchmark {
#Test
public void dummy() {
assertTrue(1+1 == 2);
System.out.println("Test has been done");
}
}
Command to run the JUnit test :
sbt clean compile test
Terminal output :
[success] Total time: 11 s, completed 18 sept. 2018 12:54:34
Test has been done
build.sbt :
name := """WebApp"""
organization := "com.organization"
version := "1.0-SNAPSHOT"
scalaVersion := "2.12.4"
sbtVersion := "1.1.1"
logLevel := Level.Warn
lazy val root = (project in file(".")).enablePlugins(PlayJava)
javaOptions ++= Seq("-Xmx3g", "-Xms3g")
libraryDependencies ++= Seq(
...,
"junit" % "junit" % "4.12",
"com.novocode" % "junit-interface" % "0.10" % "test"
)
As a second step, I would like to make it work on my remote Jenkins. If it works locally, it should be easy to make it work remotely, but just in case, below is my Jenkins configuration for the build step :
rm -r /var/lib/jenkins/workspace/WebApp/target/universal/*
cd /var/lib/jenkins/workspace/WebApp
sbt -J-Xmx3G -J-Xms3G dist
cd target/universal
unzip webapp-1.0-SNAPSHOT.zip
cd webapp-1.0-SNAPSHOT
chmod u+x bin/webapp
bash bin/webapp -Dhttp.port=9004 -J-Xms3g -J-Xmx3g &
sleep 10
newman run /home/tests/WebApp-Jenkins-Tests.postman_collection.json --reporters junit,cli --reporter-junit-export /var/lib/jenkins/workspace/WebApp/testResults/results.xml
After some tests, I've found a solution. Instead of specifying the output of JUnit, I've changed the output of newman so that newman writes results in the same folder as Junit.
Jenkins configuration :
cd /var/lib/jenkins/workspace/WebApp
sbt clean compile test
sbt -J-Xmx3G -J-Xms3G dist
cd target/universal
unzip webapp-1.0-SNAPSHOT.zip
cd webapp-1.0-SNAPSHOT
chmod u+x bin/webapp
bash bin/webapp -Dhttp.port=9004 -J-Xms3g -J-Xmx3g &
sleep 10
newman run /home/tests/Jenkins-Tests.postman_collection.json --reporters junit,cli --reporter-junit-export /var/lib/jenkins/workspace/WebApp/target/test-reports/newmanTests.xml
Also, even if I do not know if it helps, I've updated sbt to 1.2.3.
Now it works just fine :)
Related
I thought it would be a good idea to set up MarkLogic unit testing, so I've been following the unit-test-project as best as I can. Everything seems to work, but no unit tests are being performed. I've copied the four unit tests to src/test/ml-modules/root/suites/My Tests and the one module to src/main/ml-modules/root/. Everything deploys correctly. But running the unit tests gives me:
$ gradle mlUnitTest
> Task :mlUnitTest
Constructing DatabaseClient that will connect to port: 8021
Run teardown scripts: true
Run suite teardown scripts: true
Run code coverage: false
Running all suites...
Done running all suites; time: 1791ms
0 tests completed, 0 failed
The unit testing UI loads in a browser, but it also displays no tests.
This is my build.gradle file:
buildscript {
repositories {
jcenter()
mavenLocal()
}
dependencies {
classpath "com.marklogic:marklogic-unit-test-client:1.0.0"
classpath "com.marklogic:ml-gradle:4.1.0"
}
}
plugins { id "com.marklogic.ml-gradle" version "4.1.0" }
repositories {
jcenter()
}
dependencies {
mlBundle "com.marklogic:marklogic-unit-test-modules:1.0.0"
}
And gradle.properties:
mlAppName=unitTestApp
mlHost=localhost
mlUsername=admin
mlPassword=*****
mlRestPort=8020
mlTestRestPort=8021
mlModulePaths=src/main/ml-modules,src/test/ml-modules
I'm using MarkLogic 10, on 64 bit Linux, ml-gradle 4.0
Ah. It looks like the test suites are expected under the project directory in
src/test/ml-modules/root/test/suites/SuiteName
I had put suites under root and missed the second test directory.
I wanna make automated testing for my python project but I'm not sure about the correct way to use unittest module.
All of my test files are currently in one folder and have this format:
import unittest
class SampleTest(unittest.TestCase):
def testMethod(self):
# Assertion here
if __name__ == "__main__":
unittest.main()
Then I run
find ./tests -name "*_test.py" -exec python {} \;
When there are three test files, it outputs
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
It printed one summary for each test file. So the question is what can I do to make it print only one test summary, eg Ran 5 tests in 0.001s?
Thanks in advance
And I don't want to install any other module
You are invoking Python multiple times, and each process does not have any knowledge about rest of them. You need to run Python once and use unittest discover mechanism.
Run in shell:
python -m unittest discover
Depending on what is your project structure and naming conventions you may want to tweak discovery params, e.g. change --pattern option, as described in help:
Usage: python -m unittest discover [options]
Options:
-h, --help show this help message and exit
-v, --verbose Verbose output
-f, --failfast Stop on first fail or error
-c, --catch Catch Ctrl-C and display results so far
-b, --buffer Buffer stdout and stderr during tests
-s START, --start-directory=START
Directory to start discovery ('.' default)
-p PATTERN, --pattern=PATTERN
Pattern to match tests ('test*.py' default)
-t TOP, --top-level-directory=TOP
Top level directory of project (defaults to start
directory)
While you said I don't want to install any other module, I'd still recommend using another test runner. There are quite few out there, pytest or nose to name a few.
I have a Java project.
PS: In my project, I don't have any java program/source code in src/test/java. - This folder just contains a blank.txt file.
I have two different Gradle versions:
Gradle 1.6 with Java 7 (as Java 8 is NOT compatible with Gradle 1.6 or any version < less than 1.10 version if I'm correct).
The other version is: Gradle 2.3 with Java 8.
Using both of the above mentioned Gradle 1.6 + Java7 OR Gradle 2.3 + Java 8 versions my project build successfully.
Though, I noticed one thing: That while running the build, it calls "test" task automatically (as per the Gradle design, test task runs for free); I found during Gradle 1.6 + Java7 run --- I see the following output.
:jar
:assemble
:compileTestJava UP-TO-DATE
:processTestResources
:testClasses
:test
:check
As you'll notice, it says I don't have any test source code (i.e. src/test/java doesn't contain any source code OR there's nothing new for Gradle to compile this time may be nothing changed since last time gradle ran the build) and that's why compileTestJava task is showing UP-TO-DATE in front of it.
But, :test task is showing that it ran successfully. I have used jacoco (code coverage) section within test { .. } task, then it actually ran that part (as there is no UP-TO-DATE in front of test task). Jacoco section is NOT defined in my project's build.gradle but actually it's coming from a top level / GRADLE_HOME/init.d/some-common-top-level.gradle file (where test { ... has jacoco { ... } .. } section in it).
As I mentioned above, test task didn't say UP-TO-DATE, therefore, after Gradle build process was complete, I can see it created the following folder/files structure inside build/tmp/expandedArchives/org.jacoco.... folder:
$ ls -ltr build/tmp/expandedArchives/
total 4
drwxr-xr-x+ 1 e020001 Domain Users 0 Jul 7 20:45 org.jacoco.agent-0.7.2.201409121644.jar_778m6tp3jrtvcetasufl59dmau
$ ls -ltr build/tmp/expandedArchives/org.jacoco.agent-0.7.2.201409121644.jar_778m6tp3jrtvcetasufl59dmau/
total 272
drwxr-xr-x+ 1 e020001 Domain Users 0 Jul 7 20:58 META-INF
-rwxr-xr-x 1 e020001 Domain Users 2652 Jul 7 20:58 about.html
-rwxr-xr-x 1 e020001 Domain Users 272311 Jul 7 20:58 jacocoagent.jar
drwxr-xr-x+ 1 e020001 Domain Users 0 Jul 7 20:58 org
The same is NOT happening when I'm running Gradle 2.3 and Java8.
Build is successful but I'm not getting build/tmp/expandedArchives/org.jacoco.... folder containing jacocoagent.jar file.
Any idea, why Gradle 2.3 is not creating this jacoco specific .jar file.
With Gradle 2.3+Java8, the following output shows UP-TO-DATE in front of both :compileTestJava and :test tasks (which was not the case with Gradle 1.6 for test task).
I ran "gradle clean build".
:compileTestJava UP-TO-DATE
:compileTestGroovy UP-TO-DATE
:processTestResources
:testClasses
:test UP-TO-DATE
:check
I need Gradle 2.3 to generate this jacocoagent.jar under build/tmp/expandedArchives/org.jacoco..... folder so that I can use it in a downstream Jenkins job (which runs non-Unit tests) as this project does have some Integration tests and I'm fetching the jacocoagent.jar from the parent main build job (which runs gradle clean build including test task) in downstream job so that I can pass it to TOMCAT JVM while starting Tomcat (so that I can get jacocoIT.exec code coverage for IT tests). But, after I switched to Gradle 2.3, all projects where I don't have src/test/java ... now jacocoagent.jar is not getting created and the copy artifact plugin fails while trying to copy the .jar file from parent job.
One more point:
With Gradle 1.6 + Java7, if I run gradle clean build, it successfully creates that jacocoagent.jar inside build/tmp/expandedArchives/org.jacoco..... folder but it works this way, only when I run gradle clean build or "gradle clean; gradle test".
If I run gradle clean build, and then remove build/tmp folder, and now just run: gradle test, it shows me UP-TO-DATE in front of both :compileTestJava and :test tasks and doesn't create build/tmp/expandedArchives/org.jacoco.... folder containing jacocoagent.jar file.
For more info, I'm attaching the profile run (i.e. using --profile option) while running gradle test task for Gradle 1.6 + java 7.
I see that, in the profile html file that when test task is run, it first calls compileJava as per Gradle process logic and then test task and it's also calling depedencies --- :jacocoAgent (as per the dependency resolution tab):
But,
with Gradle 2.3 + Java8, the dependency Resolution / order and Task execution step is not same (or in the order as compared to Gradle 1.6) for generating or showing any reference to jacocoAgent dependency as it's not even calling it.
Running Gradle1.6 +Java7 test task with -i (or --info) option shows why it ran test task even though I had no test source code, see the reason why:
Note: Recompile with -Xlint:unchecked for details.
:processResources
Skipping task ':processResources' as it has no source files.
:processResources UP-TO-DATE
:classes
Skipping task ':classes' as it has no actions.
:compileTestJava
Skipping task ':compileTestJava' as it has no source files.
:compileTestJava UP-TO-DATE
:processTestResources
Executing task ':processTestResources' due to:
No history is available.
:testClasses
Skipping task ':testClasses' as it has no actions.
:test
file or directory '/my/workspace/project/build/classes/test', not found
Executing task ':test' due to:
No history is available.
file or directory '/my/workspace/project/build/classes/test', not found
Finished generating test XML results (0.001 secs)
Generating HTML test report...
Finished generating test html results (0.012 secs)
BUILD SUCCESSFUL
you can force the test task to be executed no matter what the status of inputs and outputs are:
test{
outputs.upToDateWhen{false}
}
for earlier gradle versions you can ensure the class directory exists by
task createTestClassesDir << {sourceSets.test.output.classesDir.mkdirs()}
test.dependsOn createTestClassesDir
Summary:
With Gradle 2.3, if there are no valid .java/.groovy (or etc) test code, then test task won't even run and thus there'll be no jacocoagent.jar created somewhere deep in build/tmp/exapandedArchives/org.jacoco.xxx.... folder.
Solution was to include the following (in top level $GRADLE_HOME/init.d/some-global-file.gradle) inside allprojects { .... } section. All we are doing is, if src/test/java (standard) or any legacy folder structure (src/java if your project structure is like this) doesn't have any valid test source code, then we can add a dummy test file (DummyTestXYZ.java or groovy) and let test task run which will generate jacocoagent.jar (which we can use / tie in Tomcat options for generating jacoco report for non-unit aka integration tests). This way, if your main build job calls a downstream/child job to run your IT tests, it won't fail as it can fetch jacocoagent.jar (from main build job's workspace) as test task will create jacocoagent.jar in build/tmp/expandedArchives/org.jacoco.xx.x.xx..x folder (that you can get using Copy Artifact plugin in Jenkins).
PS: Change the if statement logic acc. to your own folder setup i.e. in which folder you'd want to create the DummyTestXYZ.java file. In our case, all new projects were using src/test/java (standard folder structure as per Maven/Gradle standard) and during the new project creation, we are adding valid sample unit tests checked-in to the source control. Thus, in the logic below, we are actually ignoring to create this DummyTestXYZ.java in case src/test/java exists and creating this file only if src/test/java folder doesn't exist in the project (i.e. this is a project which has legacy folder structure) + test/java (legacy folder for storing JUnit unit tests) has no .java programs and/or if test/java doesn't exist then create it first and then create the dummy file. I know, we could have uploaded jacocoagent.jar at some location on Jenkins server and use that file while starting Tomcat for getting code coverage for IT tests. The dummy test file we added requires junit:junit:4.10 or 4.11 library version for the :compileTestJava task to succeed.
compileJava {
doLast {
def dirName = "${projectDir}/test/java"
if(!file( "${projectDir}/src/test/java" ).exists())
if(!file( dirName ).exists())
new File( dirName ).mkdirs()
if(file( dirName ).exists()) {
def javaCnt = new FileNameByRegexFinder().getFileNames(dirName, /.*\.java/).size()
if(javaCnt == 0) {
def f = new File( dirName , 'DummyTestXYZ.java' )
def w = f.newPrintWriter()
w.println('import org.junit.Test;')
w.println('')
w.println('public class DummyTestXYZ {')
w.println('#Test' )
w.println('public void test() {')
w.println('}')
w.println('}')
w.close()
}
}
}
}
test {
doFirst {
testResultsDirName = "test-results/UT"
testReportDirName = "tests/UT"
}
maxParallelForks = 5
forkEvery = 50
//ignoreFailures = true
// Following Jacoco section is required only in Jenkins
// But a developer can uncomment them if they want this feature to work for their
// Desktop local Gradle builds.
jacoco {
//Following vars works only with versions >= 1.7 version of Gradle
destinationFile = file("$buildDir/jacoco/UT/jacocoUT.exec")
}
doLast {
if (file("${projectDir}/test/java/DummyTestXYZ.java").exists()) {
println "++"
println "++"
println "++"
println "======================================================="
println "DEV Team – Please add valid Unit tests in this project."
println "======================================================="
println "++"
println "++"
println "++"
sleep(30 * 1000)
new File("${projectDir}/build/classes/test").deleteDir()
new File("${buildDir}/jacoco/UT").deleteDir()
new File("${buildDir}/test-results/UT").deleteDir()
delete "${projectDir}/test/java/DummyTestXYZ.java"
}
}
}
//Do the same (as above test code) for any other similar test tasks like integartionTest, acceptanceTest etc..
jacocoTestReport {
//cleaning any compile time generated (for ex: JiBx classes files) so that jacoco task won't fail for not finding the actual source files (.java/.groovy for the compile time generated .class files)
doFirst {
delete fileTree (dir: "${buildDir}/classes", include: "**/JiBX_*.class")
}
group = "Reporting"
description = "Generate Jacoco coverage reports after running tests."
//ignoreFailures = true
executionData = fileTree(dir: 'build/jacoco', include: '**/*.exec')
reports {
xml{
enabled true
//Following value is a file
destination "${buildDir}/reports/jacoco/xml/jacoco.xml"
}
csv.enabled false
html{
enabled true
//Following value is a folder
destination "${buildDir}/reports/jacoco/html"
}
}
sourceDirectories = files(['src/java','src/main/java', 'src/main/groovy'])
classDirectories = files('build/classes/main')
doLast {
if (file("${projectDir}/test/java/DummyTestXYZ.java").exists()) {
delete "${projectDir}/test/java/DummyTestXYZ.java"
}
}
}
I just installed PHPUnit on OS X + MAMP. (My phpunit was installed in my MAMP folder, so I copied it to /usr/bin because I couldn't use the "phpunit" command in terminal without the path)
New Symfony folder
When I go to a new Symfony2 folder and I try phpunit -c App/the tests start and I get a green confirmation that all tests passed.
PHPUnit 3.7.8 by Sebastian Bergmann.
Configuration read from
/Users/username/Dropbox/www/symfonydev/Symfony/App/phpunit.xml.dist
.
Time: 0 seconds, Memory: 15.50Mb
OK (1 test, 1 assertion)
Own symfony website
But when I go to my own little Symfony2 project and execute the same command, nothing happens! He doesn't even load the phpunit.xml.dist. I get no output at all on my terminal window.
The test file, however, does exist and is readable, I can start it by explicitly specifying it on the command line of phpunit:
phpunit -c app/ src/MatchTracker/Bundle/Tests/Controller/AuthenticationControllerTest.phpand this works.
It seems that something is wrong with the phpunit.xml.dist. It's the default phpunit.xml.dist and I tried renaming it to phpunit.xml. But nothing happens. What could be the problem? I think he can't find my xml or use it, or maybe he can read the phpunit.xml but doesn't find my tests.
Here's an example from my terminal: You see that the phpunit command doesn't do anything, except when I specify the test file...
MacBook-Pro:www username$ ls
LICENSE app composer.json composer.lock composer.phar src vendor web
MacBook-Pro:www username$ ls app/
AppCache.php SymfonyRequirements.php cache console phpunit.xml.dist
AppKernel.php autoload.php check.php logs
Resources bootstrap.php.cache config phpunit.xml
MacBook-Pro:www username$ phpunit -c app/
MacBook-Pro:www username$ phpunit -c app/ src/MatchTracker/Bundle/Tests/Controller/AuthenticationControllerTest.php
PHPUnit 3.7.8 by Sebastian Bergmann.
Configuration read from /Users/username/Dropbox/www/matchtracker.be/www/app/phpunit.xml
F
Time: 1 second, Memory: 21.75Mb
There was 1 failure:
1) MatchTracker\Bundle\Tests\Controller\DefaultControllerTest::testIndex
Failed asserting that 404 matches expected 200.
/Users/username/Dropbox/www/matchtracker.be/www/src/MatchTracker/Bundle/Tests/Controller/AuthenticationControllerTest.php:28
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
MacBook-Pro:www username$
What's in your phpunit.xml? <testsuites> section of this file tells phpunit which directories scan for tests.
Remember that phpunit.xml overwrites phpunit.xml.dist.
Look at the original phpunit.xml.dist file and either remove the phpunit.xml or start by copying phpunit.xml.dist into phpunit.xml.
I'm in the middle of my first ever stab at setting up Jenkins to build and run unit tests /code coverage with my CakePHP project. So far I have successfully got Jenkins fetching and building automatically from my BitBucket repository - a small victory in itself.
Next thing I want to happen is for the unit tests to run and code coverage reports to be populated.
Here is my build.xml, which is being executed in Jenkins with the (only) build command phing -f $WORKSPACE/build.xml
<?xml version="1.0" encoding="UTF-8"?>
<project name="Consumer Love" default="phpunit">
<target name="phpunit">
<exec command="cake test app --coverage-clover logs/reports/clover.xml"></exec>
</target>
</project>
I think the issue is that when you run cake test app it asks for a prompt of which specific tests you want to run, I have been unable to figure out a method to run all of my CakePHP app unit tests.
The solution was to create a custom CakePHP Test suite which adds specific files/directories to be tested, then run that suite with the command cake test app AllTests.
For example, here is my Test/Case/AllTests.php:
/*
* Custom test suite to execute all tests
*/
class AllTestsTest extends PHPUnit_Framework_TestSuite {
public static function suite() {
$path = APP . 'Test' . DS . 'Case' . DS;
$suite = new CakeTestSuite('All tests');
$suite->addTestDirectory($path . 'Model' . DS);
return $suite;
}
}
This testsuite simply adds the Models directory to the testing environment, so all my model tests now get executed. As you can see it can be extended to run more/all tests as seen fit.
Try cake test app all. I can't confirm this makes the difference just now, but I've pulled this out of a phing build file where I'm doing the same thing as you so it should be good.