I'm exploring DUnitX for the first time and rebuilding a Windows 10 calculator clone to practice test-driven development. I have three test classes (paralleling a main CalculatorLogic class, an Equation class, and an Operand class). I've written four tests for Operand that simply assert that a value (float) or operator (enum) set in the Operand object is the same received from the 'get' method. Set/Get validation.
unit OperandTest;
interface
uses
DUnitX.TestFramework, Vcl.Dialogs, Operands, Operation, System.Rtti;
type
[TestFixture]
OperandTester = class
private
op : Operand;
public
[Setup]
procedure Setup;
[TearDown]
procedure TearDown;
[Test]
[TestCase('0','0')]
[TestCase('0.0','0.0')]
[TestCase('-0.1','-0.1')]
[TestCase('maxInt','MaxInt')]
[TestCase('maxLongInt','MaxLongInt')]
[TestCase('0.00000000001','0.00000000001')]
procedure TestGetValueSet(const setValue : Real);
[Test]
[TestCase('0/1','0,1')]
[TestCase('-0.1/0.1','-0.1,0.1')]
[TestCase('99999999999/0','99999999999,0')]
[TestCase('0.00000000001/0.1','0.00000000001,0.1')]
procedure TestNotGetValueNotSet(const setValue, falseValue : Real);
[Test]
[TestCase('Add','OpType.Add')]
[TestCase('Nothing','OpType.Nothing')]
procedure TestGetOpSet(const setOp : OpType);
[Test]
[TestCase('Add/Nothing','OpType.Add,OpType.Nothing')]
[TestCase('Nothing/Multiply','OpType.Nothing,OpType.Multiply')]
[TestCase('Nothing/Nothing','OpType.Nothing,OpType.Nothing')]
procedure TestNotGetOpNotSet(const setOp, falseOp : OpType);
end;
The issue I'm facing is that the 10 tests for value (first two functions) run but the second two for operator do not. The following DUnitX report shows 14 tests, 10 for the OperandTest value tests and 4 dummy tests for the other test units. No indication of TestGetOpSet or TestNotGetOpNotSet in the report, no indication that they were ignored, and a ShowMessage in those procedures is not executed.
**********************************************************************
* DUnitX - (c) 2015-2018 Vincent Parrett & Contributors *
* *
* License - http://www.apache.org/licenses/LICENSE-2.0 *
**********************************************************************
Fixture : EquationTest
-------------------------------------------------
Fixture : EquationTest.EquationTester
-------------------------------------------------
Test : EquationTest.EquationTester.Test1
-------------------------------------------------
Running Setup for : Test1
Executing Test : Test1
Running Teardown for Test : Test1
Success.
Test : EquationTest.EquationTester.Test2
-------------------------------------------------
Running Setup for : Test2
Executing Test : Test2
Running Teardown for Test : Test2
Success.
Running Fixture Teardown Method : Destroy
Fixture : CalcTest
-------------------------------------------------
Fixture : CalcTest.VCLCalcTester
-------------------------------------------------
Test : CalcTest.VCLCalcTester.Test1
-------------------------------------------------
Running Setup for : Test1
Executing Test : Test1
Running Teardown for Test : Test1
Success.
Test : CalcTest.VCLCalcTester.Test2
-------------------------------------------------
Running Setup for : Test2
Executing Test : Test2
Running Teardown for Test : Test2
Success.
Running Fixture Teardown Method : Destroy
Fixture : OperandTest
-------------------------------------------------
Fixture : OperandTest.OperandTester
-------------------------------------------------
Test : OperandTest.OperandTester.TestGetValueSet.0
-------------------------------------------------
Running Setup for : TestGetValueSet.0
Executing Test : TestGetValueSet.0
Running Teardown for Test : TestGetValueSet.0
Success.
Test : OperandTest.OperandTester.TestGetValueSet.0.0
-------------------------------------------------
Running Setup for : TestGetValueSet.0.0
Executing Test : TestGetValueSet.0.0
Running Teardown for Test : TestGetValueSet.0.0
Success.
Test : OperandTest.OperandTester.TestGetValueSet.-0.1
-------------------------------------------------
Running Setup for : TestGetValueSet.-0.1
Executing Test : TestGetValueSet.-0.1
Running Teardown for Test : TestGetValueSet.-0.1
Success.
Test : OperandTest.OperandTester.TestGetValueSet.maxInt
-------------------------------------------------
Running Setup for : TestGetValueSet.maxInt
Executing Test : TestGetValueSet.maxInt
Running Teardown for Test : TestGetValueSet.maxInt
Success.
Test : OperandTest.OperandTester.TestGetValueSet.maxLongInt
-------------------------------------------------
Running Setup for : TestGetValueSet.maxLongInt
Executing Test : TestGetValueSet.maxLongInt
Running Teardown for Test : TestGetValueSet.maxLongInt
Success.
Test : OperandTest.OperandTester.TestGetValueSet.0.00000000001
-------------------------------------------------
Running Setup for : TestGetValueSet.0.00000000001
Executing Test : TestGetValueSet.0.00000000001
Running Teardown for Test : TestGetValueSet.0.00000000001
Success.
Test : OperandTest.OperandTester.TestNotGetValueNotSet.0/1
-------------------------------------------------
Running Setup for : TestNotGetValueNotSet.0/1
Executing Test : TestNotGetValueNotSet.0/1
Running Teardown for Test : TestNotGetValueNotSet.0/1
Success.
Test : OperandTest.OperandTester.TestNotGetValueNotSet.-0.1/0.1
-------------------------------------------------
Running Setup for : TestNotGetValueNotSet.-0.1/0.1
Executing Test : TestNotGetValueNotSet.-0.1/0.1
Running Teardown for Test : TestNotGetValueNotSet.-0.1/0.1
Success.
Test : OperandTest.OperandTester.TestNotGetValueNotSet.99999999999/0
-------------------------------------------------
Running Setup for : TestNotGetValueNotSet.99999999999/0
Executing Test : TestNotGetValueNotSet.99999999999/0
Running Teardown for Test : TestNotGetValueNotSet.99999999999/0
Success.
Test : OperandTest.OperandTester.TestNotGetValueNotSet.0.00000000001/0.1
-------------------------------------------------
Running Setup for : TestNotGetValueNotSet.0.00000000001/0.1
Executing Test : TestNotGetValueNotSet.0.00000000001/0.1
Running Teardown for Test : TestNotGetValueNotSet.0.00000000001/0.1
Success.
Running Fixture Teardown Method : Destroy
Done testing.
Tests Found : 14
Tests Ignored : 0
Tests Passed : 14
Tests Leaked : 0
Tests Failed : 0
Tests Errored : 0
Done.. press <Enter> key to quit.
I'm having a hard time finding sample code or tutorials with more than two test procedures so I don't know if there's any additional notation I need to add to successive tests. What am I missing? What formatting or DUnitX notation is incorrect/missing that would make the test suite completely ignore the last two tests? Thanks!
Procedure implementation, if it helps:
procedure OperandTester.TestGetValueSet(const setValue : Real);
begin
op.setValue(setValue);
Assert.AreEqual(op.getValue(), setValue);
end;
procedure OperandTester.TestNotGetValueNotSet(const setValue, falseValue : Real);
begin
op.setValue(setValue);
Assert.AreNotEqual(op.getValue(), falseValue);
end;
procedure OperandTester.TestGetOpSet(const setOp : OpType);
begin
ShowMessage ('TestGetOpSet');
op.setOperation(setOp);
Assert.AreEqual(op.getOperation(), setOp);
end;
procedure OperandTester.TestNotGetOpNotSet(const setOp, falseOp : OpType);
begin
op.setOperation(setOp);
Assert.IsFalse(Ord(op.getOperation())= Ord(falseOp));
end;
Related
I have a pipeline (classic view) with the task "Visual Studio Test", with task version "2.*".
After the task completes I can see that it prints in the log the test results.
How can I save 'Total Tests' and 'Passed Tests' in variables to use with further tasks of the pipeline?
I tried extracting the .trx file but it gets deleted after the task completes.
Performing VsTest gives me this (Some tests fail, but it's OK):
Adding trx file C:\vsts-agent-win-x64-2.165.2\_work\6\s\TestResults\TestResults\----.trx to run attachments
**************** Completed test execution *********************
Result Attachments will be stored in LogStore
Publishing test results to test run '3748'.
TestResults To Publish 189, Test run id:3748
Test results publishing 189, remaining: 0. Test run id: ---
Published test case results: 189
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Received the command : Stop
TestExecutionHost.ProcessCommand. Stop Command handled
SliceFetch Aborted. Moving to the TestHostEnd phase
Please use this link to analyze the test run : https://---
Test run '---' is in 'Completed' state with 'Total Tests' : 202 and 'Passed Tests' : 19.
##[error]System.Exception: Some tests in the test run did not pass, failing the task.
##########################################################################
Finishing: VsTest - testPlan
When I try to cd into the TestResults:
+ cd C:\vsts-agent-win-x64-2.165.2\_work\6\s\TestResults\TestResults
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (C:\vsts-agent-w...lts\TestResults:String) [Set-Location], ItemNotFoundE
xception
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLocationCommand
##[error]PowerShell exited with code '1'.
You can change the default test result output folder by setting the Test results folder field. See below:
Folder to store test results. When this input is not specified, results are stored in $(Agent.TempDirectory)/TestResults by default, which is cleaned at the end of a pipeline run
In above example. The test result .trx file will be stored at $(System.DefaultWorkingDirectory)\TestResults folder which will not be cleaned up.
Then you can extracting the .trx file in the following tasks and save 'Total Tests' and 'Passed Tests' in variables.
See below screenshot from my test pipeline:
Vstest task log:
Powershell task to ls the contents:
So it seems VsTest deletes all its results after the task is complete.
I solved this with a REST API command.
Make sure you convert your Personal Access Token to Base64...
Here's how I did it:
$personalToken = [your token]
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalToken)"))
$header = #{authorization = "Basic $token"}
$params = #{
Uri = 'https://dev.azure.com/[organization]/[project]/_apis/test/runs?buildIds=[BuildId]&api-version=6.0'
Headers = $header
Method = 'GET'
}
$output = Invoke-RestMethod #params
$run = $output.value | Where-Object{$_.name -match [BuildId]}
Write-Host "Total Tests: $($run.totalTests)"
Write-Host "Passed Tests: $($run.passedTests)"
Write-Host "Failed Tests: $($run.unanalyzedTests)"
Write-Host "Skipped Tests: $($run.incompleteTests)"
When running JUnit tests with the groovy command using the built-in automatic test runner, it exits 0 even when tests fail. I'd like the command to exit non-zero if there are test failures. Is there a way I can do this?
#!/usr/bin/env groovy
import org.junit.*
class BasicTest {
#Test
void test_failure() {
assert false
}
}
$ groovy --version
Groovy Version: 3.0.2 JVM: 13.0.2 Vendor: Oracle Corporation OS: Mac OS X
$ groovy basic_test.groovy
JUnit 4 Runner, Tests: 1, Failures: 1, Time: 8
Test Failure: test_failure(BasicTest)
Assertion failed:
assert false
at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:434)
...
$ echo $?
0
Thanks folks!
Not seen people running tests like that before... (writing them as a *nix script)
You can do this though, to catch a failure, and exiting with 1
#!/usr/bin/env groovy
import org.junit.*
import org.junit.rules.*
class BasicTest {
#Rule
public TestRule watchman = [
failed: {e, d ->
println d
e.printStackTrace()
System.exit(1)
}
] as TestWatcher
#Test
void test_failure() {
assert false
}
}
I have a Grails 2.5.0 app running and this test:
package moduleextractor
import grails.test.mixin.TestFor
import spock.lang.Specification
/**
* See the API for {#link grails.test.mixin.web.ControllerUnitTestMixin} for usage instructions
*/
#TestFor(ExtractorController)
class ExtractorControllerSpec extends Specification {
def moduleDataService
def mockFile
def setup() {
moduleDataService = Mock(ModuleDataService)
mockFile = Mock(File)
}
def cleanup() {
}
void "calls the moduleDataService"() {
given: 'a term is passed'
params.termCode = termCode
when: 'the getModuleData action is called'
controller.getModuleData()
then: 'the service is called 1 time'
1 * moduleDataService.getDataFile(termCode, 'json') >> mockFile
where:
termCode = "201415"
}
}
If I run grails test-app unit:spock I get this:
| Tests PASSED - view reports in /home/foo/Projects/moduleExtractor/target/test-reports
I don't understand why it sees 2 tests. I have not included spock in my BuildConfig file as it is already included in Grails 2.5.0. Also the test is not supposed to pass, as I do not have a service yet. Why does it pass?
Also when I run this grails test-app ExtractorController I get another result:
| Running 2 unit tests...
| Running 2 unit tests... 1 of 2
| Failure: calls the moduleDataService(moduleextractor.ExtractorControllerSpec)
| Too few invocations for:
1 * moduleDataService.getDataFile(termCode, 'json') >> mockFile (0 invocations)
Unmatched invocations (ordered by similarity):
None
at org.spockframework.mock.runtime.InteractionScope.verifyInteractions(InteractionScope.java:78)
at org.spockframework.mock.runtime.MockController.leaveScope(MockController.java:76)
at moduleextractor.ExtractorControllerSpec.calls the moduleDataService(ExtractorControllerSpec.groovy:27)
| Completed 1 unit test, 1 failed in 0m 3s
| Tests FAILED - view reports in /home/foo/Projects/moduleExtractor/target/test-reports
| Error Forked Grails VM exited with error
If I run grails test-app unit: I get:
| Running 4 unit tests...
| Running 4 unit tests... 1 of 4
| Failure: calls the moduleDataService(moduleextractor.ExtractorControllerSpec)
| Too few invocations for:
1 * moduleDataService.getDataFile(termCode, 'json') >> mockFile (0 invocations)
Unmatched invocations (ordered by similarity):
None
at org.spockframework.mock.runtime.InteractionScope.verifyInteractions(InteractionScope.java:78)
at org.spockframework.mock.runtime.MockController.leaveScope(MockController.java:76)
at moduleextractor.ExtractorControllerSpec.calls the moduleDataService(ExtractorControllerSpec.groovy:27)
| Completed 1 unit test, 1 failed in 0m 3s
| Tests FAILED - view reports in /home/foo/Projects/moduleExtractor/target/test-reports
| Error Forked Grails VM exited with error
First of all could somebody tell me what is the correct syntax to run spock tests?
Also what is the difference between having unit and unit: and unit:spock in the command?
(Since Spock comes with Grails 2.5.0, it will run spocks tests anyway.)
What is the correct syntax and why does it sees 2 tests instead of 1 ?
Don't be concerned with the number of tests. It's never been a problem for me. You can always check the report HTML file to see exactly what ran.
I always run my tests with either
grails test-app
or
grails test-app ExtractorController
The error you're getting means you coded the test to expect moduleDataService.getDataFile() to get called with parameters null and 'json' when controller.getModuleData() is called. However, moduleDataService.getDataFile() never got called, so the test failed.
Spock takes some getting used to. I recommend looking at examples in the Grails documentation and reading the Spock Framework Reference.
First question: for the 'grails test-app unit:spock', have you looked at the results to see the tests it says passed? The test count at the CLI can be wrong, check your results to see what actually ran (if no tests actually ran, then there were no failures).
Your test method doesn't start with 'test', nor does it have a #Test annotation, so the 'void "calls the moduleDataService"' isn't being seen as a spock test case (I believe that is the reason).
When you run 'grails test-app ExtractorController', you aren't specifying that it has to be a spock test, so grails testing finds and executes the 'calls the moduleDataService' test method.
Since spock is the de facto testing framework, you can just use:
grails test-app -unit
Second question:
#TestFor creates your controller, but if you're running a unit test, then the usual grails magic isn't happening. Your controller code is executing in isolation. If your ExtractorController usually has the moduleDataService injected, you'll have to take care of that.
I work in grails 2.4.3, and here would be my interpretation of your test (assuredly in need of tweaking since I'm inferring a lot in this example):
import grails.test.mixin.TestFor
import grails.test.mixin.Mock
import spock.lang.specification
import some.pkg.ModuleDataService // if necessary
import some.pkg.File // if necessary
#TestFor(ExtractorController)
#Mock([ModuleDataService, File])
class ExtractorControllerSpec extends Specification
def "test callsModuleDataService once for a termCode"() {
setup:
def mockFile = mockFor(File)
def mockService = mockFor(ModuleDataService, true) // loose mock
// in this mockService, we expect getDataFile to be called
// just once, with two parameters, and it'll return a mocked
// file
mockService.demand.getDataFile(1) { String termCode, String fmt ->
return mockFile.createMock()
}
controller.moduleDataService = mockService.createMock()
when:
controller.params.termCode = "201415"
controller.getModuleData()
then:
response.status == 200 // all good?
}
}
Last question: is that a Banner term code? (just curious)
I am just starting to learn Grails testing and I tried to write my first grails test.For this, I created a fresh grails project and created a controller named com.rahulserver.SomeController:
package com.rahulserver
class SomeController {
def index() { }
def someAction(){
}
}
When I created this controller, grails automatically created a com.rahulserver.SomeControllerSpec under test/unit folder.
Here is my SomeControllerSpec.groovy:
package com.rahulserver
import grails.test.mixin.TestFor
import spock.lang.Specification
/**
* See the API for {#link grails.test.mixin.web.ControllerUnitTestMixin} for usage instructions
*/
#TestFor(SomeController)
class SomeControllerSpec extends Specification {
def setup() {
}
def cleanup() {
}
void testSomeAction() {
assert 1==1
}
}
When I right click this class, and run this test, I get following:
Testing started at 5:21 PM ...
|Loading Grails 2.4.3
|Configuring classpath
.
|Environment set to test
....................................
|Running without daemon...
..........................................
|Compiling 1 source files
.
|Running 1 unit test...|Running 1 unit test... 1 of 1
--Output from initializationError--
Failure: |
initializationError(org.junit.runner.manipulation.Filter)
|
java.lang.Exception: No tests found matching grails test target pattern filter from org.junit.runner.Request$1#1f0f9da5
at org.junit.internal.requests.FilterRequest.getRunner(FilterRequest.java:35)
at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
No tests found matching grails test target pattern filter from org.junit.runner.Request$1#1f0f9da5
java.lang.Exception: No tests found matching grails test target pattern filter from org.junit.runner.Request$1#1f0f9da5
at org.junit.internal.requests.FilterRequest.getRunner(FilterRequest.java:35)
at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
|Completed 1 unit test, 1 failed in 0m 0s
.Tests FAILED
|
- view reports in D:\115Labs\grailsunittestdemo\target\test-reports
Error |
Forked Grails VM exited with error
Process finished with exit code 1
So why is it failing?
EDIT
I am using grails 2.4.3
The unit tests are defined with Spock by default:
void testSomeAction() {
assert 1==1
}
Should be written as:
void "Test some action"() {
expect:
1==1
}
See http://spockframework.github.io/spock/docs/1.0/index.html
I'm learning grails and read Grails In Action book. Try perform some tests from it, but got strange behaviour for me. I have next simple integration test:
#Test
public void testProjections() throws Exception {
User user1 = new User(mail: 'test1#test.tld', password: 'password1').save(flush: true)
User user2 = new User(mail: 'test2#test.tld', password: 'password2').save(flush: true)
assertNotNull(user1)
assertNotNull(user2)
// Chain add Tag to Post
user1.addToPosts(new Post(content: 'First').addToTags(new Tag(name: 'tag-0')))
// Separate add tag to post
Post post = user1.posts.iterator().next()
Tag tag1 = new Tag(name: 'tag-1')
post.addToTags(tag1)
// http://stackoverflow.com/questions/6288991/do-i-ever-need-to-explicitly-flush-gorm-save-calls-in-grails
// Have tried with and without next line without success:
//sessionFactory.getCurrentSession().flush()
assertEquals(['tag-0', 'tag-1'], user1.posts.iterator().next().tags*.name.sort()) // line 154
…
}
Then I run it twice subsequently:
grails>
grails> test-app -rerun -integration
| Running 5 integration tests... 2 of 5
| Failure: testProjections(com.tariffus.QueryIntegrationTests)
| java.lang.AssertionError: expected:<[tag-0, tag-1]> but was:<[tag-1]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at com.tariffus.QueryIntegrationTests.testProjections(QueryIntegrationTests.groovy:154)
| Completed 5 integration tests, 1 failed in 0m 0s
| Tests FAILED - view reports in /home/pasha/Projects/grails/com.tariffus/target/test-reports
grails>
grails> test-app -rerun -integration
| Running 5 integration tests... 2 of 5
| Failure: testProjections(com.tariffus.QueryIntegrationTests)
| java.lang.AssertionError: expected:<[3, 1, 2]> but was:<[[tag-1, tag-2, tag-0, tag-5, tag-3, tag-4], [tag-6]]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at com.tariffus.QueryIntegrationTests.testProjections(QueryIntegrationTests.groovy:164)
| Completed 5 integration tests, 1 failed in 0m 0s
| Tests FAILED - view reports in /home/pasha/Projects/grails/com.tariffus/target/test-reports
grails>
As you can see first fails on line 157 and second, runned just after that in second without any modification goes further.
I use Postgres database and environment test configured dataSource in mode dbCreate = 'update'.
What I do incorrect and why it works sometimes?
I would say that a source of problem is this line:
user1.addToPosts(new Post(content: 'First').addToTags(new Tag(name: 'tag-0')))
These dynamic addTo* methods does not propagate save to the associated instances until save() is called on the parent instance. So calling save() on user1 after should fix it:
user1.addToPosts(new Post(content: 'First').addToTags(new Tag(name: 'tag-0')))
user1.save()
This should propagate save() to Post instance at first and then to Tag instance transitively.