Test.HUnit provides a big red button to run a test:
runTestTT :: Test -> IO Counts
As there is a need to structure large test suites, Test is not a single test but is actually a labelled rose tree with Assertion in leaves:
data Test
= TestCase Assertion | TestList [Test] | TestLabel String Test
-- Defined in `Test.HUnit.Base'
It's not abstract so it's possible to process it. One particularly useful processing is extraction of subtrees by paths:
byPath = flip $ foldl f where
f (TestList l) = (l !!)
f (TestLabel _ t) = const t
f t = const t
So for example I can run a single subsuite runTestTT $ byPath [1] tests or a particular test runTestTT $ byPath [1,7,3] tests identified by test path instead of waiting for whole suite.
One disadvantage of the homegrown tool is that test paths are not preserved (shortened).
Is there such processing helper tool already on Hackage?
The closest to your needs seem to be the libraries and programs that abstract over HUnit, Quickcheck and other tests, and have their own test name grouping and management infrastructure, e.g. test-framework. It provides you with a main function that takes command line arguments, including one that allows you to specify a test or test group to run (by globbing on the name).
Related
I have a Rust test which delegates to a C++ test suite using doctest and wants to pass command-line parameters to it. My first attempt was
// in mod ffi
pub fn run_tests(cli_args: &mut [String]) -> bool;
#[test]
fn run_cpp_test_suite() {
let mut cli_args: Vec<String> = env::args().collect();
if !ffi::run_tests(
cli_args.as_mut_slice(),
) {
panic!("C++ test suite reported errors");
}
}
Because cargo test help shows
USAGE:
cargo.exe test [OPTIONS] [TESTNAME] [-- <args>...]
I expected
cargo test -- --test-case="X"
to let run_cpp_test_suite access and pass on the --test-case="X" parameter. But it doesn't; I get error: Unrecognized option: 'test-case' and cargo test -- --help shows it has a fixed set of options
Usage: --help [OPTIONS] [FILTER]
Options:
--include-ignored
Run ignored and not ignored tests
--ignored Run only ignored tests
...
My other idea was to pass the arguments in an environment variable, that is
DOCTEST_ARGS="--test-case='X'" cargo test
but then I need to somehow split that string into arguments (handling at least spaces and quotes correctly) either in Rust or in C++.
There are two pieces of Rust toolchain involved when you run cargo test.
cargo test itself looks for all testable targets in your package or workspace, builds them with cfg(test), and runs those binaries. cargo test processes the arguments to the left of the --, and the arguments to the right are passed to the binary.
Then,
Tests are built with the --test option to rustc which creates an executable with a main function that automatically runs all functions annotated with the #[test] attribute in multiple threads. #[bench] annotated functions will also be run with one iteration to verify that they are functional.
The libtest harness may be disabled by setting harness = false in the target manifest settings, in which case your code will need to provide its own main function to handle running tests.
The “libtest harness” is what rejects your extra arguments. In your case, since you're intending to run an entire other test suite, I believe it would be appropriate to disable the harness.
Move your delegation code to its own file, conventionally located in tests/ in your package directory:
Cargo.toml
src/
lib.rs
...
tests/
cpp_test.rs
Write an explicit target section in your Cargo.toml for it, with harness disabled:
[[test]]
name = "cpp_test"
# path = "tests/cpp_test.rs" # This is automatic; you can use a different path if you really want to.
harness = false
In cpp_test.rs, instead of writing a function with the #[test] attribute, write a normal main function which reads env::args() and calls the C++ tests.
[Disclaimer: I'm familiar with these mechanisms because I've used Criterion benchmarking (which similarly requires disabling the default harness) but I haven't actually written a test with custom arguments the way you're looking for. So, some details might be wrong. Please let me know if anything needs correcting.]
In addition to Kevin Reid's answer, if you don't want to write your own test harness, you can use the shell-words crate to split an environment variable into individual arguments following shell rules:
let args = var ("DOCTEST_ARGS").unwrap_or_else (|_| String::new());
let args = shell_words::split (&args).expect ("failed to parse DOCTEST_ARGS");
Command::new ("cpptest")
.args (args)
.spawn()
.expect ("failed to start subprocess")
.wait()
.expect ("failed to wait for subprocess");
Should property tests run with unit tests when using the RGR methodology?
RGR: Red -> Green -> Refactor
I noticed that a unit test that I have executes in 18ms.
However, my property test for the same method takes 215ms.
module Tests.Units
open FsUnit
open NUnit.Framework
open NUnit.Core.Extensibility
open FsCheck.NUnit
open FsCheck.NUnit.Addin
open FsCheck
let add x y = (x + y)
let commutativeProperty x y =
let result1 = add x y
let result2 = add y x // reversed params
result1 = result2
[<Test>]
let ``When I add two numbers, the result should not depend on parameter order``()=
Check.Quick commutativeProperty
[<Test>]
let ``add two numbers`` () =
add 2 3 |> should equal (5)
So my property test takes a quarter of a second to execute.
In addition, this is just one simple property test.
What is an effective method for running property tests?
Just check-ins?
With default settings, each FsCheck property runs 100 times, so it's not surprising that it's slower. Notice, though, that it isn't 100 times slower.
I often use the Red/Green/Refactor process when writing property tests (for the targeted function), and find that it works well.
It's slower than when doing TDD in C# (also because the F# compiler is slower than the C# compiler). On the other hand, the F# type system is much more expressive, so I also find that I rely more on the type system than I would in C#. This means that I need to write fewer tests.
All in all, I find that the combination of F# and FsCheck a net win over C# and plain unit testing.
When writing tests I find myself writing all kinds of small little helper functions to make assertions. I searched for an assertion library and didn't find anything. In my tests I often have things like this:
value_in_list(_Value, []) ->
false;
value_in_list(Value, [Item|List]) ->
case Value == Item of
true ->
true;
false ->
value_in_list(Value, List)
end.
test_start_link(_Config) ->
% should return the pid and register the process as my_app
{ok, Pid} = my_app:start_link(),
true = is_pid(Pid),
value_in_list(my_app, registered()).
I end up having to write a whole function to check if my_app is a registered process. It would be much nicer if I could just call something like assertion:value_in_list(my_app, registered()) or assertion:is_registered(my_app).
I come from a Ruby background so I hate having to clutter up my tests with utility functions just to make a few assertions. It would be much cleaner if I could just do:
test_start_link(_Config) ->
% should return the pid and register the process as my_app
{ok, Pid} = my_app:start_link(),
true = is_pid(Pid),
assertion:value_in_list(my_app, registered()).
So my questions are:
Why doesn't an assertion library exist for Common Test?
Would it be possible to build a third party library that would be accessible during all tests?
Some ideas for this:
Move your application startup to the suite's startup section:
init_per_suite(Config) ->
{ok, Pid} = my_app:start_link(),
true = is_pid(Pid),
[{app, Pid} | Config].
Then write your test registration as:
test_registration(Config) ->
Pid = ?config(app, Config),
true = lists:member(Pid, registered()).
There is no need to assert things via explicit assertion functions since they are "built in". Just make a failing match like above and the test process will crash. And hence report that the test case went wrong. Each test case is run in its own process. Which is also why you want to start the application in the init_per_suite/1 callback. Otherwise, my_app will be terminated once your test case has run since you are linked to the process-per-test-case.
So the answer is: assertions are built in. Hence there is less need for an assertion library as such.
On a side note, it's terser and more efficient to write that first block in pattern matching in the signature, rather than to add a case.
value_in_list(_Value, [] ) -> false;
value_in_list( Value, [Value|List] ) -> true;
value_in_list( Value, [ _ |List] ) -> value_in_list(Value, List).
I realize this should probably just be a comment to the original question, but that's murderously difficult to read without monospace and newlines.
You can just use EUnit assertions in Common Test.
-include_lib("eunit/include/eunit.hrl").
And all the regular assertions are available.
I decided to write an Erlang assertion library to help with cases like this. It provides this functionality.
I'm new to haskell and working on unit testing, however I find the ecosystem to be very confusing. I'm confused as to the relationship between HTF and HUnit.
In some examples I see you set up test cases, export them in an tests list, and then run in ghci with runTestsTT (like this HUnit example).
In other examples, you create a test runner tied into the cabal file that uses some preprocessor magic to find your tests like in this git example. Also it seems that HTF tests need to be prefixed with test_ or they aren't run? I had a hard time finding any documentation on that, I just noticed the pattern that everyone had.
Anyways, can someone help sort this out for me? What is considered the standard way of doing things in Haskell? What are the best practices? What is the easiest to set up and maintain?
Generally, any significant Haskell project is run with Cabal. This takes care of building, distribution, documentation (with the help of haddock), and testing.
The standard approach is to put your tests in the test directory and then set up a test suite in your .cabal file. This is detailed in the user manual. Here's what the test suite for one of my projects looks like
Test-Suite test-melody
type: exitcode-stdio-1.0
main-is: Main.hs
hs-source-dirs: test
build-depends: base >=4.6 && <4.7,
test-framework,
test-framework-hunit,
HUnit,
containers == 0.5.*
Then in the file test/Main.hs
import Test.HUnit
import Test.Framework
import Test.Framework.Providers.HUnit
import Data.Monoid
import Control.Monad
import Utils
pushTest :: Assertion
pushTest = [NumLit 1] ^? push (NumLit 1)
pushPopTest :: Assertion
pushPopTest = [] ^? (push (NumLit 0) >> void pop)
main :: IO ()
main = defaultMainWithOpts
[testCase "push" pushTest
,testCase "push-pop" pushPopTest]
mempty
Where Utils defines some nicer interfaces over HUnit.
For lighter-weight testing, I strongly recommend you use QuickCheck. It lets you write short properties and test them over a series of random inputs. For example:
-- Tests.hs
import Test.QuickCheck
prop_reverseReverse :: [Int] -> Bool
prop_reverseReverse xs = reverse (reverse xs) == xs
And then
$ ghci Tests.hs
> import Test.QuickCheck
> quickCheck prop_reverseReverse
.... Passed Tests (100/100)
I'm also newbie haskeller and I've found this introduction really helpful: "Getting started with HUnit". To summarize, I'll put here simple testing example of HUnit usage without .cabal project file:
Let's assume that we have module SafePrelude.hs:
module SafePrelude where
safeHead :: [a] -> Maybe a
safeHead [] = Nothing
safeHead (x:_) = Just x
we can put tests into TestSafePrelude.hs as follows:
module TestSafePrelude where
import Test.HUnit
import SafePrelude
testSafeHeadForEmptyList :: Test
testSafeHeadForEmptyList =
TestCase $ assertEqual "Should return Nothing for empty list"
Nothing (safeHead ([]::[Int]))
testSafeHeadForNonEmptyList :: Test
testSafeHeadForNonEmptyList =
TestCase $ assertEqual "Should return (Just head) for non empty list" (Just 1)
(safeHead ([1]::[Int]))
main :: IO Counts
main = runTestTT $ TestList [testSafeHeadForEmptyList, testSafeHeadForNonEmptyList]
Now it's easy to run tests using ghc:
runghc TestSafePrelude.hs
or hugs - in this case TestSafePrelude.hs has to be renamed to Main.hs (as far as I'm familiar with hugs) (don't forget to change module header too):
runhugs Main.hs
or any other haskell compiler ;-)
Of course there is more then that in HUnit, so I really recommend to read suggested tutorial and library User's Guide.
You've had answers to most of your questions, but you also asked about HTF, and how that works.
HTF is a framework that is designed for both unit testing -- it is backwards compatible with HUnit (it integrates and wraps it to provide extra functions) -- and property-based testing -- it integrates with quickcheck. It uses a preprocessor to locate tests so that you don't have to manually build a list. The preprocessor is added to your test source files using a pragma:
{-# OPTIONS_GHC -F -pgmF htfpp #-}
(alternatively, I guess you could add the same options to your ghc-options property in your cabal file, but I've never tried this so don't know if it is useful or not).
The preprocessor scans your module for top-level functions named test_xxxx or prop_xxxx and adds them to a list of tests for the module. You can either use this list directly by putting a main function in the module and running them (main = htfMain htf_thisModuleTests) or export them from the module, and have a main test program for multiple modules, which imports the modules with tests and runs all of them:
import {-# HTF_TESTS #-} ModuleA
import {-# HTF_TESTS #-} ModuleB
main :: IO ()
main = htfMain htf_importedTests
This program can be integrated with cabal using the technique described by #jozefg, or loaded into ghci and run interactively (although not on Windows - see https://github.com/skogsbaer/HTF/issues/60 for details).
Tasty is another alternative that provides a way of integrating different kinds of tests. It doesn't have a preprocessor like HTF, but has a module that performs similar functions using Template Haskell. Like HTF, it also relies on naming convention to identify your tests (in this case, case_xxxx rather than test_xxxx). In addition to HUnit and QuickCheck tests, it also has modules for handling a number of other test types.
I am looking specifically for frameworks that allow me to take advantage of unique features of the language. I am aware of FsUnit. Would you recommend something else, and why?
My own unit testing library, Unquote, takes advantage of F# quotations to allow you to write test assertions as plain, statically checked F# boolean expressions and automatically produces nice step-by-step test failure messages. For example, the following failing xUnit test
[<Fact>]
let ``demo Unquote xUnit support`` () =
test <# ([3; 2; 1; 0] |> List.map ((+) 1)) = [1 + 3..1 + 0] #>
produces the following failure message
Test 'Module.demo Unquote xUnit support' failed:
([3; 2; 1; 0] |> List.map ((+) 1)) = [1 + 3..1 + 0]
[4; 3; 2; 1] = [4..1]
[4; 3; 2; 1] = []
false
C:\File.fs(28,0): at Module.demo Unquote xUnit support()
FsUnit and Unquote have similar missions: to allow you to write tests in an idiomatic way, and to produce informative failure messages. But FsUnit is really just a small wrapper around NUnit Constraints, creating a DSL which hides object construction behind composable function calls. But it comes at a cost: you lose static type checking in your assertions. For example, the following is valid in FsUnit
[<Test>]
let test1 () =
1 |> should not (equal "2")
But with Unquote, you get all of F#'s static type-checking features so the equivalent assertion would not even compile, preventing us from introducing a bug in our test code
[<Test>] //yes, Unquote supports both xUnit and NUnit automatically
let test2 () =
test <# 1 <> "2" #> //simple assertions may be written more concisely, e.g. 1 <>! "2"
// ^^^
//Error 22 This expression was expected to have type int but here has type string
Also, since quotations are able to capture more information at compile time about an assertion expression, failure messages are a lot richer too. For example the failing FsUnit assertion 1 |> should not (equal 1) produces the message
Test 'Test.Swensen.Unquote.VerifyNunitSupport.test1' failed:
Expected: not 1
But was: 1
C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\FsUnit.fs(11,0): at FsUnit.should[a,a](FSharpFunc`2 f, a x, Object y)
C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\VerifyNunitSupport.fs(29,0): at Test.Swensen.Unquote.VerifyNunitSupport.test1()
Whereas the failing Unquote assertion 1 <>! 1 produces the following failure message (notice the cleaner stack trace too)
Test 'Test.Swensen.Unquote.VerifyNunitSupport.test1' failed:
1 <> 1
false
C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\VerifyNunitSupport.fs(29,0): at Test.Swensen.Unquote.VerifyNunitSupport.test1()
And of course from my first example at the beginning of this answer, you can see just how rich and complex Unquote expressions and failure messages can get.
Another major benefit of using plain F# expressions as test assertions over the FsUnit DSL, is that it fits very well with the F# process of developing unit tests. I think a lot of F# developers start by developing and testing code with the assistance of FSI. Hence, it is very easy to go from ad-hoc FSI tests to formal tests. In fact, in addition to special support for xUnit and NUnit (though any exception-based unit testing framework is supported as well), all Unquote operators work within FSI sessions too.
I haven't yet tried Unquote, but I feel I have to mention FsCheck:
http://fscheck.codeplex.com/
This is a port of Haskells QuickCheck library, where rather than specifying what specific tests to carry out, you specify what properties about your function should hold true.
To me, this is a bit harder than using traditional tests, but once you figure out the properties, you'll have more solid tests. Do read the introduction: http://fscheck.codeplex.com/wikipage?title=QuickStart&referringTitle=Home
I'd guess a mix of FsCheck and Unquote would be ideal.
You could try my unit testing library Expecto; it's has some features you might like:
F# syntax throughout, tests as values; write plain F# to generate tests
Use the built-in Expect module, or an external lib like Unquote for assertions
Parallel tests by default
Test your Hopac code or your Async code; Expecto is async throughout
Pluggable logging and metrics via Logary Facade; easily write adapters for build systems, or use the timing mechanism for building an InfluxDB+Grafana dashboard of your tests' execution times
Built in support for BenchmarkDotNet
Build in support for FsCheck; makes it easy to build tests with generated/random data or building invariant-models of your object's/actor's state space
Hello world looks like this
open Expecto
let tests =
test "A simple test" {
let subject = "Hello World"
Expect.equal subject "Hello World" "The strings should equal"
}
[<EntryPoint>]
let main args =
runTestsWithArgs defaultConfig args tests