We are using F# for our HTTP calls automation tests and I stumbled upon one runtime error from FsUnit that I am not sure I understand, I admit I wrote F# only 2 times in my life therefore this might be stupid question but bear with me.
This is the test:
[<Test>]
[<AsyncStateMachine(typeof<Task>)>]
member x.``When all dependent microservices and kafka are up and running service should be healty``() = toTask <| async {
let! responseContent = Http.AsyncRequestString(url = config.ApiHost + "/BetsReportingApiModule/healthcheck")
let expectedIsAlive = JsonValue.Parse(responseContent).GetProperty("alive").AsBoolean
return expectedIsAlive |> should be True
}
And this is the error message:
Message: Expected: True
But was: << StartupCode$SBTech-BetsReportingAPI-AutomationTests>.$HttpApiTests+expectedIsAlive#1689>
I can guess that this is some runtime generated string but I am not sure what is the cause if it.
Related
I am implementing a virtual machine in Rust as an exercise. Some integration tests run small programs on that virtual machine and make assertions about the final result.
However, it can happen that a bug leads to the machine getting stuck in an infinite loop, meaning that cargo test never finishes.
Is there a way to make a unit / integration test fail after a given amount of time? Maybe something similar to #[should_panic], but instead, say, #[time_limit(500)]?
The ntest crate with its #[timeout(<ms>)] macro as mentioned in the comments by SirDarius may be a simpler solution than the following.
As an alternative, there's a crate that allows creation of processes that run closures, which has some test related features. Processes, unlike threads, can be killed if they run too long.
The crate procspawn has this closure feature, and some other nice features useful for tests where timeouts are desired. Another one of these features is serialization of the return value from the code invoked in the child process. The test code can get the result back and check it in a very straightforward way.
The code to be tested:
use std::thread;
use std::time::Duration;
fn runs_long() -> i32
{
thread::sleep(Duration::from_millis(5000));
42
}
Within the same file, we can add some tests with timeouts:
#[cfg(test)]
mod tests {
use std::time::Duration;
use procspawn;
use crate::*;
procspawn::enable_test_support!();
#[test]
fn runs_long_passes()
{
let handle = procspawn::spawn((), |_| runs_long());
match handle.join_timeout(Duration::from_millis(6000)) {
Ok(result) => assert_eq!(result, 42),
Err(e) => panic!("{}", e),
}
}
#[test]
fn runs_long_fails()
{
let handle = procspawn::spawn((), |_| runs_long());
match handle.join_timeout(Duration::from_millis(1000)) {
Ok(result) => assert_eq!(result, 42),
Err(e) => panic!("{}", e),
}
}
}
And running the tests with cargo test on the command line, we get:
running 3 tests
test tests::procspawn_test_helper ... ok
test tests::runs_long_fails ... FAILED
test tests::runs_long_passes ... ok
failures:
---- tests::runs_long_fails stdout ----
thread 'tests::runs_long_fails' panicked at 'process spawn error: timed out', src/main.rs:50:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
tests::runs_long_fails
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.01s
error: test failed, to rerun pass '--bin procspawn'
To enable the test support feature, include the following in your Cargo.toml file:
[dependencies]
procspawn = { version = "0.10", features = ["test-support"] }
I have a simple Publish-Subscriber that I want to write tests for.
The methods called here are all behaviours, except get_number_consumed_messages that would be a function.
class iso _SinglePubSub is UnitTest
fun name(): String => "single publish/consume"
fun apply(h: TestHelper) =>
let p = Publisher("publisher message", h.env.out)
let queue = Queue(1, h.env.out)
let c = Consumer(h.env.out)
p.publish_message(queue)
p.publish_message(queue)
c.consume_message(queue)
c.consume_message(queue)
//Run after all behaviours are done
let n = c.get_number_consumed_messages()
h.assert_eq[USize](2, n)
How would someone implement the get_number_consumed_messages function/behaviour or how would you have to modify the test function?
First of all, c.get_number_consumed_messages() must be a behaviour as well. It is the only way to let one actor communicate with another. This has the added benefit of behaviours being run in the same order as they are called, which means c.get_number_consumed_messages() would run after both calls to c.consume_message(queue).
Given that, since Consumer is also an actor, calling it with behaviours -- and not methods -- means that we cannot return data from it directly. To actually receive data from another actor, you should use the Promise pattern, for example:
use "promises"
actor Consumer
var message_count: USize = 0
be consume_message(queue: OutStream) =>
... // Do some things
message_count = message_count + 1
... // Do other things
be get_number_consumed_messages(p: Promise[USize]) =>
p(message_count)
To actually test it, you would need to follow an adapted version of the Testing Notifier Interactions pattern for long tests, for example:
use "ponytest"
use "promises"
class iso _SinglePubSub is UnitTest
fun apply(h: TestHelper) =>
h.long_test(2_000_000_000)
... // Produce and consume messages
let p = Promise[USize]
p.next[None]({(n: USize): None =>
h.assert_eq[USize](2, n)
h.complete(true) })
c.get_number_consumed_messages(p)
(Notice the extra calls to h.long_test and h.complete, as well as the promise wrapping a lambda with the end of our test.)
For more information on these concepts, I would recommend familiarizing yourself with the stdlib documentation on Promises and the "Long tests" section of Ponytest.
Getting a System.MissingMethodException, Method not found: 'Void RouteBuilder.set_Item()
Get["/foo"] = parameters => { return Bar(Request);};
This runs fine when calling from browser, but fails when testing with this setup
var browser = new Browser(with =>
{
with.Module<Foobar>();
}
var response = brower.Get("/Foo", with => {with.HttpRequest();});
Any clue why the Routebuilder for testing won't pick up this route?
Turns out I had created the test project using the pre-release version of Nancy.Testing. This in turn made TinyIOC unhappy when trying to build routes/dependencies. So, if you see this mysterious message, check that your working code and test code are referencing the same packages.
I have what I think are standard functional tests set up for the intern and I can get them to pass consistently in several browsers. I'm still evaluating if it makes sense to use the intern for a project so I'm trying to see what happens when tests fail, and currently if I make one test fail, it always seems to cause all the tests in the suite to fail.
My tests look a bit like :
registerSuite({name : 'demo',
'thing that works' : function () {
return this.remote.get('http://foo.com')
.waitForCondition("typeof globalThing !== 'undefined'", 5000)
.elementById('bigRedButton')
.clickElement()
.end()
.eval('jsObj.isTrue()')
.then(function(result){
assert.isTrue(result);
})
.end(); // not sure if this necessary...
},
'other thing that works': function() {
// more of the same
}
});
I'm going to try and debug to figure out this for myself, but I was just wondering if anyone knows if this is expected behaviour (1 test failure causes whole test suite to fail, and report that all tests in suite have failed), or whether its more likely that my set up is wrong and I have bad interactions between the promises or something?
Any help would be awesome, and happy to provide any more info if helpful :)
Thanks!
I ran into the exact same issue a few weeks ago and created a ticket on github for the issue: https://github.com/theintern/intern/issues/46.
It's tagged 'needs-triage' at the moment, I've no idea of what it means.
I am looking specifically for frameworks that allow me to take advantage of unique features of the language. I am aware of FsUnit. Would you recommend something else, and why?
My own unit testing library, Unquote, takes advantage of F# quotations to allow you to write test assertions as plain, statically checked F# boolean expressions and automatically produces nice step-by-step test failure messages. For example, the following failing xUnit test
[<Fact>]
let ``demo Unquote xUnit support`` () =
test <# ([3; 2; 1; 0] |> List.map ((+) 1)) = [1 + 3..1 + 0] #>
produces the following failure message
Test 'Module.demo Unquote xUnit support' failed:
([3; 2; 1; 0] |> List.map ((+) 1)) = [1 + 3..1 + 0]
[4; 3; 2; 1] = [4..1]
[4; 3; 2; 1] = []
false
C:\File.fs(28,0): at Module.demo Unquote xUnit support()
FsUnit and Unquote have similar missions: to allow you to write tests in an idiomatic way, and to produce informative failure messages. But FsUnit is really just a small wrapper around NUnit Constraints, creating a DSL which hides object construction behind composable function calls. But it comes at a cost: you lose static type checking in your assertions. For example, the following is valid in FsUnit
[<Test>]
let test1 () =
1 |> should not (equal "2")
But with Unquote, you get all of F#'s static type-checking features so the equivalent assertion would not even compile, preventing us from introducing a bug in our test code
[<Test>] //yes, Unquote supports both xUnit and NUnit automatically
let test2 () =
test <# 1 <> "2" #> //simple assertions may be written more concisely, e.g. 1 <>! "2"
// ^^^
//Error 22 This expression was expected to have type int but here has type string
Also, since quotations are able to capture more information at compile time about an assertion expression, failure messages are a lot richer too. For example the failing FsUnit assertion 1 |> should not (equal 1) produces the message
Test 'Test.Swensen.Unquote.VerifyNunitSupport.test1' failed:
Expected: not 1
But was: 1
C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\FsUnit.fs(11,0): at FsUnit.should[a,a](FSharpFunc`2 f, a x, Object y)
C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\VerifyNunitSupport.fs(29,0): at Test.Swensen.Unquote.VerifyNunitSupport.test1()
Whereas the failing Unquote assertion 1 <>! 1 produces the following failure message (notice the cleaner stack trace too)
Test 'Test.Swensen.Unquote.VerifyNunitSupport.test1' failed:
1 <> 1
false
C:\Users\Stephen\Documents\Visual Studio 2010\Projects\Unquote\VerifyNunitSupport\VerifyNunitSupport.fs(29,0): at Test.Swensen.Unquote.VerifyNunitSupport.test1()
And of course from my first example at the beginning of this answer, you can see just how rich and complex Unquote expressions and failure messages can get.
Another major benefit of using plain F# expressions as test assertions over the FsUnit DSL, is that it fits very well with the F# process of developing unit tests. I think a lot of F# developers start by developing and testing code with the assistance of FSI. Hence, it is very easy to go from ad-hoc FSI tests to formal tests. In fact, in addition to special support for xUnit and NUnit (though any exception-based unit testing framework is supported as well), all Unquote operators work within FSI sessions too.
I haven't yet tried Unquote, but I feel I have to mention FsCheck:
http://fscheck.codeplex.com/
This is a port of Haskells QuickCheck library, where rather than specifying what specific tests to carry out, you specify what properties about your function should hold true.
To me, this is a bit harder than using traditional tests, but once you figure out the properties, you'll have more solid tests. Do read the introduction: http://fscheck.codeplex.com/wikipage?title=QuickStart&referringTitle=Home
I'd guess a mix of FsCheck and Unquote would be ideal.
You could try my unit testing library Expecto; it's has some features you might like:
F# syntax throughout, tests as values; write plain F# to generate tests
Use the built-in Expect module, or an external lib like Unquote for assertions
Parallel tests by default
Test your Hopac code or your Async code; Expecto is async throughout
Pluggable logging and metrics via Logary Facade; easily write adapters for build systems, or use the timing mechanism for building an InfluxDB+Grafana dashboard of your tests' execution times
Built in support for BenchmarkDotNet
Build in support for FsCheck; makes it easy to build tests with generated/random data or building invariant-models of your object's/actor's state space
Hello world looks like this
open Expecto
let tests =
test "A simple test" {
let subject = "Hello World"
Expect.equal subject "Hello World" "The strings should equal"
}
[<EntryPoint>]
let main args =
runTestsWithArgs defaultConfig args tests