I have a complex app and we do write test cases for testing.
While writing a test case for one case, we came to know about this issue.
I'm having a file in which two acceptance tests are written, something like this:
module() {
test('acceptance 1', function() {
...logic
}
test('acceptance 2', function() {
...logic
}
}
As we all know, whenever a property changes both beginPropertyChanges and endPropertyChanges function will be called. What happens in my case is, when the acceptance 1 test is running it increments the deferred variable in beginPropertyChanges and due to some error endPropertyChanges haven't triggered. Due to this all observers are getting broken. (As deferred variable will be 1 and not zero)
function beginPropertyChanges() {
deferred++;
suspendedObserverDeactivation();
}
function endPropertyChanges() {
deferred--;
if (deferred <= 0) {
flushSyncObservers();
resumeObserverDeactivation();
}
}
So when acceptance 2 test run, it fails due to observers brokage.
How I can fix this (how to make deferred as zero when test 2 is running) so that test 2 will get passed?
Related
I'm writing a generator. I'm testing it with RITEway. It checks if window.ethereum is defined. If its not, it should throw and stop. Basically it should satisfy the following tests:
describe('handle initialize Web3 saga', async assert => {
global.window = {}
assert({
given: 'nothing, the window object',
should: 'have no property called Web3',
actual: window.web3,
expected: undefined
})
const gen = cloneableGenerator(handleInitializeWeb3)()
{
// The important parts are in this block scope
const clone = gen.clone()
assert({
given: 'window.ethereum undefined',
should: 'throw',
actual: clone.next().value.message,
expected: '[WARNING]: window.ethereum has no provider!'
})
assert({
given: 'nothing',
should: 'be done',
actual: clone.next().done,
expected: true
})
}
class Provider {}
window.ethereum = new Provider()
// ... more tests
})
Here is how I tried implementing it.
function* handleInitializeWeb3() {
if (!window.ethereum) {
yield new Error('[WARNING]: window.ethereum has no provider!')
}
// ... more yields
}
but this saga doesn't stop. The test where it should: 'be done' fails and the saga gives back the values from the yields outside of the if statement. How can I have these tests pass and the saga stop when the error is thrown?
yielding an error instance acts the same as yielding any other value (i.e. the generator keeps running). If you want to stop the generator you should throw new Error(... like in a normal function.
If for some reason you don't want to throw and do in fact want to yield an error instance and then stop, simply return; after you've yielded the error.
I am using Artos runner. In our development environment we keep
<property name="stopOnFail">true</property> so I can debug my changes without having to deal with dependent test cases failing. In production environment we keep <property name="stopOnFail">false</property> so test execution does not stop upon failure and we can analyse log in the morning.
Now I have a different requirement,
I have some tests that are pre-requisite for rest of the units, so if critical test fails then I would like to skip rest of the unit, otherwise it can put our product into bad state.
Is there a way in Artos to skip rest of the unit only if specific test case or test unit fails?
Or can we perform specific steps, incase test fails to ensure we are safe to perform rest of the tests?
Depending on the requirement there are multiple ways to achieve it in Artos
First of all, ensure all of your units are having a sequence number so they execute in the same order all the time.
Let's say testUnit_1() is critical unit and it must be executed successfully in order to execute rest of the following units. In that case, set dropRemainingUnitsUponFailure = true as shown below. This will ensure that the rest of the units are dropped from the execution list if testUnit_1() fails.
#TestPlan(preparedBy = "user", preparationDate = "19/02/2019", bdd = "GIVEN..WHEN..AND..THEN..")
#TestCase(sequence = 1)
public class TestCase_1 implements TestExecutable {
#Unit(sequence = 1, dropRemainingUnitsUponFailure = true)
public void testUnit_1(TestContext context) {
context.getLogger().info("do something");
}
#Unit(sequence = 2)
public void testUnit_2(TestContext context) {
context.getLogger().info("do something");
}
#Unit(sequence = 3)
public void testUnit_3(TestContext context) {
context.getLogger().info("do something");
}
}
If the test cases that are dependent upon each other then you can do similar at test case level
Ensure TestCases are assigned with sequence number so they follow same execution order (Similar to units)
As shown below. If dropRemainingTestsUponFailure = true and dropRemainingUnitsUponFailure = true then upon testUnit_1() failure, not only rest of the units will be dropped but also remaining test cases will be dropped from execution list so you can achieve clean exit.
#TestPlan(preparedBy = "user", preparationDate = "19/02/2019", bdd = "GIVEN..WHEN..AND..THEN..")
#TestCase(sequence = 1, , dropRemainingTestsUponFailure = true)
public class TestCase_1 implements TestExecutable {
#Unit(sequence = 1, dropRemainingUnitsUponFailure = true)
public void testUnit_1(TestContext context) {
context.getLogger().info("do something");
}
#Unit(sequence = 2)
public void testUnit_2(TestContext context) {
context.getLogger().info("do something");
}
#Unit(sequence = 3)
public void testUnit_3(TestContext context) {
context.getLogger().info("do something");
}
}
In the log file you will see warning
=========================================================================
========== DROP REMAINING UNITS UPON FAILURE IS TRIGGERED ===============
================== REMAINING UNITS WILL BE DROPPED ======================
=========================================================================
and
=========================================================================
========== DROP REMAINING TESTS UPON FAILURE IS TRIGGERED ===============
================== REMAINING TESTS WILL BE DROPPED ======================
=========================================================================
so you will know what happened.
To Answer your second question
(Question: If there is any ways to perform clean up if test unit fails, so before you perform next unit test, you can recover your product from the bad state)
If I understood it correctly then it can be done using annotation #AfterFailedUnit
If you create a method as shown below in your runner class
#AfterFailedUnit
public void globalAfterFailedTestUnit(TestContext context) throws Exception {
context.getLogger().info("This method executes after failed test unit");
}
then it will be executed after each test unit failure, you should implement clean up logic in this method.
Hopefully this answers your questions
I'm using Mocha to test a method that has an asynchronous method inside of it. I'm stubbing that dependency with Sinon, and returning a resolved promise. But the promise is never resolved, or at least when the assertion is run it hasn't resolved yet.
Here is the method under test
function identify(traits) {
//THIS GETS CALLED SUCCESSFULLY
userService.get().then(function(user){
//CODE NEVER REACHES HERE
userService.set(user).then(function(){
//do something
}, function(){
//handle error
});
});
}
And here is the test
it('should identify a valid email address', function(){
var user = { email: 'test#example.com' };
var getUserStub = sinon.stub(userService, "get");
var setUserStub = sinon.stub(userService, "set");
var userReturn = { email: 'test#example.com', urls: ['http://some.url.com'] };
getUserStub.returns(Promise.resolve(userReturn));
//THE METHOD UNDER TEST
identifyController.identify(user);
sinon.assert.calledOnce(userService.get); //WORKS FINE
sinon.assert.calledOnce(userService.set); //FAILS
getUserStub.restore();
});
The assertion on userService.set fails, it says it was called 0 times. What am I doing wrong?
I've finally found the problem.
Promises are essentially asynchronous, but sinon calls are synchronous.
See this code pen.
What happens:
You call identifyController.identify(user);
It will call get, which returns a promise, which is asyncrhonous.
The main thread of the program will still be running, so your both sinon.assert.calledOnce calls will happen in sequence, synchronously
By the time get resolves itself and calls set, because promises are non-blocking, the assertion will already have been executed, so it will fail.
So, you can do like this:
function identify(traits) {
return userService.get().then(function(user){
console.log('get');
userService.set(user).then(function(){
console.log('set');
//do something
});
});
}
And change this:
identify(user).then(function(){
sinon.assert.calledOnce(myObj.get); //WORKS FINE
sinon.assert.calledOnce(myObj.set); //WORKS FINE NOW
});
I'm using NSubstitute to mock a class that my method under test uses. This mocked class may throw a particular exception under certain conditions.
The method that I'm testing has some "retry" logic that it executes when it catches this exception. I'm trying to test this retry logic. So, I need a particular method of this mocked class to throw the exception sometimes, but not other times. Unfortunately, the method that throws this exception has no parameters, so I can't base the throw logic on parameters.
How can I make the mocked object's method throw the exception either:
A) ...the first N times it's called
or
B) ...based on the parameters some other method that's called before it
or
C) ...under any other condition other than the parameters passed in
To give you a clearer picture of what I'm trying to do, my code is something like:
IDataSender myDataSender = GetDataSender();
int ID = GetNextAvailableID();
myDataSender.ClearData();
myDataSender.Add(ID,"DataToSend");
bool sendSuccess = false;
while (!sendSuccess)
{
try
{
myDataSender.SendData();
sendSuccess = true;
}
catch (IDCollisionException)
{
ID++;
MyDataSender.ClearData();
myDataSender.Add(ID,"DataToSend");
}
}
So, I need to test my retry logic, and I need to simulate that IDCollisionException. However, I can't have the SendData() throwing the exception every single time, or the retry loop will never succeed.
What can I do here?
If I understand the question correctly, you can use When..Do and close over a local variable to get this behaviour.
const int throwUntil = 3;
var callsToSendData = 0;
var dataSender = Substitute.For<IDataSender>();
dataSender
.When(x => x.SendData())
.Do(x =>
{
callsToSendData++;
if (callsToSendData < throwUntil)
{
throw new DbCollisionException();
}
});
Similarly, you can also use callbacks to locally capture parameters passed to other methods, and access them within the Do block (rather than just using a counter).
I've got a number of Boost test cases ordered in several test suites. Some test cases have one, some more than one check.
However, when executing all tests, they all get executed – no matter how many fail or pass. I know, that I can stop the execution of one test case with several checks by using BOOST_REQUIRE instead of BOOST_CHECK. But that's not want I want.
How can I tell Boost to stop the whole execution after the first test case failed? I would prefer a compiled solution (e.g. realized with a global fixture) over a runtime solution (i.e. runtime parameters).
BOOST_REQUIRE will stop the current test case in a test suite but go on on others.
I don't really see what you wanted when you asked for a "compiled solution" but here is a trick that should work. I use a boolean to check the stability of the whole test suite. If it is unstable i.e a BOOST_REQUIRE has been triggered then I stop the whole thing.
Hope it can help you.
//#include <...>
//FIXTURES ZONE
struct fixture
{
fixture():x(0.0),y(0.0){}
double x;
double y;
};
//HELPERS ZONE
static bool test_suite_stable = true;
void in_strategy(bool & stable)
{
if(stable)
{
stable = false;
}
else
{
exit();
}
}
void out_strategy(bool & stable)
{
if(!stable)
{
stable = true;
}
}
BOOST_AUTO_TEST_SUITE(my_test_suite)
//TEST CASES ZONE
BOOST_FIXTURE_TEST_CASE(my_test_case, fixture)
{
in_strategy(test_suite_stable);
//...
//BOOST_REQUIRE() -> triggered
out_strategy(test_suite_stable);
}
BOOST_FIXTURE_TEST_CASE(another_test_case, fixture)
{
in_strategy(test_suite_stable); //-> exit() since last triggered so stable = false
//...
//BOOST_REQUIRE()
out_strategy(test_suite_stable);
}
BOOST_TEST_SUITE_END()
Benoit.
Why not just use assert? Not only you abort immediately the whole program, you will also be able to see stack if necessary.