I have a big set of unit and some integration tests implemented with google test framework or gtest.
Since there is no tagging I am using the disable convention to separate tests in groups or prefixing them with GROUPA_, GROUPB_, etc.
This works well. I can filter different groups, to run in different situations etc.
The problem I have is with typed tests that belong to different groups. Since the name of the test is fixed no matter what arguments I pass to the test fixture I cannot assign the same test to more than one group.
My question is, can I control the name of the test somehow runtime before the runner or something. Any other way to control the name of a typed tests?
As a workaround, you can include a typed test into the different groups but for all the types. You could use as many prefixes as needed:
TYPED_TEST(FooTest, GROUPA_GROUPB_Bar)
{
}
Then use filtering strings like FooTest.*GROUPX*_Bar.
I cannot think of a way to map each type the test is instanciated for to a a different group.
Related
It's possible to create a GoogleTest filter without wildcard in it ?
I have a test suite TestSuiteA
TEST_P(TestSuiteA, Blabla) {
// ...
}
INSTANTIATE_TEST_CASE_P(Apple, TestSuiteA, testing::Combine(...));
INSTANTIATE_TEST_CASE_P(Ananas, TestSuiteA, testing::Combine(...));
And a second test suite TestSuiteB
TEST_P(TestSuiteB, Kokoko) {
// ...
}
INSTANTIATE_TEST_CASE_P(Apple, TestSuiteB, testing::Combine(...));
INSTANTIATE_TEST_CASE_P(Ananas, TestSuiteB, testing::Combine(...));
I know precisely which tests I want to launch, so I don't want use the wildcard.
My filter
Apple/TestSuiteA:Apple/TestSuiteB
But it doesn't work -> 0 tests to launch
But with this
Apple/TestSuiteA.*:Apple/TestSuiteB.*
It works
What does this .* mean and why it doesn't work without ?
Thanks
Each tests needs to have a unique name, and you haven't provided the exact correct name. That's why no tests were run. Make sure you know how exactly the tests are named - you can use --gtest_list_tests to list all the test cases defined in a given test executable. Then use the exact names with --gtest_filter separated with :. To read more how to run multiple tests, see gtest advanced documentation.
I have a large suite of tests that are currently running in series. I would like to make the suite parallelizable as much as possible. One big problem I have is that some tests require certain global application state to be set one way, and some require it to be set another way. Is there a way I can make the NUnit test runner group my tests based on which global state they require, and then run the tests within each group in parallel with each other, and modify the global state between groups?
For example, let's say there is a global setting Foo and a custom attribute [RequiresFoo(X)] that can be used to annotate which Foo value a test requires. At runtime I want NUnit to group all tests by their argument to RequiresFoo, counting unmarked tests as having some default Foo value.
Then for each group, I want it to
Set Foo = N where N is the Foo value for that group.
Run all tests in that group in parallel.
In an ideal world I would have a mock system for this global state, but that would take a lot of time that I don't have right now. Can I get NUnit to do this or something like it?
Note, I need to be able to execute any method between groups, not just set a variable. The global context I'm actually dealing with can involve starting or stopping microservices, updating configuration files, etc. I can serialize any of these requirements to a string to pass to a custom attribute, but at runtime I need to be able to run arbitrary code to parse the requirements and reconfigure the environment.
Here is a pseudo-code example.
By default tests execute in series like this:
foreach (var t in allTests)
{
Run(t);
}
NUnits basic parallel behavior is like this:
Parallel.ForEach(allTests, t => Run(t));
I want something like this:
var grouped = allTests
.GroupBy(t => GetRequirementsFromCustomAttributes(t));
foreach (var g in grouped)
{
SetGlobalState(g.Key);
Parallel.ForEach(g, t => Run(t));
}
As you state the problem, it's not yet possible in NUnit. There is a feature planned but not yet implemented that would allow arbitrary grouping of tests that may not run together.
Your workaround is to make each "group" a test, since NUnit only allows specification of parallelization on tests. Note that by test, we mean either a test case or a group of tests, i.e. fixture or namespace suite.
Putting [Parallelizable] on a test anywhere in the hierarchy causes that test to run in parallel with other tests at the same level. Putting [NonParallelizable] on it causes that same test to run in isolation.
Let's say you have five fixtures that require a value of Foo. You would make each of those fixtures non-parallelizable. In that way, none of them could run at the same time and interfere with the others.
If you want to allow those fixtures to run in parallel with other non-Foo fixtures, simply put them all in the same namespace, like Tests.FooRequred.
Create a SetupFixture in that namespace - possibly a dummy without any actual setup action. Put the [Parallelizable] attribute on it.
The group of all foo tests would then run in parallel with other tests while the individual fixtures would not run in parallel with one another.
When using sinatra-r18n to handle internationalisation, the r18n lib exposes a variable t for use within your helpers, routes and templates, as per these instructions.
I have written a simple unit test using rack-unit to confirm that some of my pluralisations work but the test throws an error claiming t is nil.
I've tried referencing it via app.t, MySillyApp.t (where MySillyApp is the name of my Sinatra app), MySillyApp.settings.t etc and none of them give me access to the t I need.
What I am trying to achieve is a confirmation that my translation files include all the keys I need corresponding to plurals of various metric units my app needs to understand. Perhaps there is a more direct way of testing this without going via the Sinatra app itself. I'd welcome any insight here.
I had similar task to check localized strings in my Cucumber scenarios.
I've made working example.
Here you can find how strings got translated.
This file halps to understand how to add R18n support to your testing framework:
require 'r18n-core'
...
class SinCucR18nWorld
...
include R18n::Helpers
end
As you can see instead of rack/unit I'm using RSpec/Cucumber, sorry.
I'm applying TDD to my first event centric project (CQRS, Event sourcing etc) and I'm writing my tests according to Greg Young's simple testing framework Given, When, Expect. My test fixture takes a command, commandhandler and aggregate root and then tests the events outputted.
CommandTestFixture<TCommand, TCommandHandler, TAggregateRoot>
For example here is a typical test
[TestFixture]
public class When_moving_a_group :
CommandTestFixture<MoveGroup, MoveGroupHandler, Foo>
I am very happy with these tests on the whole but with the the above test I've hit a problem. The aggregate root contains a collection of groups. The command MoveGroup reorders the collection, taking a from & to index. I setup the test and asserted that the correct GroupMoved event was generated with the correct data.
As an additional test I need to assert that the reordering of the Groups collection actually took place correctly? How do I do this when the aggregate root has no public getters/setters. I could add a method to retrieve the group at a particular index but isn't this breaking encapsulation simply to be testable?
What's the correct way to go about this?
EDIT
The reordering of the groups takes place in the GroupMoved handler on the Aggregate root.
private void Apply(GroupMoved e)
{
var moved = groups[e.From];
groups.RemoveAt(e.From);
groups.Insert(e.To, moved);
}
The friction here comes because you want to assert something about the internal implementation, but what you have at hand is at the top level.
Your tests and assertions need to be at the same logical level. There are two ways to re-arrange this:
What effect does re-ordering groups have on subsequent commands or queries which you do have at the top level?
This should give you an avenue for asserting that the correct outcome occurs without needing to assert anything about the ordering of the groups directly. This keeps the test at the top level and would allow all sorts of internal refactoring (e.g. perhaps lazy sorting of the groups).
Can you test at a lower level?
If you feel that testing as described above is too complicated, you might want to frame your test at a more detailed level. I think of this like focusing in on a section of detail to get it right.
Down at this level (rather than your composite root), the interfaces will know about groups and you'll have the opportunity to assert what you want to assert.
Alternatively, do you need this test at all?
If you can not find a suitable test at either of the above levels, then are you sure you need this test at all? If there is no visible external difference then there is no need to lock the behaviour in place with a test.
Is it possible to have a fixture change between test methods? If so, how can I do this?
My syntax for this problem :
In the cakephp framework i am building tests for a behavior that is configured by adding fields to the table. This is intended to work in the same way that adding the "created"
and "modified" fields will auto-populate these fields on save.
To test this I could create dozens of fixtures/model combos to test the different setups, but it would be a hundred times better, faster and easier to just have the fixture change "shape" between test methods.
If you are not familiar with the CakePHP framework, you can maybe still help me as it uses SimpleTest
Edit: rephrased question to be more general
I'm not familiar specifically with CakePHP, but this kind of thing seems to happen anywhere with fixtures.
There is no built in way in rails at least for this to happen, and I imagine not in cakePHP or anywhere else either because the whole idea of a fixture, is that it is fixed
There are 2 'decent' workarounds I'm aware of
Write a changefixture method, and just before you do your asserts/etc, run it with the parameters of what to change. It should go and update the database or whatever needs to be done.
Don't use fixtures at all, and use some kind of object factory or object generator to create your objects each time
This is not an answer to my quetion, but a solution to my issue example.
Instead of using multiple fixtures or changing the fixtures, I edit the Model::_schema arrays by removing the fields that I wanted to test without. This has the effect that the model acts as if the fields was not there, but I am unsure if this is a 100% test. I do not think it is for all cases, but it works for my example.