I know that PIT Mutation Test framework can export mutation coverage information based on the test suite or the test class. However, I was wondering if there is an option to extract or export mutation coverage information based on the test case methods (test cases under the #Test annotation), so that I can see which test cases are written well and which are not. If it is not possible, the simplest solution that comes to my mind is commenting all the test methods and uncommenting only one of the test methods, run it and export the information. I wanted to know if there is an elegant solution.
Note: I know that MuJava provides such information.
This can be done with the (badly/un)documented matrix feature.
Assuming you're using maven you'll need to add
<fullMutationMatrix>true</fullMutationMatrix>
<outputFormats>
<param>XML</param>
</outputFormats>
To your pom.
The XML output will then contain pipe separated test names in the killing test nodes.
<killingTests>foo|foo2</killingTests>
<succeedingTests>bar</succeedingTests>
Related
I'm currently working on a project where I'm using Jest for unit testing and code coverage.
Everything is working fine, except coverage for mocked classes/methods. I don't seem to get the desired coverage results. I've tried to find something in the Jest docs and searched online for an answer, but I can't seem to find anything about it.
The thing is that when I use a mocked implementation (for example ./services/__mocks__/UserService.js), the actual implementation (./services/UserService.js) results in having 0% coverage. This is a logical outcome, since the implementation is overwritten by the mock.
I can get around this by using /* istanbul ignore next */ on every method in the actual service or simply add the actual services to the coveragePathIgnorePatterns property in the Jest setup file and let it generate coverage for all mocked classes instead, but I wonder if there is any way to have Jest use the mocked implementation automatically for generating coverage results.
What is the way to go for mocked classes/functions and code coverage?
Thanks in advance!
As in documentation says for manual mocks, you'll use ./services/__mocks__/UserService.js only if you explicitly called something like jest.mock('./services/UserService');.
If you want to write tests for ./services/UserService, be convinced you don't use jest.mock('./services/UserService'); before this tests.
On a few simple groovy classes I included the junit test in with the class--If you annotate the test methods with #Test (from Junit) and run the main-less class with "groovy MyClass.groovy" it automatically runs the unit tests.
I like this because It really requires zero overhead (no additional files or junk code, just one annotation).
Question is, can I tell it to run a single test method? I tried "groovy MyClass.groovy myMethod" but I didn't really expect that to work. I also tried -Dtest=myMethod which I also didn't have much hope for.
Is there a trick someone knows? I suppose since it is just a .groovy file I could comment out the tests I don't want to run or add a main that calls the various tests, but I'm just wondering if there is a way to leverage this automatic run JUnit tests thing already built into groovy.
Scenario: I need to write a complex nHibernate query, that would return projected DTO, but I want to use TDD approach. The method would look like this:
public PrintDTO GetUsersForPrinting(int userId)
{
Session.QueryOver<User>().//some joins, conditions etc.
//returns projected dto
}
Questions:
Since the most common approach is to use in memory database for this kind of operations. Should I write integration test?
If I am using in memory db can I write Unit tests?
Is one test is enough?
Since my integration test probably will check projection, how should I name it? "GetUserForPrinting_return_correct_DTO" seems too abstract and silly.
I ask because:
There is lots of abstract information about TDD and integration testing, but when it comes to concrete implementation it is very difficult to apply that information.
TDD suggests that integration test should be made of unit tests:
This is not really a very good problem to learn TDD with. I assume you don't already know what the complex query looks like, and you want to use test-driven techniques to drive it out. Awesome :)
But let's see if I can answer your questions.
Yes
any test that includes a real db, whether it is in-memory or on-disk, is not a unit test. A unit test would use a mock db.
Maybe - if you query is complex enough, then no.
testGetUsersForPrinting or getUsersForPrintingTest or similar
Most probably I would drive out the query in a SQL interpreter, not in code. The aim would be to produce a series of integration tests against an in-memory db based on what I learn during this process.
Start from the minimum possible DTO you can think of, and build up from there.
Finally convert the query into nhibernate calls, then make the integration tests pass.
Test-driven, but not really unit-test-driven.
If you are willing to accept maximum TDD discipline and deal with working slower and being more annoyed than usual, you can automate each integration test as you develop it and write code to make it pass. This will mean you are switching frequently among 3 levels of abstraction / editors / environments (direct SQL queries, integration tests, c# code) - I deal with this by setting up techniques to force myself to follow the right steps each time.
This last bit is why this is not a good problem to learn TDD with. You will need a lot of discipline you probably haven't forced yourself to acquire yet!
Good luck.
ok some concrete examples. I would modify your code sample to look like this
public PrintDTO GetUsersForPrinting(int userId, ISession session)
{
var data = session.QueryOver<User>().//some joins, conditions etc.
return data; // or whatever
}
In your unit test you would write
public testDTO()
{
//Arrange
StubSession session = .... setup a stub session, which returns hardcoded values
// Act
PrintDTO users = GetUsersForPrinting(111, session);
// Assert
Assert.That(users.size(), Is.EqualTo(1));
Assert.That(users.get(0).userId, Is.EqualTo(111));
}
In your integration test, you would use a real db, and your session object would actually connect to it, and the queries would be resolved against that db
Arrange-Act-Assert is a standard method for organizing unit tests.
Generally you want as few Asserts as possible in a unit test. And you will have multiple unit tests.
When you are writing a unit test, start by writing the Assert, then fill in the rest to make it compile/get the result you want. Make the test fail first, because then you know you have really delivered something when it passes.
In this example to implement a stub ISession you would derive a local StubSession class (only visible to the test suite) from ISession and just fill in the absolute minimum to get it to compile, and return the minimum data to get the test to pass.
To build up to your whole DTO - assuming you know what you want in your DTO - proceed, as you say in the comments, incrementally. Build up each part of your DTO
a piece at a time, add a unit test for each piece.
Keeping track of this is another piece of TDD discipline.
Set yourself up with a TODO list - just a simple text file, or possibly a lengthy comments at the start of your test suite. List all the things you want to test e.g. zero results, one result, two results, 20 results. User id, whatever other pieces of information you need to have.
If you are doing a complex query across tables or whatever add an todo item for each join, each part of the where clause, etc.
Add items for ordering and paging etc if you are using those.
Pick the simplest things first. Only do one small thing (in a single red-green-refactor cycle) at a time. As you work through your list, you might want to break items up into smaller pieces, or you might think of additional things you need to do. Add them to the TODO list rather than working directly on them.
In this particular case I would swap - after each red-green-refactor cycle - into the SQL environment and/or the sqlite integration test to work out how to make the next piece work. I guess this is a sort of step between red and green - choose what you will test next, write the test (which fails obviously), fiddle around in SQL until you know how to make it pass, write the nHibernate calls to make your test green, then refactor.
Be aware some of the things you list might run out not to be necessary, or take too long, etc. It's good to write them down still, so you know what are not doing as well as what you are doing. Keep focused on your goal.
I tend to also develop a list of "smells" and/or refactorings that I can see I will want to do but am not quite ready for this cycle. Remember to minimise duplication/refactor your tests as well as your SUT (System Under Test).
It's a doing rather then seeing thing. The list of what unit tests you end up with, and the code they exercise, is not a very good description of the journey. Kent Beck's original TDD book is slim and will give you some good overall pointers, but not really about constructing queries.
Does any of that help?
Since the most common approach is to use in memory database for this kind of operations. Should I write integration test?
Using in memory database still is an integration test (because it actually tests if your query generates correct SQL and execute it against a database, see).
If I am using in memory db can I write Unit tests?
No, it would be an integration test
Is one test is enough?
Probably not, you should check each condition of your query, for example one test per one where clause, one for paging and one for sorting if applicable.
Since my integration test probably will check projection, how should I
name it? "GetUserForPrinting_return_correct_DTO" seems too abstract
and silly.
GivenUserForPrinting_WhenGetUserForPrinting_ThenMapToDTO would be a better naming
Consider something like this:
class UsersTable
{
public function findUserById($id)
{
$sql = "...";
return $this->adapter->execute($sql);
}
}
The entire class is pretty much nothing but methods that wrap SQL statements, and there is really no complex logic. I'd essentially be testing the SQL itself.
I know that tests that hit the database are often integration tests, but is this still an integration test since it's being tested so directly, as a unit test would be?
I would say that because your testing more than just the logic of that specific unit (findUserById), that it's an integration test. If you want to properly unit test, I would look into mock objects and dependency injection. Since it looks like you're using PHP, I'll guess you're probably using phpUnit to unit test, and phpUnit allows mocking. To unit test this function, you would want to mock the execute method of the adapter member and assert that it was called once with the correct $sql string. For these purposes, I would say you should assume that the execute method is working correctly, so there's no need to test it. Here is a link to phpUnit's which describes their object mocking.
I find that these integration tests are very useful, and compliment your pure unit tests in a nice way. They give you a good indication that your db is wired up correctly and compatible with your object model.
When I include these types of test I don't tend to mock the db since I can just as easily do all the testing in the integration test.
I do however recommend adding the data your are testing against as a precondition to the test.
I essentially follow this pattern:
1)Add user to DataBase
2)Call your method under test here
3)Assert that your retrieved user is the same as the one just created.
With this pattern you don't have to rely on existing data in the db that might change over time and make your tests fragile.
I was using MSTest and all was fine. Not long ago, I needed to write a large number of data driven unit tests.
Moreover, I needed to know the name of the test just before I run it, so I could populate the data sources with the correct parameters (that were fetched from an external remote service).
Nowhere in MSTest could I find a way to get the name of the tests that are about to run before their actual execution. At this point it was, of course, already too late, since the data sources were already populated.
What I need is to know the names of the test that are about to execute so I could configure their data sources in advance, before their execution.
Somebody suggested I "check out NUnit". I am completely clueless about NUnit. For now I have started reading its documentation but am still at a loss. Have you any advice?
If you really need the test's name -- It's not well documented, but NUnit exposes a feature that let's you get access to the current test information:
namespace NUnitOutput.Example
{
using NUnit.Framework;
[TestFixture]
public class Demo
{
[Test]
public void WhatsMyName()
{
Console.WriteLine(TestContext.CurrentContext.Test.FullName);
Console.WriteLine(TestContext.CurrentContext.Test.Name);
}
}
}
Provides:
NUnitOutput.Example.Demo.WhatsMyName
WhatsMyName
Note this feature isn't guaranteed to implemented by custom TestRunners, like ReSharper. I have tested this in NUnit 2.5.9 (nunit.exe and nunit-console.exe)
However, re-reading your question I think you should check out is the TestCaseSource or TestCase attribute that can be used to parameterize your tests.
If I'm understanding your problem correctly, you want to get the name of the currently-running test so that you can use it as a key to look up a set of data with which to populate the data sources used by the code under test. Is that right?
If that's the case then I don't think you need to look for special functionality in your unit testing framework. Why not use the Reflection API to fetch the name of the currently-executing method? System.Reflection.MethodBase.GetCurrentMethod() will get you a MethodBase object representing the method, and that has a Name property.
However, I'd suggest that using the method name as a key for looking up the appropriate test data is a bad idea. You'd be coupling the name of the method to your data set, and that seems like a recipe for fragile code to me. You need to be remain free to refactor your code and the names of methods without worrying about whether that will break the database lookup behind-the-scenes.
As an alternative, why not consider creating a custom Attribute that you can use to mark those test methods that need a database lookup, and using a property on that attribute to hold the key?
For things like this you should rely on a fixture to initialize the state you want before you run the test.
The simplest way which works in (any) testing framework is to create a fixture which loads any data given a data identifier (string). Then in each test case you just provide the test string for data lookup for the data you want in that test.
Aside from this, it's not recommended to have unit tests access files and other external resources because it means slower unit tests and higher probability of failure (as you're relying on something outside the in-memory code). This of course depends on the amount of data you have and the type of testing you're doing, but I generally have the data compiled-in for unit tests.