I'm trying to write a test to verify a compiling error. It's about assigning a number to a String type property. But since it's a compiling error, the unit test code won't even be compiled at the first place. So anyone have any suggestion on how to do this?
I'm thinking maybe I can assign the number in runtime and check to see if certain exception got thrown? But I'm not sure how exactly to do that.
Thanks in advance!
If I understand you correctly, you have some piece of code that may not compile and you want to write the unit test that fails if the code really doesn't compile. If that is the case, then you should not write any unit tests. You should understand that you should write unit tests only for your code, not the code that somebody else wrote.
You didn't specify the programming language, so I will do some pseudo-code. Say you are writing a function to add two numbers:
function add(a, b) {
return a + b;
}
Very simple. You should unit test this, e.g. by making tests like the below:
function testAdd() {
assertEquals(4, add(2, 2));
assertEquals(46, add(12, 34));
}
However, you should not write a test that tests whether + operator is working fine. This is the job of whoever wrote the library that implements how + operator really works.
So, if that's the case, don't write any unit tests. Compiling your code is job of your compiler. The compiler should report that there is a compilation error in a proper way. You should not test whether compiler does its job correctly - testing that is the job of the people who are writing the compiler.
You did not specify the language
Ruby, Python -- dynamic & strong typesystem. This means that the types deduced during runtime (dynamic), but implicit conversions between types are prohibited
Js, Perl -- dynamic & weak typesystem.
C++ -- static and strong typesystem.
Let's assume we are talking about C++. Moreover I can create more really example. Imagine that you implement static assert for your project which doesn't not use c++11 compiler
template <bool>
struct my_static_assert;
template <>
struct my_static_assert<true> {};
If you want to check that such mechanism works ok. You should create unittest function which do the follow steps:
Create some file to compiler
Create external process of the compiler and pass test compile unit to it
Wait for compiler process completion
Retrive return code from the compiler process
Your function check return code from 4.
I checked google-test guide but it seems that they doesn't support such concept https://github.com/google/googletest/blob/master/docs/advanced.md
Related
I have a class A where I deleted the default constructor.
class A {
public:
A() = delete;
A(int a): m_myInt(a) {}
private:
const int m_myInt;
};
int main () {
A foo(1); // works perfect
A bar; // won't compile
}
How do I write a good unit test ensuring that A bar; remains not valid? I could write a not compiling test and take the compile error as a test requirement. I wonder, if there is a better way to write a unit test?
2) If std::is_trivially_constructible<T>::value is true, provides the member constant value equal to true, otherwise value is false.
http://en.cppreference.com/w/cpp/types/is_default_constructible
There is nothing wrong with a unit test that hinges on confirming that some code constructs fail to compile. The purpose of unit testing is determining if one or more sections of code are collectively "fit for use" - and the criteria can represent functional or non-functional (e.g. quality) attributes. If a "fit for use" criterion is "code will not compile if ....." then an obvious approach is to write a unit test that deliberately seeks to cause a failed compilation.
This is certainly a valid unit testing approach for a compiler - in which testing how the compiler responds to samples of bad code is completely appropriate as a unit test, or set of unit tests.
Given that the requirement is that default-instantiation of an A (to use the OP's words) "remains not valid", the basic criterion for a passed unit test would be a failed compilation of code which performs
A bar;
Depending on requirement, such code may need to be tested in different contexts (e.g. within an instrumented function that is a member or friend of A, in a unrelated function, at file scope, etc). So this requirement may require a set of unit tests.
To implement such a test, it would be necessary for the build process to capture whether compilation fails. A successful compilation would need to be captured as a failed test and, conversely, a failed compilation would need to be captured as a passed unit test.
Depending on how much evidence needs to be captured to justify a claim of a passed (or failed) unit test, the build process may need to capture and parse the compiler diagnostics - for example, to determine which lines in a source file actually cause compilation to fail.
Equally, there would probably need to be a unit test, or set of unit tests, that check some code constructs do compile.
You might use traits for that:
static_assert(!std::is_constructible<A>::value, "!");
For the development of my own Pass I want to write unit tests - i have lots of 'pure' helper methods, so they seem ideal candidates for unit test. But some of them require an instance of llvm::LoopInfo as an argument.
In my (Function-)Pass I just use
void getAnalysisUsage(llvm::AnalysisUsage &AU) const override {
AU.setPreservesCFG();
AU.addRequired<llvm::LoopInfoWrapperPass>();
}
...
llvm::LoopInfo &loopInfo = getAnalysis<LoopInfoWrapperPass>(F).getLoopInfo();
to get this information object.
In my unit test I currently parse my llvm::Function void foo() (that I want to run my analysis on) from disk like this:
llvm::SMDiagnostic Err;
llvm::LLVMContext Context;
std::unique_ptr<llvm::Module> module(parseIRFile(my_bc_filename, Err, Context));
llvm::Function* foo = module.operator*().getFunction("foo");
to finalize my test I would have to fill in following stub:
llvm::LoopInfo& = /*run LoopInfoWrapperPass on foo and return LoopInfo element */;
My first attempts were based on using the PassManager<Function> (in Header "llvm/IR/PassManager.h"), AnalysisManager<Function>, and the class LoopInfoWrapperPass, but I couldn't find any example usage online for LLVM 4.0 - and older examples seemed to be using a previous version of PassManager, and I did not see how to make use of the LegacyPassManager. I tried to look into the sources for PassManager but could not make enough sense of the typedefs and template arguments (and they are increasing my irrational dislike for C++ as a language).
How do I fill in that stub? How do I call this Analysis Pass (and get LoopInfo) in my plain C++ code?
PS: There are more passes other than LoopInfoWrapperPass I need to use, but I'm assuming the way should be transferable to any Analysis Pass.
PPS: I'm using googletest as a unit test framework, with a CMake build configuration that makes the unit tests their own target, and I'm building my Pass out-of-tree against binary libs of LLVM 4.0.1, if any of that is somehow relevant.
I am not sure how you have your unit tests structured, but looking around in the LLVM source tree is a good idea.
One example can be found in CFGTest.cpp here.
You need to create the PassManager and the pipeline yourself. From my short experience on this, it works well for small tests, but once you need anything bigger or pass data in/out it's really restricting, since the LoopInfo data have only meaning within the pipeline (aka runOn() methods and friends).
Otherwise, you might want to opt (no pun intended) for the simpler, IMHO, method of creating the set of the required analysis yourself (only dominators in the case of LoopInfo) without using the pass manager infrastructure. An example of this can be seen here.
Hope this helps.
Say I have the following two functions:
add_five (number) -> number + 2
add_six (number) -> add_five(number) + 1
As you can see, add_five has a bug.
If I now test add_six it would fail because the result is incorrect, but the code is correct.
Imagine you have a large tree of functions calling each other, it would be hard to find out which function contains the bug, because all the functions will fail (and not only the one with the bug).
So my question is: should unit tests fail because of incorrect behaviour (wrong results) or because of incorrect code (bugs).
should unit tests fail because of incorrect behaviour (wrong results) or because of incorrect code (bugs)?
Unit tests usually fail because of wrong results. That's what you write in assertions: you call a method and you define the expected result.
Unit tests cannot identify incorrect code. If the operation is return number+5 and your CPU or your RAM has a hardware problem and return something different, then the test will fail as well, even if the code is correct.
Also consider:
public int add_five(int number)
{
Thread.Sleep(5000);
return number+5;
}
How shall the unit test know whether the Sleep is intended or not?
So, if any unit test fails, it's your job to look at it, find out why it fails and if it fails in a different method, write a new unit test for that method so you can exclude that method next time.
Presumably you have a test for add_five/1 and a test for add_six/1. In this case, the test for add_six/1 will fail alongside the test for add_five/1.
Let's assume you decide to check add_six/1 first. You see that it depends on add_five/1, which is also failing. You can immediately assume that the bug in add_five/1 is cascading up.
Your module dependencies form a directed (hopefully acyclic) graph. If a dependency of you function or module is broken, that should be what you target for debugging first.
Another option is to mock out the add_five function when testing your add_sixfunction, but this quickly creates a lot of extra typing and logic duplication. If you change the spec of add_five, you have to change every place you reimplemented it as a mock.
If you use a quickcheck-style testing library, you can test for certain logic errors based on the properties of what you're testing. These bugs are detected using randomly generated cases that produce incorrect results, but all you as a tester write are library-specific definitions of the properties you're testing for. However, this will also suffer from dependency breakages unless you've mocked out dependent modules/functions.
I've got a problem when unit testing my program.
The problem is simple but i'm not sure why this is not working.
1 -> i build all my program
2 -> i build my unitTest
3 -> the test is running.
All is ok when it is not about getting global data from the data segment. It seems as if the variable are not initialized / or simply found. So of course all my tests become wrong.
My question is:
Is it totally wrong to build an executable, then running the test on it? Or should i must compile all my code + the unit test in the same time, and then running it? Or is it just a lack of SenTesting framework?
I forgot to mention that this is a C++ const string. Dunno if that change something.
*EDIT***
My assumption was wrong, but i still don't understand the magic beyond! Seems a C++ magic hoydi hoo?
char cstring[] = "***";
std::string cppString = "***";
NSString* nstring = #"***";
- (void)testSync{
STAssertNotNil(nstring, nil); // fine
STAssertNotNil((id)strlen(bbb), nil); // fine
STAssertNotNil((id)cppString.size(), nil); // failed
}
EDIT 2**
Actually this is normal that the C++ is not initialized at this part of the code. If i do a nm on my executable, it appears that my C and Obj-C global are put into the dataSegment. I thought my C++ string was in the same case, but it is actually put into the bss segment. That's means it's uninitialized. The fact is the C++ compiler do some magic beyond and the C++ string is initialized after the main() call and act like if it were into the dataSegment.
I didn't know that testSuit doesn't have main() call, so the C++ object are never initialized. There is some technique in order to call the .ctor before the testSuit. But i am too lazy too explain and it's some kind of topic. I have just replaced my C++ string with a simple char array, and it work perfectly since my value are now POD.
By the way there is no devil in global variable if they are just read-only. ;)
OK, I can see a few faults here.
First of all, this code gives errors on my environment (Xcode 5) and for good reasons (with ARC enabled). I don't know how you got the thing to compile. The reason is that you are casting an integer (or long) to an object, and this will result in many errors, as it is normally an invalid operation. So, the real question is not why the third "assert" failed, but why the second one succeeded.
As far as the second part of your question is concerned, I have to admit that I do not completely understand your question, and you may have to explain it more thoroughly.
In general, unit testing is testing specific parts of your code. Therefore, you typically don't perform the tests on an actual final executable (this is not called unit testing, I believe), nor do you have to compile "all your c++ code + your unit tests at the same time".
Since you are using Xcode, I will give you some indications.
Write your application (or at least a part of it), and find the aspects / functions / objects you want to perform unit tests on.
In separate files, write unit tests that instantiate these objects and test their methods, call them and compare the inputs and outputs.
You should have a second target in your application, that will compile only the unit test source code and the relevant main program code.
Build this target, or press command-U and it will report successes and failures.
So, you need to separate your source code and isolate your classes / methods to make them testable like this. This needs a good architecture and design of the application on your part, and you may need to make some compromises in flexibility (that is up to you to decide). Oh, and I believe that in a testable code you should avoid global variables in general, for various reasons. Global variables are helpful sometimes, but they generally make unit testing really difficult, (and if misused may lead to spaghetti code, but this is a whole different story)
I hope I helped, even without fully understanding the second part of your post.
Suppose I am writing a template library consisting of a function template
template<T> void f(T);
with the requirement that it works with a predefined set of classes A, B, C, and D, e.g., the following must compile:
template<> void f(A);
template<> void f(B);
template<> void f(C);
template<> void f(D);
Which test framework can I use to write test cases that captures this requirement at runtime instead of failing at compilation of the test code? In another word, I would like the framework to instantiate the templates at runtime and produce a nicely formatted error report if a subset of them fails.
I know I can forego test frameworks altogether and simply write a simple cc file containing the 4 lines above. But I was hoping I could incorporate this requirement into regular, standard test cases for generation of test status reports. For example,
test f works with A: passed.
test f works with B: passed.
test f works with C: failed! Cannot cast type C!
test f works with D: passed.
3 of 4 tests passed.
1 of 4 tests failed.
Write a test case that spawns the compiler... that's how e.g. autoconf tests for existence of features.
I don't understand why failing at runtime is preferable to failing at compile time. The earlier you fail in the unit testing process the better. It is preferable to have your unit tests not compile than fail. Its even easier to fix, In fact it probably won't even be committed to source control. Your unit test should just include those four lines and assert true at the end. Note this isn't the way I would go about doing it myself.
C++ templates are a compile time feature. In many cases they will fail at compile time, by design. You simply can't get around this without doing something really crazy.
However, you're also going to want to know that your template specializations are correct, because the specializations override the behavior you would otherwise get from the template. So test the specializations. But realize you will never get around the compile time aspects of templates.
Based on what you're trying to test here, checking if the thing can compile is the only sensible test you can perform.
Testing should not be for the sake of testing, but to ensure functional correctness. If you want to have proper tests around your class, you should write tests that verify the functionality of your template with all of the 4 different classes it can be compiled with.