Is it possible to use cxxunit or any other unit testing framework (excluding QtTestLib) to test qt widgets?
If yes, then there are two more questions :
How?
Since I am running unit tests using valgrind, can this report some errors?
Yes, it should be possible. I'm not sure about cxxunit specifically, but it is theoretically possible.
To properly test Qt objects, you will probably need to create/destroy a QApplication object in your global setup and teardown functions. Unless you are specifically testing QApplication functionality, you should only create one for the entire run of the test application. This will allow you to test portions of the widget's logic, but not easily the appearance or UI interactivity of the widget. Also, testing certain items may rely on having the application's event loop running, which would be more difficult.
Valgrind may report some errors. It may also report errors with Qt's code, in particular static allocations that are left to application teardown to reclaim.
If you want to test your UI, I suggest to use a UI testing tool like Squish. Unit tests I find more suited to test the logic behind the widgets, not the widgets itself. If you really want to unit-test your Qt widgets, I don't think there is a better solution than QtTestLib.
Valgrind: There is a valgrind plugin for Squish. I haven't used that one myself though. Other unit tests can of course easily be run in valgrind, although I don't know of any solution that fully automates this. One would have to make sure to really suppress all warnings from outside one owns code so that some error in e.g. x11 libs doesn't trigger the unit test to fail.
Related
I realize it may sound like an odd request, and it certainly will not do wonders for test performance, but it's critical that I get a new AppDomain for the start of each unit test.
Currently I'm using xUnit and Resharper as the test runner. But I'm willing to change if there's a different framework that would yield the behaviour that I need.
The xunit resharper runner doesn't have this kind of functionality, and I don't know any test framework that does this out of the box. If you need each test to run in a new AppDomain, I'd write it so that each test created a new AppDomain and ran some custom code in there.
You could probably use some of xunit's features to make this a little easier - the BeforeAfterTestAttribute allows you to run code before and after, or you could pass in a fixture that provides functionality to setup/teardown the AppDomain.
I am using google test in a C++ project. Some functions use assert() in order to check for invalid input parameters. I already read about Death-Tests (What are Google Test, Death Tests) and started using them in my test cases.
However, I wonder if there is a way to suppress the runtime errors caused by failing assertions. At this time each failing assertion creates a pop-up window I have to close everytime I run the tests. As my project grows, this behaviour increasingly disturbs the workflow in an unacceptable way and I tend to not test assert()-assertions any longer.
I know there are possibilities to disable assertions in general, but it seems more convenient to suppress the OS-generated warnings from inside the testing framework.
Ok, I found the solution myself: You have to select the test-style threadsafe. Just add the following line to your test code:
::testing::FLAGS_gtest_death_test_style = "threadsafe";
You can either do this for all tests in the test-binary or for affected tests only. The latter is faster. I got this from the updated FAQ: Googletest AdvancedGuide
We are just about to start a new project. The Proof of Concept (PoC) for this project was done simply using Win32. The plan is/was to flesh out the PoC, tidy the uglier parts and meet the requirements set by the project owners.
One of the requirements for the actual project is 100% code coverage but I can see problems ahead: How can I acheive 100% code coverage with Win32 - the message pump will be exceptionally difficult to test effectively?! I could compile to a DLL but won't there be code in the main app that won't be under coverage?
I am thinking of dropping the Win32 code and moving to MFC - at least then a lot of the boiler plate stuff will be hidden from view (and therefore coverage).
Any thoughts on the problem?
I mean WndProc, but the same applies for WinMain. How can you unit-test that?
I do testing but not unit-testing: I do system/integration testing.
If you exercise your (whole) application while it's running under a debugger/profiler/code coverage analyser, then of course you will find (and the coverage analyser will show) that WinMain etc. are being run (are being covered).
The question then might be, how do you automate the system/integration testing of the whole application? You might have a test framework with automates driving the GUI; I don't know of any myself, but for example there's a list here. Alternatively, it might be acceptable (to the client) if the acceptance test suite is a sequence of non-automated/manual tests.
See also Should one test internal implementation, or only test public behaviour?
I'm just starting to use QTestLib. I have gone through the manual and tutorial. Although I understand how to create tests, I'm just not getting how to make those tests convenient to run. My unit test background is NUnit and MSTest. In those environments, it was trivial (using a GUI, at least) to alternate between running a single test, or all tests in a single test class, or all tests in the entire project, just by clicking the right button.
All I'm seeing in QTestLib is either you use the QTEST_MAIN macro to run the tests in a single class, then compile and test each file separately; or use QTest::qExec() in main() to define which objects to test, and then manually change that and recompile when you want to add/remove test classes.
I'm sure I'm missing something. I'd like to be able to easily:
Run a single test method
Run the tests in an entire class
Run all tests
Any of those would call the appropriate setup / teardown functions.
EDIT: Bounty now available. There's got to be a better way, or a GUI test runner that handles it for you or something. If you are using QtTest in a test-driven environment, let me know what is working for you. (Scripts, test runners, etc.)
You can run only selected test cases (test methods) by passing test names as command line arguments :
myTests.exe myCaseOne myCaseTwo
It will run all inits/cleanups too. Unfortunately there is no support for wildcards/pattern matching, so to run all cases beginning with given string (I assume that's what you mean by "running the tests in an entire class"), you'd have to create script (windows batch/bash/perl/whatever) that calls:
myTests.exe -functions
parses the results and runs selected tests using first syntax.
To run all cases, just don't pass any parameter:
myTests.exe
The three features requested by the OP, are nowadays integrated in to the Qt Creator.
The project will be automatically scanned for tests and they apear on the Test pane. Bottom left in the screenshot:
Each test and corresponding data can be enabled by clicking the checkbox.
The context menu allows to run all tests, all tests of a class, only the selected or only one test.
As requested.
The test results will be available from the Qt Creator too. A color indicator will show pass/fail for each test, along additional information like debug messages.
In combination with the Qt Creator, the use of the QTEST_MAIN macro for each test case will work well, as each compiled executable is invoked by the Qt Creator automatically.
For a more detailed overview, refer to the Running Autotests section of the Qt Creator Manual.
I am attempting add some tests to an existing QT GUI application using QTest. The GUI uses quite a bit of complicated start-up code so I'd rather not write another main() to start it again.
To me, it seems like the easiest way would be instantiate the app and then run the tests on it. I am just not sure, however what function I could plug my test object into so that I don't block the flow of messages.
I could send a special message to start the test or set a timer but that's complicated and tests are supposed to simplify things.
So where would be the best place in existing GUI to insert and qexec a Qtest object?
I'm willing to be proven wrong, but test frameworks in general (and QTest specifically from what I've used of it) tend to assume that the test framework will be driving the code to be tested, as opposed to running along side of it.
I'm also concerned about "The GUI uses quite a bit of complicated start-up code...." Are you intending on testing the startup code? Or testing other stuff around it?
Generally speaking, when I start looking at adding tests to an application, I try to find smaller pieces that are used in a lot of the application, and write tests for those. I then build up to testing the bigger pieces that integrate those smaller ones. My general idea is that if the small pieces work properly, then either I've put them together correctly and things should work, or I haven't and things should obviously fail when I try to run the application.
I should mention that there are other options for testing GUIs for Qt applications in particular. They tend to be more like scripts run on your program, with the output recorded. If that interests you, then you could look into Squish for Qt.