I have a task to automate Qt QML based desktop applications through open source frameworks. I am completely new to testing and also to Qt QML. I am good with Java and Python and also Javascript.
Can someone please suggest a way to write automation tests for Qt QML desktop application?
Can I code things to read the elements and widgets in the Qt QML applications?, So I can validate various cases.
If so, are there any tutorials or sample automation test projects for Qt QML applications?
Have a look at Spix. Once you link against it, you can remote control your QML UI by sending fake events, either from C++, or from a script using RPC. Any scripting language that supports XML-RPC, like python, can be used.
This way, you can automate and test your Qt/QML application.
Objects are identified by paths, so you don't have to deal with coordinates. Spix finds the item by the objectName property set in QML.
You can also query object properties from the script...
The library is fairly new and under development, but I think it already covers a lot of the more common use cases in desktop apps...
I am having a product which runs on Vxworks on end product hardware. But development is done in Visual studio using cross compiler and downloaded to hardware for testing. I am planning to write unit test cases for product. My question is because my development is done on windows and how can I run unit test cases as it is not resemebling real scenario?
Any inputs are welcome
I suspect that you have code that interacts a lot with VxWorks through system calls. Putting a layer of abstraction in there will be hard.
Are you using c or c++?
If you are using c++ and you can identify parts of the system that:
are subject to frequent change; and
are mostly handling internal data; or
just relating to a predefined/formalized subset of the surrounding system (e.g. protocol handling or individual PLC control logic modules).
Then you should first inject c++ interface(s) between the module and the rest of the system. This module should only relate to the interface(s)/adapter(s). Then you have an isolated piece that can be strapped into a visual studio test harness.
Then you should try to identify areas in your system that are error prone to bugs, subject to (frequent) change or audit. You will probably never achieve even 50% coverage on the target system, but you can achieve a system where 90% of the daily coding happens within the covered 40% of the code base.
Not possible without additional effort (new project; compile twice; for your host and let it run on your host) In this case search for an development environment that support your target for unit tests, for example http://www.parasoft.com/jsp/products/embedded_cpptest.jsp
Go on reading at parasoft C++ unit test question .
You do it just like any other system:
Write the unit tests
compile and load into target system
run unit tests
verify results
Where it runs is immaterial. The biggest difficulty is on an embedded system with limited output capability. But even if there is only a single LED, it should still be possible to signal success and failure. Only it is a little more abstract than showing "passed".
What tools are generally used for unit testing and especially continuous integration for embedded systems?
I am especially thinking that you usually have to cross-compile and deploy, and also that you can't easily visualize the target platform. Also it can be difficult to run test-code and frameworks.
What could I use too alleviate these difficulties?
(I think it should be some kind of dual targeting, where the build server runs its tests on a easier target)
For unit testing, take a look at Unity.
http://sourceforge.net/apps/trac/unity/wiki
It is a really lightweight test harness (2 x .h and 1 x .c file) supported by Ruby scripts. We have been using in an embedded ARM7 target system for unit testing (redirecting test reporting over a serial port).
It is also supported by CMock for (surprise, surprise) Mocking. Even though not extensive, the great thing about these is they are so easy to use.
Regarding CI, then Hudson (now Jenkins) is very good if you're Linux based.
Also look at CppUTest and check out James Grenning's book "TDD for Embedded C" at http://renaissancesoftware.net/
At work I use the embUnit framework:
http://embunit.sourceforge.net/embunit/index.html
The nice thing about this framework is, that it's lean. It does not require any external libraries (not even libc). You can hook your own output function with ease so if you work on a system where the only connection to the outside world is jtag or an UART, then embUnit will still work.
I have used RCUNIT and CANTATA++ for unit testing embedded code on the PC. Any Nunit should easily integrate into any continuous test platform. We found it an lot easier to just simulate the hardware on the PC and only test on the target during final integration.
Hardware interface abstraction is crucial for unit testing embedded code on the PC. This works well with continuous integration since it is run on a pc with just the hardware access simulated. With a little effort we could test 95% of the code on a PC for continues integration.
You could also look at these questions:
Unit Testing C Code
Testing Frameworks for C
Unit Testing Embedded Software
I've seen C(pp)unit used on a system that let you launch to the target via JTAG.
Helps to have console comms etc sorted.
But it can work.
I'm working on an open-source tool for that, that is either a minimal test or can provide detailed information when run through the debugger.
we have one main application, which executes up to 5 different exes. These exes run independently and communicate with each other via UDP. Changing this architecture is not planned at the moment.
We want to migrate this whole thing from VS6 to VS2008.
I'm thinking about adding unit tests to make sure that after migration everything still works. Right now, there is not a single unit test.
So now I have a couple of questions:
Which unit test framework works fine with VS6 and VS2008? CppUnit seems to work with both compilers, at least I got it running.
How do I implement the unit tests in the above mentioned architecture? I see a problem that I have executables and no libraries, so it seems to be a bit difficult for me at that point
Is creating the unit tests for both platforms worth the effort or do you have a simpler suggestion?
Suggestions, best practices, new ideas welcome :)
Thanks
If they communicate via UDP and you want to make sure the integration between pieces works now and later, you could write automated tests against the UDP interfaces in MSTest or NUnit in VS2008.
Write the tests in VS2008 and send input/verify output of the interfaces to the the .exe applications. Then, when you switch to .NET, just point your tests at the new endpoint for the .NET .exe and run them to verify all input and output are the same.
I think you would get more bang for your time to write the unit tests exclusively for the new app, but write tests against the external UDP interfaces to verify the migration was successful.
If you are willing to consider each of the executables as as 'module', then there is a way to introduce tests without being too invasive.
This would create modules that will be a bit large for many people to be comfortable with (typical sizes when people talk about modules is a class or a source file), but your communication mechanism provides a nice way to hook in the tests.
With the architecture that you have and the fact that you don't have an existing test harness for the code, I would write a test harness that can communicate with each of the executables in isolation, using the UDP interfaces.
When executing the tests, only one executable is running (besides the test code). The test code simulates the other executables by listening on their respective UDP ports and providing responses as the testcase requires. The testcase also stimulates the executable under test by sending UDP requests to it. If there is also a GUI/TUI involved, it might be needed that parts of the tests are executed manually.
I have a situation where I need to write some unit tests for some device drivers for embedded hardware. The code is quite old and big and unfortunately doesn't have many tests. Right now, the only kind of testing that's possible is to completely compile the OS, load it onto the device, use it in real life scenarios and say that 'it works'. There's no way to test individual components.
I came across an nice thread here which discusses unit testing for embedded devices from which I got a lot of information. I'd like to be a little more specific and ask if anyone has any 'best practices' for testing device drivers in such a scenario. I don't expect to be able to simulate any of the devices which the board in question is talking to and so will probably have to test them on actual hardware itself.
By doing this, I hope to be able to get unit test coverage data for the drivers and coax the developers to write tests to increase the coverage of their drivers.
One thing that occurs to me is to write embedded applications that run on the OS and exercise the driver code and then communicate the results back to the test harness. The device has a couple of interfaces which I can use to probably drive the application from my test PC so that I can exercise the code.
Any other suggestions or insights would be very much appreciated.
Update: While it may not be exact terminology, when I say unit testing, I meant being able to test/exercise code without having to compile the entire OS+drivers and load it onto the device. If I had to do that, I'd call it integration/system testing.
The problem is that the pieces of hardware we have are limited and they're often used by the developers while fixing bugs etc. To keep one dedicated and connected to the machine where the CI server and automated testing is done might be a no no at this stage. That's why I'm looking for ways to test the driver without having to actually build the whole thing and upload it onto the device.
Summary
Based on the excellent answers below, I think a reasonable way to approach the problem would be to expose driver functionality using IOCTLs and then write tests in the application space of the embedded device to actually exercise the driver code.
It would also make sense to have a small program residing in the application space on the device which exposes an API that can exercise the driver via serial or USB so that the meat of the unit test can be written on a PC which will communicate to the hardware and run the test.
If the project was just being started, I think we'd have more control over the way in which the components are isolated so that testing can be done mostly at the PC level. Given the fact that the coding is already done and we're trying to retrofit the test harness and cases onto the system, I think the above approach is more practical.
Thanks everyone for your answers.
In the old days, that was how we tested and debugged device drivers. The very best way to debug such a system was for engineers to use the embedded system as a development system and—once adequate system maturity was reached— take away the original cross-development system!
For your situation, several approaches come to mind:
Add ioctl handlers: each code exercises a particular unit test
With conditional compilation, add a main() to the driver which conducts functional unit tests in the driver and outputs results to stdout.
For initial ease in debugging, maybe this could be made multi-platform operable so you don't have to debug on the target hardware.
Perhaps conditional code can also emulate a loopback-style device.
The code that really is dependent on the hardware (the lowest level of the driver stack in a layered architecture) can't really be tested anywhere except on the hardware, or a high-quality simulation of the hardware.
If your driver has some component of higher-level functionality that doesn't rely directly to the hardware (e.g., a protocol handler for sending messages to hardware in a particular format) and if that part is nicely self-contained in the code, then you could unit-test that separately in a PC-based unit-test framework.
Going back to the lowest level—if it's dependent on the hardware, then the test jig needs to include the hardware. You can make a test jig that includes the hardware, the driver, and some test software. The main thing, I think, is to get the normal product's application code out of the test, and put in some test code instead. The test code can systematically test all the driver's features and corner-cases (which the application code may not), and can also really hammer the driver intensively in a short amount of time (which the application probably doesn't). Thus it's more efficient use of your limited hardware than just running the application, and gives you better results.
If you can get a PC into the loop, then the PC might help with the testing. E.g. if you're writing a serial port driver for an embedded device, then you could:
Write test code for the embedded device that sends various known data streams.
Connect it to a PC's serial port, running test code that verifies the transmitted data streams.
Same in the other direction—PC sends data; embedded device receives it and verifies it, and notifies the PC of any errors.
The tests can stream data at full speed, and play with a range of different byte timings (I once found a microcontroller UART silicon bug that only appeared if bytes were sent with a ~5 ms delay between bytes).
You could do a similar thing with an Ethernet driver, a Wi-Fi driver.
If you're testing a storage device driver, such as for an EEPROM or Flash chip, then the PC couldn't get involved in the same way. In that case, your test harness could test all sorts of write conditions (single-byte, block...), and verify data integrity using all sorts of read conditions.
I had a similar problem two or three years ago. I've ported a device driver from VxWorks to Integrity. We had changed only operating system dependent parts of the driver but it was a safety critical project, so all the unit tests, integration tests are redone. We have used a automated testing tool called LDRA testbed for our unit tests. 99% of our unit tests are done on Windows machines with Microsoft Compilers. Now I'll explain how to do this
Well first of all, when you are doing unit testing you are testing a software. When you include the real device in your tests, you are also testing the device. Sometimes there may be issues with hardware or documentation of the hardware. When you are designing the software, if you have described the behaviour of the each function clearly, it is very easy to make unit testing, for example, Think about the function;
readMessageTime(int messageNo, int* time);
//This function calculates the message location, if the location is valid,
//it reads the time information
address=calculateMessageAddr(messageNo);
if(address!=NULL) {
read(address+TIME_OFFSET,time);
return success;
}
else {
return failure;
}
Well, here you are just testing if readMessageTime is doing what it is supposed to do. You do not have to test if calculateMessageAddr is calculating the right result or, read reads the right address. That is the responsibility of some other unit tests.. So what you have to do is write stubs for calculateMessageAddr and read(OS function) and check if it calls the functions with correct parameters. This is the case If you are not accessing the memory directly from your driver. You can test any kind of driver code without any OS or device with this mentality.
If you have mapped the device memory directly into your memory space and device driver reads and writes to device memory as it is its own memory, it gets a little bit complicated. Using automated testing tools, now you have to watch values of pointers and define pass/fall criterieas according to the values of these pointers. if you are reading a value from memory, you have to define the expected value. This may be hard in some cases.
There is also one more issue, developers always confuse in unit testing of drivers such like:
readMessageTime(int messageNo, int* time);
//This function calculates the message location, if the location is valid,
//it does some jobs to make the device ready to read then
//it reads the time information
address=calculateMessageAddr(messageNo);
if(address!=NULL) {
do_smoething(); // Get the device ready to read!
do_something_else() // do some other stuff so you can read the result in 3us.
status=NOT_READY;
while(status==NOT_READY) // mustn't be longer than 3us.
status=read(address+TIME_OFFSET,time);
return success;
} else
{
return failure;
}
Here do_something and do_something_else does some jobs on device to make it ready to read. Developers always ask themselves "What if the device do not get ready forever and my code have a deadlock here" and they tend to test this kind of stuff on device.
Well, you have to trust the device manufacturer and the technical author. If they are saying that device will be ready in 1-2us, you do not need to worry about this. If your code fails here, you have to report it to device manufacturer, it is not your job to find a workaround to overhelm this problem. Did you see my point?
I hope this helps….
I had this exact task just two months ago.
Let me guess:
You probably have "snippets" of code that speak low level details to the device. You know that these snippets work, but you can't get coverage on them because they have a dependency to the device drivers.
Likewise, it does not make sense to test every single line of it individually. They are never run in isolation, and your unit test would end up looking like a mirror reflection of the production code.
For example, if you wish to start the device, you need to create a connection, pass it a specific low level reset command, then an initialize parameter struct etc etc.
And if you need to add a piece of configuration, this may require you to take it off line, add the configuration and then take it online.
Stuff like that.
You do NOT want to test low level stuff. Your unit tests would then only reflect how you assume that the device work without confirming anything.
The key here is to create three items: a controller, an abstraction and an adapter implementation of that abstraction. In Cpp, Java or C# you would create either a base class or an interface to represent this abstraction. I will assume that you created an interface.
You break up the snippets into atomic operations. For example you create a method called "start" and "add(parameter)" in the interface. You put your snippets in the device adapter.
The controller acts on the adapter through the interface.
Identify pieces of logic within the snippets that you have placed in the adapter. Then you need to decide wether this logic is low level (protocol handling details etc) or wether this is logic that should belong in the controller.
You can then test in two stages:
* Have a simple test panel application that acts on the concrete adapter. This is used to confirm that the adapter actually works. That it starts when you press "start". That, for example, if you press "go offline", "transmit(192)" and "go online" in sequence, that the device responds as expected. This is your integration test.
You do not unit test the details in the adapter. You test it manually because the only success criteria is how the device responds.
However, the controller is completely unit tested. It only has a dependency to the abstraction, which is mocked out in your test code. Thus, your code has no dependency to your device driver because the concrete adapter is not involved.
Then you write unit tests to confirm that, for instance, the method "Add(1)" actually invokes "Go offline" then "Transmit(1)" and then "Go online" on the mocked out abstraction.
The challenge here is to draw the distinction between the adapter and the controller. What goes where? What worked for me was to create the aforementioned test panel first and then manipulate the device through it.
The adapater should hide the details you will only have to change if the device changes.
If the control panel is cumbersome to operate with lots of sequences that needs to be repeated again and again, or that very device specific knowledge is required to operate the panel, then you have too high granularity and should bulk some of them together. The test panel should make sense.
If end user requirements changing have impact on the adapter code, then you probably have too low granularity and should split the operations up, so that the requirements change can be accommodated with test driven development in the controller class.
I'd recommend for application-based testing. Even if the scaffolding can be hard and costly to build, there is a lot to gain here:
crash only once process as opposed to one system
ability to use standard tool set (debugger, memory checker ...)
overcome the hardware availability limitation
faster feedback: no installation in device, just compile and test
...
As far as naming is concerned, this can be called component testing.
The application can either initialize the device driver the same way the target OS does, or use directly the interns of the driver. The former is more expensive but leads to more coverage. Then the linker will tell which functions are missing, stub them, possibly using exploding stubs.
Vocabulary
I don't expect to be able to simulate any of the devices which the board in question is talking to and so will probably have to test them on actual hardware itself.
Then, you are stepping out of unit testing. Maybe you could use one of these expressions instead?
Automated testing : testing happen without user input (the contrary of Manual Testing).
Integration testing : testing several components together (the contrary of Unit testing).
On a bigger scale, if you test a whole system and not just a few components together, it is called System testing.
ADDED after comments and updates in the question :
Component testing : like integration testing or System testing, but on an even smaller scale.
Note : All three Component-Integration-System Testings share the same set of problems, on different scales. On the contrary, Unit Testing does not (see lower).
Advantages of "real" Unit Testing
With Integration- (or System- or Component-) Testing, it is certainly interesting to get some feedback, like test coverage. It is certainly useful to do.
But it is very hard (read "very costly") to make progress beyond some point, so
I suggest you use complementary approaches, like adding some real Unit Tests. Why? :
It is very hard to simulate the edge or error conditions. (Examples : the computer clock crosses a day or year during a transaction ; the network cable is unplugged ; power went down then up on some component, or the whole system ; the disk is full). Using Unit Testing, because you simulate these conditions rather than try to reproduce them, it is much easier. Unit Testing is your only chance to get a really good code coverage.
Integration testing takes time (because of access to external resources). You could execute thousands of unit test during the execution of one Integration Test. So testing many combinations are only possible with Unit Tests...
Requiring access to specific resources (hardware, Licence etc...), Integration Testing is often limited in time or scale. If the resources are shared by other projects, each project might use them only during a few hours per day. Even with exclusive access, maybe only one machine can use it, so you can't run tests in parallel. Or, your company may buy a resource (Licence or Hardware) for production, but not have it (or early enough) for development...