Now I'm experiencing a problem about in testing,
We have developed a platform with C++, and this platform include 2 layers, one is engine layer, the other one is widget layer, and I'm a dev in widget layer,
The widget layer used by client app (those app is our product), now the problem is this:
1, We want to add some unit test for widget layer to enhance our whole platform quality and reliability, but in fact, this platform has been release for several versions, so now, what's kind of unit test could be most suitable for our project? To add some unit test in kinds of validation/bad input/path cover/stress or to add some unit test that simulate the client scenario? I think the former one will enhance code coverage very well, but i also think about that if we can simulate how our client using those api in their application, those unit test maybe more make sense for our platform, right?
2, just as i mentioned before, i'm in widget layer which is above and depended on engine layer, but we has no authority for engine layer code, and almost all widget API could call in engine layer, so it's difficult for us to implement good path cover unit test for those api, because we have no idea(and no document)about engine layer code, but i still want to ask, is there any ways, tools or frameworks could make someone write high quality unit test to cover path for those black box API?
Since you work in the widget API, and do not have access to the engine code, write your own engine instead.
Use this engine to test the widget code. The engine can be very thin, almost not functional at all, doing just the minimum to fool the widget layer into believing it works.
The test engine can even be a shim on top of the real engine. How you implement this varies a lot depending on the environment you work in, but in Windows it could be that a DLL forwards all its calls to the "real" DLL, but logs calls to it first.
Another technique: since you do have access to the Widget layer, and the engine API is very big, you can reprogram the Widget layer to use the "test layer" in only some places, but the real layer in remaining places. This can be combined with the shim technique.
Suitable (or available) tools for generating stubs or stub engines varies a lot depending on which language and environment you are working in.
Also see this DirectDraw hack for inspiration, it started life as a thin, logging shim upon the original DirectDraw implementation.
You do not mention what widget framework you are using but Qt has a test framework that allows the test code to trigger the same Qt signals that normally get generated by user input (for example mouse clicks or key pressed). See this for more detail.
If you are using a different framework, there might be similar functionality that you could use.
Related
Can a UI application be unit tested? Lets consider a UI application which has a simple login form. What would form part of unit test for this UI application. Should the backend response validation form part of unit test for login form?
In my opinion, It should include;
appropriate rendering of UI components in the form.
enabling/Disabling of components based on user actions (password can not be empty message when password is not entered and form is submitted).
Are there any guidelines/rule of thumbs which should be used while devising unit test cases for UI application? Should an application behavior considered in unit testing?
In general, testing UI components is not an easy task - you need a good architecture on the tested code, quite some tools (which may come with some license costs...), and at least one virtual machine.
The most important part is to have some 'Model-View-younameit' architecture in place, because only this allows you to isolate the UI stuff from the rest of the application. A test for the login form can then be as follows:
A tool (e.g. White or Ranorex) mimics a user typing in some user/password combination.
The test verifies (by the help of e.g. a mocking framework or Typemock), that the correct methods are called (or not called under certain circumstances).
What you have to avoid for UI testing in any case is calling some business logic underneath. If you can't isolate the UI this way (which is often the case especially for legacy projects...), then automating UI testing may not be beneficial at all, because it often comes with way too much effort in such a situation. Sometimes you better go with manual testing...
HTH!
Thomas
UI testing is not unit testing in the strict sense, however there are tools/frameworks to help that. I assume you mean a web UI - for this you can try e.g. Selenium or HttpUnit.
Note that "appropriate rendering of UI components" is a very loose term - it can mean things which can not be verified automatically, only by human eye.
Testing functionality via the UI is always more cumbersome than testing the code directly via classic unit tests. Hence it is always recommended to separate the business logic from the UI as much as possible - then you can thoroughly test all the nooks and crannies of the logic itself via unit tests, and need only integration tests on the UI to verify that the app as a whole works as expected.
In your example, ideally the input validation and the authentication would be separated from the GUI, so you could unit test these (preferably by mocking out the DB). Thus you could verify that the login works as expected for all sorts of normal/abnormal inputs (empty userid/password, strange characters, very long input, logging in the same user twice, ...). Then in the UI tests you need only test a few common test cases to verify that the app indeed is functional.
If your UI is a native graphical application, you may want to consider AutoIt (Windows only), which allows you to programmatically click any place on the screen and simulate text typed into an widget or just keyboard presses. It also has the ability to read the color of pixels at specified positions. Or you can use it to take screen shots and do an image comparison.
If it's web based, you may also want to consider using xpath to determine (non-graphically) whether elements exist in your generated HTML. It doesn't help you with how pages are rendered though. AutoIt may help with that as well, but you won't be able to perform the same type of test on a different platform, assuming you would like all users from all platforms to be able to reasonably access your page.
If your UI is console based, consider using Expect. I prefer the Perl based expect, which is based on the TCL Expect.
I know I'm not answering directly about rules or guidelines, but I think knowing what tools are available out there will open up the possibilities that you think may not exist or had not considered. Just as an example, you can use AutoIt to determine whether a component is disabled or enabled because usually the widget is of a different color from the enabled version when it is disabled. So by using AutoIt to obtain the color of a pixel or region of the screen, you can programmatically test whether that widget is enabled or not.
I've heard of unit testing, and written some tests myself just as tests but never used any testing frameworks. Now I'm writing a wxPython GUI for some in-house data analysis/visualisation libraries. I've read some of the obvious Google results, like http://wiki.wxpython.org/Unit%20Testing%20with%20wxPython and its link http://pywinauto.openqa.org/ but am still uncertain where to start.
Does anyone have experience or good references for someone who sort of knows the theory but has never used any of the frameworks and has no idea how it works with GUIs?
I am on a Windows machine developing a theoretically cross-platform application that uses NumPy, Matplotlib, Newville's MPlot package, and wxPython 2.8.11. Python 2.6 with plans for 3.1. I work for a bunch of scientists, so there is no in-house unit-testing policy.
If you want to unit-test your application, you haven't to focus on GUI testing techniques. It is much better to write the application using MVC, MVP, or other meta-pattern like these. So you get business logic and presentation layer separated.
It is much more important to cover the business layer with tests since this is your code. Presentation layer is tested already by wxWidgets developers. To test the business layer it will be enough just basic tools like standard unittest module and maybe nose.
To make sure the whole application behave correctly, you should add few acceptance tests that will test functionality from end to end. These will deal with GUI, but there will be few such tests comparing to number of unit-tests.
If you will limit yourself with acceptance tests only, you'll get low coverage, fragile and very slow test code base.
To unit test your application without requiring lots of mock objects/stubs, your GUI's event handlers should basically delegate to other method calls, passing in values from the Event object as parameters to the delegated method.
Otherwise you'll be unable to test your application without having to mock wx's objects.
Take a look at the PyPubSub project for a great module to help with MVC.
In one early project of mine I really test wxPython application using GUI layer. So tests really spin live wxApp object, pops up real windows and then starts messing with a real MainLoop(). Very soon I realize it was a wrong way to do testing. My tests was run very slow and unreliable. Much better way is to separate GUI stuff aside and test only the "model" level of your application. Note that you can actually create model for presentation level logic (model that represent some visual part of your application) and test it. But this model should not involve any "real" gui objects (windows, dialogs, widgets).
We are developing applications for use within AutoCAD.
Basically we create a Class Library Project, and load the .dll in AutoCAD with a command (NETLOAD).
As so, we can use commands, "palettes", user controls, forms etc...
AutoDesk provides an API through some dll's, running in their program directory.
When referencing these dll's you can only call the dll's at runtime while loading your app in AutoCAD (This is a licensing security from AutoDesk).
For us, while developing, this is not a problem, we need to visually test within the context of AutoCAD, so we just set the Debug Properties so that they start acad.exe and load our dll with a script in the acad.exe parameters.
The problem is, when trying to unit test our code, NUnit or mstest are not running from within the AutoCAD context and they also cannot start it.
There exist a tool called Gallio, which has provided an interface with AutoCAD, so that it can run Unit test through IPC with Named Pipes.
However, this solution is, for me, too much of a hassle. I want to be able to quickly write tests without having to leave my beloved IDE.
So, what, from a "good design view" would be a good approach to this problem? I'm thinking I would basically need a testable codebase which is not referencing the AutoCAD dll's and a non-testable that does references the untestable AutoCAD dll's.
I'm sure there are ways to get this to work: ( IOC, DI, Adapter Pattern,. .) I just don't these principles in depth and thus I don't know which route will best suit my purposes and goals.
The first step is to triage your code for parts which need AutoCAD and parts which are really independent. Create unit tests for the independent parts as you usually would.
For the other parts, you need mockups which behave like AutoCAD. Make them as simple as possible (for example, just return the correct answers in the methods without doing any calculations). Now, you need several sets of classes:
A set of interfaces which your code uses to achieve something (for example, load a drawing).
A set of implementations for said set of interfaces which call the AutoCAD dlls.
A set of classes which try the implementations within the context of AutoCAD. Just create a small UI with a couple of buttons where you can run this code. It is used to reassure yourself that your mockups do the right thing. Log method parameters and results to some file so you can try how AutoCAD responds. If a mockup breaks, you can use this code to verify what AutoCAD is doing and you can use it as a reference when developing the mockups.
When you know how AutoCAD responds, create the mockups. In your tests, create them with the desired results (and errors, so you can test error handling, too). So when you have boolean loadDrawing(File filename), create a mockup which returns true for the filename exists.dxf and false for anything else.
Use a factory or DI to tell your application code which implementation to use. I tend to have a big global config class with a lot of public fields where I simply store the objects to use. I can set this up in the beginning, it's fast, it's easy to understand. If you need to create objects at runtime, then put factories in the config class which generate the objects for you, so you can swap them out.
I wrote ... and later broke ... a Test runner for AutoCAD. It is at https://github.com/CADbloke/CADtest. If you're interested in it nudge me along and I'll fix it faster. I am waiting for NUnit v3 release before I tackle it.
If you reset to the 3rd commit in that repo (I think) and fiddle with it from there it should run.
How do you unit test a large MFC UI application?
We have a few large MFC applications that have been in development for many years, we use some standard automated QA tools to run basic scripts to check fundamentals, file open etc. These are run by the QA group post the daily build.
But we would like to introduce procedures such that individual developers can build and run tests against dialogs, menus, and other visual elements of the application before submitting code to the daily build.
I have heard of such techniques as hidden test buttons on dialogs that only appear in debug builds, are there any standard toolkits for this.
Environment is C++/C/FORTRAN, MSVC 2005, Intel FORTRAN 9.1, Windows XP/Vista x86 & x64.
It depends on how the App is structured. If logic and GUI code is separated (MVC) then testing the logic is easy. Take a look at Michael Feathers "Humble Dialog Box" (PDF).
EDIT: If you think about it: You should very carefully refactor if the App is not structured that way. There is no other technique for testing the logic. Scripts which simulate clicks are just scratching the surface.
It is actually pretty easy:
Assume your control/window/whatever changes the contents of a listbox when the user clicks a button and you want to make sure the listbox contains the right stuff after the click.
Refactor so that there is a separate list with the items for the listbox to show. The items are stored in the list and are not extracted from whereever your data comes from. The code that makes the listbox list things knows only about the new list.
Then you create a new controller object which will contain the logic code. The method that handles the button click only calls mycontroller->ButtonWasClicked(). It does not know about the listbox or anythings else.
MyController::ButtonWasClicked() does whats need to be done for the intended logic, prepares the item list and tells the control to update. For that to work you need to decouple the controller and the control by creating a interface (pure virtual class) for the control. The controller knows only an object of that type, not the control.
Thats it. The controller contains the logic code and knows the control only via the interface. Now you can write regular unit test for MyController::ButtonWasClicked() by mocking the control. If you have no idea what I am talking about, read Michaels article. Twice. And again after that.
(Note to self: must learn not to blather that much)
Since you mentioned MFC, I assumed you have an application that would be hard to get under an automated test harness. You'll observe best benefits of unit testing frameworks when you build tests as you write the code.. But trying to add a new feature in a test-driven manner to an application which is not designed to be testable.. can be hard work and well frustrating.
Now what I am going to propose is definitely hard work.. but with some discipline and perseverance you'll see the benefit soon enough.
First you'll need some management backing for new fixes to take a bit longer. Make sure everyone understands why.
Next buy a copy of the WELC book. Read it cover to cover if you have the time OR if you're hard pressed, scan the index to find the symptom your app is exhibiting. This book contains a lot of good advice and is just what you need when trying to get existing code testable.
Then for every new fix/change, spend some time and understand the area you're going to work on. Write some tests in a xUnit variant of your choice (freely available) to exercise current behavior.
Make sure all tests pass. Write a new test which exercises needed behavior or the bug.
Write code to make this last test pass.
Refactor mercilessly within the area under tests to improve design.
Repeat for every new change that you have to make to the system from here on. No exceptions to this rule.
Now the promised land: Soon ever growing islands of well tested code will begin to surface. More and more code would fall under the automated test suite and changes will become progressively easier to make. And that is because slowly and surely the underlying design becomes more testable.
The easy way out was my previous answer. This is the difficult but right way out.
I realize this is a dated question, but for those of us who still work with MFC, the Microsoft C++ Unit Testing Framework in VS2012 works well.
The General Procedure:
Compile your MFC Project as a static library
Add a new Native Unit Test Project to your solution.
In the Test Project, add your MFC Project as a Reference.
In the Test Project's Configuration Properties, add the Include directories for your header files.
In the Linker, input options add your MFC.lib;nafxcwd.lib;libcmtd.lib;
Under 'Ignore Specific Default Libraries' add nafxcwd.lib;libcmtd.lib;
Under General add the location of your MFC exported lib file.
The https://stackoverflow.com/questions/1146338/error-lnk2005-new-and-delete-already-defined-in-libcmtd-libnew-obj has a good description of why you need the nafxcwd.lib and libcmtd.lib.
The other important thing to check for in legacy projects. In General Configuration Properties, make sure both projects are using the same 'Character Set'. If your MFC is using a Multi-Byte Character Set you'll need the MS Test to do so as well.
Though not perfect, the best I have found for this is AutoIt http://www.autoitscript.com/autoit3
"AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys). AutoIt is also very small, self-contained and will run on all versions of Windows out-of-the-box with no annoying "runtimes" required!"
This works well when you have access to the source code of the application being driven, because you can use the resource ID number of the controls you want to drive. In this way you do not have to worry about simulated mouse clicks on particular pixels. Unfortunately, in a legacy application, you may well find that the resource ID are not unique, which may cause problems. However. it is very straightforward to change the IDs to be unique and rebuild.
The other issue is that you will encounter timing problems. I do not have a tried and true solution for these. Trial and error is what I have used, but this is clearly not scalable. The problem is that the AutoIT script must wait for the test application to respond to a command before the script issues the next command or check for the correct response. Sometimes it is not easy to find a convenient event to wait and watch for.
My feeling is that, in developing a new application, I would insist on a consistent way to signal "READY". This would be helpful to the human users as well as test scripts! This may be a challenge for a legacy application, but perhaps you can introduce it in problematical points and slowly spread it to the entire application as maintenance continues.
Although it cannot handle the UI side, I unit test MFC code using the Boost Test library. There is a Code Project article on getting started:
Designing Robust Objects with Boost
Well we have one of these humongous MFC Apps at the workplace. Its a gigantic pain to maintain or extend... its a huge ball of mud now but it rakes in the moolah.Anyways
We use Rational Robot for doing smoke tests and the like.
Another approach that has had some success is to create a small product-specific language and script tests that use VBScript and some Control handle spying magic. Turn common actions into commands.. e.g. OpenDatabase would be a command that in turn will inject the required script blocks to click on Main Menu > File > "Open...". You then create excel sheets which are a series of such commands. These commands can take parameters too. Something like a FIT Test.. but more work. Once you have most of the common commands identified and scripts ready. It's pick and assemble scripts (tagged by CommandIDs) to write new tests. A test-runner parses these Excel sheets, combines all the little script blocks into a test script and runs it.
OpenDatabase "C:\tests\MyDB"
OpenDialog "Add Model"
AddModel "M0001", "MyModel", 2.5, 100
PressOK
SaveDatabase
HTH
Actually we have been using Rational Team Test, then Robot, but in recent discussions with Rational we discovered they have no plans to support Native x64 applications focusing more on .NET, so we decided to switch Automated QA tools. This is great but licensing costs don't allow us to enable it for all developers.
All our applications support a COM API for scripting, which we regression test via VB, but this tests the API no the application as such.
Ideally I would be interested on how people integrate cppunit and similar unit testing frameworks into the application at a developer level.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am working on the issue of testing my GUI and I am not entirely sure of the best approach here. My GUI is built using a traditional MVC framework so I am easily able to test the logic parts of the GUI without bringing up the GUI itself. However, when it comes to testing the functionality of the GUI, I am not really sure if I should worry about individually testing GUI components or if I should mainly just focus on functional testing the system. It is a pretty complex system in which testing the GUI frequently involves sending a message to the server and then observing the response on the GUI. My initial thoughts are that functional testing is the way to go here since I need a whole system running to really test the UI. Comments on this issue would be appreciated.
Other GUI-testing tools I can offer are:
Thoughtworks White,
PyWinAuto,
AutoIt,
AutoHotKey.
One thing to keep in mind when trying to automate GUIs is that the only way you can do that is to build the GUI with automation in mind. Crush devs that think their GUIs should not support testability early on in the project and happily expose all the hooks that can help in automation on demand as your testing needs require that.
You have (at least) 2 issues - the complexity of the environment (the server) and the complexity of the GUI.
There are many tools for automating GUI testing. All of them are more or less fragile and require pretty much constant maintenance in the face of changing layout. There is benefit to be gained from using them, but it's a long term benefit.
The environment, on the other hand, is an area that can be tamed. If your application is architected using the Dependency Injection/Inversion technique (where you 'inject' the server component into the application), then you can use a 'mock' of the relevant server interfaces to enable you to script test cases.
Combining these two techniques will allow you to automate GUI testing.
Depending on where in the spectrum of MVC (that's an overused term) you sit, testing the view could be a mechanical process of ensuring that the correct model methods are called in response to the correct inputs to the view to testing some client side validation to who knows.
A lot of the patterns that have been evolved out of MVC (I'm thinking passive view, supervising controller) are striving to make the view require very little testing because it's really just wiring user inputs to the presenter or model (depending on the exact variant of the pattern you're using).
"testing the GUI frequently involves sending a message to the server and then observing the response on the GUI" This statement worries me.
I'm immediately thinking that the GUI should be tested using a mock or stub of the server to test that the correct interactions are occurring and the GUI responds appropriately.
If you need automated functional tests of the server, I don't see the need to have the GUI involved in those.
Mercury QuickTest Pro, Borland SilkTest, and Ranorex Recorder are some GUI testing tools.
If your application is web-based you can write tests using tools like WatiN or Selenium.
If your application is Windows .NET based, you could try White.
My advice: forget the traditional GUI testing. It's too expensive. Coding the tests takes a lot of time, the tools aren't really stable so you will get unreliable test results. The coupling between the code and the test is very strong and you'll spend a lot of time with the maintenance.
The new trend is to ignore the GUI tests. See the ModelViewPresenter pattern from Fowler as a guideline link text
The clearest way I can say this is:
Don't waste your time writing automated GUI tests.
Especially when your working with an MVC app - in your case, when you send a message to the server, you can make sure the right message number comes back and be done. You can add some additional cases - or another test completely to make sure that the GUI is converting the message id's into the right strings, but you just need to run that test once.
We do incorporate GUI testing in our project, and it has its side effects. The developers however have one critical design principle: Keep the GUI layer as thin as possible!
That means no logic in the GUI classes. Separate this in presentation models responsible for input validation etc.
For testing on a Unix machine we use the Xvfb server as the DISPLAY when running the tests.
Try the hallway usability test. It's cheap and useful: go to the nearest hallway, grab the first person that passes, make them sit at your computer and use your software. Watch over their shoulder, you will see what they try to do, what frustrates them, and so on. Do this a few times and notice the patterns.
What you're looking for is "acceptance testing." How you do it depends on the frameworks you're using, what type of application you are creating and in what language. If you google your particular technology and the above phrase, you should find some tools you can use.
I've found WinTask to be a very good way to do GUI testing. Provided you don't constantly change the way the OS refers to each element of the UI, WinTask addresses the UI elements by name, so even if the layout changes, the UI elements can still be pressed / tweaked / selected.
Don't miss the 'U' in 'GUI'
I mean: if what you're trying to test is all works right and works as it was planned to work, then you may follow Seb Rose's answer.
But please, don't forget a USER interface has to be made thinking about USERS, and not ANY user but the TARGET USER the application was made for. So, after you are sure all works like it have to work, put every single view/screen/form in a test with a team made of users representing every group of different users that may use your application: advanced users, administrators, MS Office users, low computer profile users, high computer profile users... and then, get the critiques of every user, make a mix, re-touch your GUI if it's neccesary and back again to GUI user's test.
For SIMPLE Web based GUI testing try iMacros ( a simple Firefox plug-in , has a cool feature to send the entire test to another person )
Note that SIMPLE was spelled with Initials ...