I have recently completed a detailed investigation regarding GP functional test automation possibilities with QTP, TestComplete, and other GUI recognition/interaction tools.
In short, none of the tools acted well. Mentioned above did best but still featured a lot of hard-coding in recorded sample scripts. QTP did significantly better though.
Some of the GUI were recognized under .NET (swf... in QTP) and handled well. Some other were recognized as swfObject only but accessing native methods and properties allowed performing required interaction.
Finally, object internally named "Microsoft.Dexterity.Shell.DexDialogHost" renders objects that seem don't have Windows Handle and thus unrecognizable by QTP. I haven't found any detailed documentation on those objects family (like class reference to find out names of methods).
So I was wondering if someone could share experience automating GP and what tools / approach were used.
Thanks.
If QTP doesn't recognise some controls you can extend the set of controls it supports by using .NET Extensibility. This allows you to add new functionality which is relevant for these controls but is not supported out of the box by QTP. Note that some .NET programming is required.
The documentation for .NET Extensibility is available at Start > Programs > QuickTest Professional > Extensibility > Documentation > .NET Add-in Windows Forms Extensibility Help
Fact of the day: The SWF in .NET objects' names in QTP stands for System Windows Forms.
With regards to the objects that were partially successful, the ones you noted were recognized as SwfObject - if these logically map to a standard class, you can configure QTP to treat it as a standard class. For example, if you have an SwfObject that behaves like a button, you can configure QTP to record and replay as if it is a standard button.
[removed suggested keystroke+clipbaord work-around as it is inappropriate for this situation given further information that there are multiple Dexterity dialogs, each with many objects]
It took a while for me to conduct a research, and then practically prove the concept.
I started series of posts about Great Plains automation success story where I put all the details and steps of my investigation along with sample code.
http://automation-beyond.com/2009/08/24/great-plains-automation/
http://automation-beyond.com/2009/08/26/dynamics-great-plains-gui/
http://automation-beyond.com/2009/09/01/gp-automation-utilizing-com/
(to be continued)
Here's the summary.
Microsoft Dynamics Great Plains has its own completely independent GUI/Event system (Dexterity) that makes it cross-platform product. The same thing makes it almost completely "black box" not accessible from outside.
Platform-specific engine supports COM Automation and various integration models for Windows platform. All these are gathered under Continuum Integration Library name.
What is especially useful for Test Automation needs, Continuum provides high-level methods to simulate user inputs for the GUI thus acting the same way as Test Automation Tool does.
Wherever Continuum doesn't cover Test Automation needs, direct calls of sanScript (Dexterity's internal scripting language) could be executed in real-time.
With all the above, I created custom "Dexterity GUI" component for Quick Test Professional enabling functional test automation of Microsoft Dynamics Great Plains application. Of course, without any use of keyboard/mouse hard-coded workarounds.
Thank you.
You can download the .NET Add-In from hp.
link text
Related
Ranorex is a user interface test tool for Windows. If UI elements support Microsoft Active Accessiblity (MSAA) then it can retrieve rich information about the elements, which is very useful for writing test scripts.
We have a lot of legacy code in VB6, which does not support MSAA.
I am trying to add this support to some of our VB6 UserControls as follows.
I have created an ActiveX component in C++ using ATL. This contains a helper class which implements the interface IAccessible. It also contains an initialization function which can be called from VB6.
The VB6 UserControl creates the helper object and passes its Window Handle and name to the initialization function. The helper object subclasses the window and handles the WM_GETOBJECT message, to provide the IAccessible interface for the window.
I have debugged through the code and as far as I can tell, it is working. The Ranorex Spy tool gets the IAccessible interface and calls methods on the interface.
However, Ranorex does not actually show any of the information which it appears to have fetched via this interface.
I know this is a very specific problem, but does anybody have experience of implementing IAccessible for use in Ranorex, or in other similar test tools?
I would be happy to provide my code, but I think it is too much for a normal posting.
This doesn't ever seem to have been widely used. I have found a few breadcrumbs I can share, so perhaps these will be of some value?
First:
Microsoft Active Accessibility 2.0 ships as a standard system component in Microsoft Windows XP...
This is from the MSAA20RDK\redist.txt and it goes on to talk about how it can be installed into Win2K and earlier. As far as I can determine the pieces and parts are included in XP and later so you probably don't need the RDK download, which has basically no info at all and mostly just the DLLs.
Here's a link though:
Microsoft Active Accessibility 2.0 Redist
Something perhaps more useful, note that this is a "live" link to download the Word DOC:
Microsoft Active Accessibility 2.0
I haven't found a "page" link at MS Downloads that discusses it. This is a larger document giving many details, almost entirely for C++ programmers. There is however a little bit on VB6.
That said it tells you little except:
Visual Basic Sample Programs
The book Advanced Microsoft Visual Basic 6.0, 2nd Edition (Mandelbrot Set
International, Ltd.) includes a chapter that discusses Active
Accessibility. The chapter provides sample code that demonstrates how to
use key Active Accessibility APIs. (Microsoft Press, ISBN 1-57231-893-7)
There is a little more there but not much. The book cited could be pretty hard to find anymore, I only saw one in Italian on Amazon but dig around and a 3rd party might sell there or on eBay, etc.
At this link: Microsoft Active Accessibility the info from that Word DOC appears to be duplicated, though perhaps it is more current.
I am aware that this may potentially be an unsuitably broad question, but I am unsure where else to ask about it. Hopefully it can serve as a basis for information for me and other seeking to construct richer web applications.
I am very interested interested in the availability of frameworks that ease the construction of rich web "desktop" apps. The best single example I can think of is AirDroid (demo: http://web.airdroid.com/). I have tried to deduce from studying the HTML and JS itself what it is built by, but it all appears to be proprietary.
If I want to construct something similar in a robust fashion, what are some mature, open source technologies, libraries and frameworks I should be looking into?
I have found at least one framework with the necessary features for implementing the kind of system asked for in the question.
os.js is a Javascript framework with a "web desktop implementation for your browser with a fully-fledged window manager, Application APIs, GUI toolkits and filesystem abstraction.". As such, it has the essential features needed to craft a rich desktop environment, along with an underlying toolkit helping to tie it to backend functionality.
This is a good, extendable starting for crafting more basic solutions.
Reference:
http://os.js.org/
I am planning to use yui or jquery ui as front end for native c++ applications. I found no bindings present for this purpose. Will it be feasible/possible to do this ? If it is so, then how shall i proceed? please suggest advice.thanks
#rwik - I can see your point and desire to do what you intend and I would presume it to be a wise move to be making at this point in time considering the vast facilities browsers bring to the table as well as other aspects... I know we are in September now and I have come across this pretty late, but this is also for others who have a similar problem...
As far as I know, descending from JavaScript in to C++ is pretty murky, mostly due to the heterogenous nature of Javascript engines [Rhino, Tracemonkey, Spidermonkey, V8, Caracan,Charka, etc] employed by different browsers. But there are indeed ways to do this - depending on how dirty you want to get your hands and, indeed, on your abilities...
If you were to go with Google as a browser, you should take a look at projects such as cproxyv8 - http://code.google.com/p/cproxyv8/ or v8-juice - http://code.google.com/p/v8-juice/ which both offer interesting facilities...
My preference however is with Mozilla - simply because of XML User-Interface Language (XUL) which, if you do not know, allows you to speedily create your own user interfaces by giving direct access to its layout engine. I.e., you can even do away with the browser look and have your own independent layout while still having all browser facilities at your disposal.
With respect to C++/Javascript interaction - projects should exist for all alternatives - it pays to check.
I wish you good luck if you haven't already solved this problem.
I'd recommend QT instead.
However since native C++ has nothing to do with GUI I'm going to assume you are making a library in C++. However Yui is basically just JavaScript as is jQuery. So if you're asking can a web page or script be a C++ application the answer is not really.
What you could do is create a web service which uses your C++ library and exposes an interface for your web based application to call it indirectly.
One way to accomplish this is to embed WebKit into your application and provide your custom C++ functionality via a plugin. Both the WebKit framework on OSX and the Chromium Embedded Framework on Windows allow this style of application development. This allows you to write a UI using local Javascript/HTML files without the need for a webserver.
There are no direct bindings in JQuery and YUI because these libraries run in a web browser. Thus unless you’re planning to write your own browser plugins, the integration with the C++ will happen at the webserver that serves the data to the browser.
You could feed the data from your C++ code directly into JQuery/YUI by building web services directly in C++. But you might want to build a web application that will be serving your JQuery/YUI pages and that act as user interface controller (like in the MVC pattern). Web application languages like Java, PHP, etc. all have ways to call native code. Java has JNDI and in PHP you build extensions that link to C++ code.
unfortunatelly I have a GUI desktop application running on Windows and compiled using Borland C++ compiler. I know it's a very old technology, but that's the life. The company I work for have lots of free licences for QTP. But to be honest I don't like this tool and I want to change it with some open source alternative.
I have an experiance with automation Web services using Selenium, and some java swing applets.
I created my tests based on Java/JUnit. I'm wondering if I can create some tests in Java to test my C++ application - that would be great. If that's not possible if I can use any C/C++ libraries to create my own test framework.
That shouldn't be unit tests. I want to create rather a suite of functional tests.
I want to be able to localize objects like buttons, tables, cells, etc., perform action like clicking, changing focus, etc.
In addition I have access to source. But this is rather an old spaghetti code and I don't have much experiance with C++ apps.
What do you suggest?
We use Sikuli and have been quite happy with it. It works outside of the various automation APIs so it handles non-standard UIs very well.
Open Source Tools
White is a .NET based UI automation tool that can automate win32 applications among others types. As long as the controls used in your app expose their content/behaviour using Windows' standard UIAutomation hooks then you should be able to use it to do what you need.
UIAutomation Verify is another CodePlex project that works with the same APIs to automate UIA compliant UIs.
Visual Studio
Finally if you have access to it then Visual Studio 2010 has scripted UI test functionality built in Premium/Ultimate editions - Coded UI Tests. This supports .NET, native and web applications with record/replay and scripting functionality.
Robert, if your company already has licenses of QTP, I suggest you use that. It's the right tool for the job, and there's no sense reinventing the wheel if your company already owns it. It has its warts - my main objections are to the limited IDE and VBScript scripting language - but it did not become the market-leading tool by accident. There are not a lot of open-source options for testing a Windows C++ GUI app. The main commercial options you should consider are HP QTP, IBM Rational Functional Tester and VS2010 Coded UI Tests.
I agree with Robert that QTP is sometimes not the best tool, especially when you want it to wait for the desktop application to finish its processing. QTP commands are not synchronous, meaning that QTP does not wait until the previous command finishes execution before moving to the next call. We had a lot of issues with this. We are now planning to write a c# dll that can do this for QTP and include that in our QTP programs to make QTP to dynamically wait until the Desktop application finishes its processing. Currently, to my knowledge, no such wait functionality is supported by QTP.
Also, coming back to the object recognition issue, We did have a lot of issue with this and took the alternative route of invoking commands using Mnemonics through keystrokes in QTP. As Simon has already mentioned, this could be due to non-exposure of the GUI elements. We did not have much control in this regard and so we did not investigate further on this.
How do you unit test a large MFC UI application?
We have a few large MFC applications that have been in development for many years, we use some standard automated QA tools to run basic scripts to check fundamentals, file open etc. These are run by the QA group post the daily build.
But we would like to introduce procedures such that individual developers can build and run tests against dialogs, menus, and other visual elements of the application before submitting code to the daily build.
I have heard of such techniques as hidden test buttons on dialogs that only appear in debug builds, are there any standard toolkits for this.
Environment is C++/C/FORTRAN, MSVC 2005, Intel FORTRAN 9.1, Windows XP/Vista x86 & x64.
It depends on how the App is structured. If logic and GUI code is separated (MVC) then testing the logic is easy. Take a look at Michael Feathers "Humble Dialog Box" (PDF).
EDIT: If you think about it: You should very carefully refactor if the App is not structured that way. There is no other technique for testing the logic. Scripts which simulate clicks are just scratching the surface.
It is actually pretty easy:
Assume your control/window/whatever changes the contents of a listbox when the user clicks a button and you want to make sure the listbox contains the right stuff after the click.
Refactor so that there is a separate list with the items for the listbox to show. The items are stored in the list and are not extracted from whereever your data comes from. The code that makes the listbox list things knows only about the new list.
Then you create a new controller object which will contain the logic code. The method that handles the button click only calls mycontroller->ButtonWasClicked(). It does not know about the listbox or anythings else.
MyController::ButtonWasClicked() does whats need to be done for the intended logic, prepares the item list and tells the control to update. For that to work you need to decouple the controller and the control by creating a interface (pure virtual class) for the control. The controller knows only an object of that type, not the control.
Thats it. The controller contains the logic code and knows the control only via the interface. Now you can write regular unit test for MyController::ButtonWasClicked() by mocking the control. If you have no idea what I am talking about, read Michaels article. Twice. And again after that.
(Note to self: must learn not to blather that much)
Since you mentioned MFC, I assumed you have an application that would be hard to get under an automated test harness. You'll observe best benefits of unit testing frameworks when you build tests as you write the code.. But trying to add a new feature in a test-driven manner to an application which is not designed to be testable.. can be hard work and well frustrating.
Now what I am going to propose is definitely hard work.. but with some discipline and perseverance you'll see the benefit soon enough.
First you'll need some management backing for new fixes to take a bit longer. Make sure everyone understands why.
Next buy a copy of the WELC book. Read it cover to cover if you have the time OR if you're hard pressed, scan the index to find the symptom your app is exhibiting. This book contains a lot of good advice and is just what you need when trying to get existing code testable.
Then for every new fix/change, spend some time and understand the area you're going to work on. Write some tests in a xUnit variant of your choice (freely available) to exercise current behavior.
Make sure all tests pass. Write a new test which exercises needed behavior or the bug.
Write code to make this last test pass.
Refactor mercilessly within the area under tests to improve design.
Repeat for every new change that you have to make to the system from here on. No exceptions to this rule.
Now the promised land: Soon ever growing islands of well tested code will begin to surface. More and more code would fall under the automated test suite and changes will become progressively easier to make. And that is because slowly and surely the underlying design becomes more testable.
The easy way out was my previous answer. This is the difficult but right way out.
I realize this is a dated question, but for those of us who still work with MFC, the Microsoft C++ Unit Testing Framework in VS2012 works well.
The General Procedure:
Compile your MFC Project as a static library
Add a new Native Unit Test Project to your solution.
In the Test Project, add your MFC Project as a Reference.
In the Test Project's Configuration Properties, add the Include directories for your header files.
In the Linker, input options add your MFC.lib;nafxcwd.lib;libcmtd.lib;
Under 'Ignore Specific Default Libraries' add nafxcwd.lib;libcmtd.lib;
Under General add the location of your MFC exported lib file.
The https://stackoverflow.com/questions/1146338/error-lnk2005-new-and-delete-already-defined-in-libcmtd-libnew-obj has a good description of why you need the nafxcwd.lib and libcmtd.lib.
The other important thing to check for in legacy projects. In General Configuration Properties, make sure both projects are using the same 'Character Set'. If your MFC is using a Multi-Byte Character Set you'll need the MS Test to do so as well.
Though not perfect, the best I have found for this is AutoIt http://www.autoitscript.com/autoit3
"AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys). AutoIt is also very small, self-contained and will run on all versions of Windows out-of-the-box with no annoying "runtimes" required!"
This works well when you have access to the source code of the application being driven, because you can use the resource ID number of the controls you want to drive. In this way you do not have to worry about simulated mouse clicks on particular pixels. Unfortunately, in a legacy application, you may well find that the resource ID are not unique, which may cause problems. However. it is very straightforward to change the IDs to be unique and rebuild.
The other issue is that you will encounter timing problems. I do not have a tried and true solution for these. Trial and error is what I have used, but this is clearly not scalable. The problem is that the AutoIT script must wait for the test application to respond to a command before the script issues the next command or check for the correct response. Sometimes it is not easy to find a convenient event to wait and watch for.
My feeling is that, in developing a new application, I would insist on a consistent way to signal "READY". This would be helpful to the human users as well as test scripts! This may be a challenge for a legacy application, but perhaps you can introduce it in problematical points and slowly spread it to the entire application as maintenance continues.
Although it cannot handle the UI side, I unit test MFC code using the Boost Test library. There is a Code Project article on getting started:
Designing Robust Objects with Boost
Well we have one of these humongous MFC Apps at the workplace. Its a gigantic pain to maintain or extend... its a huge ball of mud now but it rakes in the moolah.Anyways
We use Rational Robot for doing smoke tests and the like.
Another approach that has had some success is to create a small product-specific language and script tests that use VBScript and some Control handle spying magic. Turn common actions into commands.. e.g. OpenDatabase would be a command that in turn will inject the required script blocks to click on Main Menu > File > "Open...". You then create excel sheets which are a series of such commands. These commands can take parameters too. Something like a FIT Test.. but more work. Once you have most of the common commands identified and scripts ready. It's pick and assemble scripts (tagged by CommandIDs) to write new tests. A test-runner parses these Excel sheets, combines all the little script blocks into a test script and runs it.
OpenDatabase "C:\tests\MyDB"
OpenDialog "Add Model"
AddModel "M0001", "MyModel", 2.5, 100
PressOK
SaveDatabase
HTH
Actually we have been using Rational Team Test, then Robot, but in recent discussions with Rational we discovered they have no plans to support Native x64 applications focusing more on .NET, so we decided to switch Automated QA tools. This is great but licensing costs don't allow us to enable it for all developers.
All our applications support a COM API for scripting, which we regression test via VB, but this tests the API no the application as such.
Ideally I would be interested on how people integrate cppunit and similar unit testing frameworks into the application at a developer level.