I've heard of unit testing, and written some tests myself just as tests but never used any testing frameworks. Now I'm writing a wxPython GUI for some in-house data analysis/visualisation libraries. I've read some of the obvious Google results, like http://wiki.wxpython.org/Unit%20Testing%20with%20wxPython and its link http://pywinauto.openqa.org/ but am still uncertain where to start.
Does anyone have experience or good references for someone who sort of knows the theory but has never used any of the frameworks and has no idea how it works with GUIs?
I am on a Windows machine developing a theoretically cross-platform application that uses NumPy, Matplotlib, Newville's MPlot package, and wxPython 2.8.11. Python 2.6 with plans for 3.1. I work for a bunch of scientists, so there is no in-house unit-testing policy.
If you want to unit-test your application, you haven't to focus on GUI testing techniques. It is much better to write the application using MVC, MVP, or other meta-pattern like these. So you get business logic and presentation layer separated.
It is much more important to cover the business layer with tests since this is your code. Presentation layer is tested already by wxWidgets developers. To test the business layer it will be enough just basic tools like standard unittest module and maybe nose.
To make sure the whole application behave correctly, you should add few acceptance tests that will test functionality from end to end. These will deal with GUI, but there will be few such tests comparing to number of unit-tests.
If you will limit yourself with acceptance tests only, you'll get low coverage, fragile and very slow test code base.
To unit test your application without requiring lots of mock objects/stubs, your GUI's event handlers should basically delegate to other method calls, passing in values from the Event object as parameters to the delegated method.
Otherwise you'll be unable to test your application without having to mock wx's objects.
Take a look at the PyPubSub project for a great module to help with MVC.
In one early project of mine I really test wxPython application using GUI layer. So tests really spin live wxApp object, pops up real windows and then starts messing with a real MainLoop(). Very soon I realize it was a wrong way to do testing. My tests was run very slow and unreliable. Much better way is to separate GUI stuff aside and test only the "model" level of your application. Note that you can actually create model for presentation level logic (model that represent some visual part of your application) and test it. But this model should not involve any "real" gui objects (windows, dialogs, widgets).
Related
what is the different purpose of those both? I mean, in which condition I should do each of them?
as for the example condition. if you have the backend server and several front-end webs, which one you'll do?do-unit testing the backend server first or do-UI testing in the web UI first?
given the condition, the server and the front-end webs already exist, so it's not an iterative design to build along with (TDD)...
Unit testing aims to test small portions of your code (individual classes / methods) in isolation from the rest of the world.
UI testing may be a different name for system / functional / acceptance testing, where you test the whole system together to ensure it does what it is supposed to do under real life circumstances. (Unless by UI testing you mean usability / look & feel etc. testing, which is typically constrained to details on the UI.)
You need both of these in most of projects, but at different times: unit testing during development (ideally from the very beginning, TDD style), and UI testing somewhat later, once you actually have some complete end-to-end functionality to test.
If you already have the system running, but no tests, practically you have legacy code. Strive to get the best test coverage achievable with the least effort first, which means high level functional tests. Adding unit tests is needed too, but it takes much more effort and starts to pay back later.
Recommended reading: Working Effectively with Legacy Code.
Unit test should always be done. Unittests are there to provide proof that each UNIT (read: object) of your technical solution delivers the expected results. To put it very (maybe too) simple, user testing is there to verify that your system fulfills the needs and demands of the user.
Test pyramid [1] is important concept here, well described by Martin Fawler.
In short, tests that run end-to-end through the UI are: brittle and expensive to write. You may consider test recording tools [2] to speed recording and re-recording up. Disclaimer - I'm developer of such tool.
[1] https://martinfowler.com/articles/practical-test-pyramid.html
[2] https://anwendo.com
In addition to the accepted answer, today I just came up with this question of why not just programmatically trigger layout functions and then unit-test your logic around that as well?
The answer I got from a senior dev was: programmatically trigger layout functions will not be an absolute copy of the real user-experience. In the real world, the system will trigger many callbacks, like when the user of an app backgrounds or foregrounds the app. Obviously you can trigger such events manually and test again, but would you be sure you got all events in all sequences right?!
The real user-experience is one where user makes actual network calls, taps on screens, loads multiple screen on top of each other and at times you might get system callbacks. Callbacks which you forgot to mock that you didn't properly mock. In unit-tests you're mainly testing in isolation. In UI test, you setup the app, may have to login, etc. That stack you build is much more complex vs a unit-test. Hence it's better to not mix unit-testing with UI testing.
Now I'm experiencing a problem about in testing,
We have developed a platform with C++, and this platform include 2 layers, one is engine layer, the other one is widget layer, and I'm a dev in widget layer,
The widget layer used by client app (those app is our product), now the problem is this:
1, We want to add some unit test for widget layer to enhance our whole platform quality and reliability, but in fact, this platform has been release for several versions, so now, what's kind of unit test could be most suitable for our project? To add some unit test in kinds of validation/bad input/path cover/stress or to add some unit test that simulate the client scenario? I think the former one will enhance code coverage very well, but i also think about that if we can simulate how our client using those api in their application, those unit test maybe more make sense for our platform, right?
2, just as i mentioned before, i'm in widget layer which is above and depended on engine layer, but we has no authority for engine layer code, and almost all widget API could call in engine layer, so it's difficult for us to implement good path cover unit test for those api, because we have no idea(and no document)about engine layer code, but i still want to ask, is there any ways, tools or frameworks could make someone write high quality unit test to cover path for those black box API?
Since you work in the widget API, and do not have access to the engine code, write your own engine instead.
Use this engine to test the widget code. The engine can be very thin, almost not functional at all, doing just the minimum to fool the widget layer into believing it works.
The test engine can even be a shim on top of the real engine. How you implement this varies a lot depending on the environment you work in, but in Windows it could be that a DLL forwards all its calls to the "real" DLL, but logs calls to it first.
Another technique: since you do have access to the Widget layer, and the engine API is very big, you can reprogram the Widget layer to use the "test layer" in only some places, but the real layer in remaining places. This can be combined with the shim technique.
Suitable (or available) tools for generating stubs or stub engines varies a lot depending on which language and environment you are working in.
Also see this DirectDraw hack for inspiration, it started life as a thin, logging shim upon the original DirectDraw implementation.
You do not mention what widget framework you are using but Qt has a test framework that allows the test code to trigger the same Qt signals that normally get generated by user input (for example mouse clicks or key pressed). See this for more detail.
If you are using a different framework, there might be similar functionality that you could use.
I m just starting using gwt and so far so good, however after reading some sample code I wonder is it necesary to have a high level of test coverage? (I can see that most code is declarative and then add some attributes I can see the sense in checking so me particular attributes are there but not all)
Also i would be interested to know anything about what are the gotchas in TDDing with GWT
I m using eclipse so also if you are really happy with some particualrs add ins for GWT I would be happy to hear about that
Thanks for the input
edit: maybe I m asking a very wide question, but even little pieces of information will help
I come from having nvelocity views with jquery/extJs/prototype/scriptaculous and this is a bit different
When designing GWT applications to be easily testable, it's best to move as much logic out of the view as possible. Use a design pattern which makes GUI testing easier such as Model-View-Presenter (MVP), which is used widely in building desktop applications (The C#/.NET folks have written a lot about this pattern.)
You can use GWTTestCases to test remote communication and code that ultimately executes raw JavaScript (most of the GWT core classes require this, especially widgets). However, these tests are slow to execute, so you should prefer designs which put all that logic in objects that can be tested in plain ol' JUnit TestCases.
For more information about writing GWT applications test-first, I've written an article for Better Software magazine, which is available as a PDF online at my blog.
I think the best reference at the moment would be this Testing Methodologies Using Google Web Toolkit
I think you asked a pretty broad question, which is part of the reason why you didn't get a reply for a while.
Compared to traditional AJAX web development, one could argue a GWT application requires less testing. Because the GWT team has worked so hard to make sure that its widgets work consistently across all web browsers, you don't have to worry about cross-browser compatibility nearly as much for your own application.
That frees you up to focus on your own application. Create a separate test case for each of your own custom widgets and test that they behave as you expect, and then write higher-level tests for each module. Take the extra step to make your tests fully automatable - that way every time you make a change or are about to release, it's easy to run all of your tests.
http://code.google.com/docreader/#p=google-web-toolkit-doc-1-5&s=google-web-toolkit-doc-1-5&t=DevGuideJUnitIntegration
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am working on the issue of testing my GUI and I am not entirely sure of the best approach here. My GUI is built using a traditional MVC framework so I am easily able to test the logic parts of the GUI without bringing up the GUI itself. However, when it comes to testing the functionality of the GUI, I am not really sure if I should worry about individually testing GUI components or if I should mainly just focus on functional testing the system. It is a pretty complex system in which testing the GUI frequently involves sending a message to the server and then observing the response on the GUI. My initial thoughts are that functional testing is the way to go here since I need a whole system running to really test the UI. Comments on this issue would be appreciated.
Other GUI-testing tools I can offer are:
Thoughtworks White,
PyWinAuto,
AutoIt,
AutoHotKey.
One thing to keep in mind when trying to automate GUIs is that the only way you can do that is to build the GUI with automation in mind. Crush devs that think their GUIs should not support testability early on in the project and happily expose all the hooks that can help in automation on demand as your testing needs require that.
You have (at least) 2 issues - the complexity of the environment (the server) and the complexity of the GUI.
There are many tools for automating GUI testing. All of them are more or less fragile and require pretty much constant maintenance in the face of changing layout. There is benefit to be gained from using them, but it's a long term benefit.
The environment, on the other hand, is an area that can be tamed. If your application is architected using the Dependency Injection/Inversion technique (where you 'inject' the server component into the application), then you can use a 'mock' of the relevant server interfaces to enable you to script test cases.
Combining these two techniques will allow you to automate GUI testing.
Depending on where in the spectrum of MVC (that's an overused term) you sit, testing the view could be a mechanical process of ensuring that the correct model methods are called in response to the correct inputs to the view to testing some client side validation to who knows.
A lot of the patterns that have been evolved out of MVC (I'm thinking passive view, supervising controller) are striving to make the view require very little testing because it's really just wiring user inputs to the presenter or model (depending on the exact variant of the pattern you're using).
"testing the GUI frequently involves sending a message to the server and then observing the response on the GUI" This statement worries me.
I'm immediately thinking that the GUI should be tested using a mock or stub of the server to test that the correct interactions are occurring and the GUI responds appropriately.
If you need automated functional tests of the server, I don't see the need to have the GUI involved in those.
Mercury QuickTest Pro, Borland SilkTest, and Ranorex Recorder are some GUI testing tools.
If your application is web-based you can write tests using tools like WatiN or Selenium.
If your application is Windows .NET based, you could try White.
My advice: forget the traditional GUI testing. It's too expensive. Coding the tests takes a lot of time, the tools aren't really stable so you will get unreliable test results. The coupling between the code and the test is very strong and you'll spend a lot of time with the maintenance.
The new trend is to ignore the GUI tests. See the ModelViewPresenter pattern from Fowler as a guideline link text
The clearest way I can say this is:
Don't waste your time writing automated GUI tests.
Especially when your working with an MVC app - in your case, when you send a message to the server, you can make sure the right message number comes back and be done. You can add some additional cases - or another test completely to make sure that the GUI is converting the message id's into the right strings, but you just need to run that test once.
We do incorporate GUI testing in our project, and it has its side effects. The developers however have one critical design principle: Keep the GUI layer as thin as possible!
That means no logic in the GUI classes. Separate this in presentation models responsible for input validation etc.
For testing on a Unix machine we use the Xvfb server as the DISPLAY when running the tests.
Try the hallway usability test. It's cheap and useful: go to the nearest hallway, grab the first person that passes, make them sit at your computer and use your software. Watch over their shoulder, you will see what they try to do, what frustrates them, and so on. Do this a few times and notice the patterns.
What you're looking for is "acceptance testing." How you do it depends on the frameworks you're using, what type of application you are creating and in what language. If you google your particular technology and the above phrase, you should find some tools you can use.
I've found WinTask to be a very good way to do GUI testing. Provided you don't constantly change the way the OS refers to each element of the UI, WinTask addresses the UI elements by name, so even if the layout changes, the UI elements can still be pressed / tweaked / selected.
Don't miss the 'U' in 'GUI'
I mean: if what you're trying to test is all works right and works as it was planned to work, then you may follow Seb Rose's answer.
But please, don't forget a USER interface has to be made thinking about USERS, and not ANY user but the TARGET USER the application was made for. So, after you are sure all works like it have to work, put every single view/screen/form in a test with a team made of users representing every group of different users that may use your application: advanced users, administrators, MS Office users, low computer profile users, high computer profile users... and then, get the critiques of every user, make a mix, re-touch your GUI if it's neccesary and back again to GUI user's test.
For SIMPLE Web based GUI testing try iMacros ( a simple Firefox plug-in , has a cool feature to send the entire test to another person )
Note that SIMPLE was spelled with Initials ...
Unit testing and ASP.NET web applications are an ambiguous point in my group. More often than not, good testing practices fall through the cracks and web applications end up going live for several years with no tests.
The cause of this pain point generally revolves around the hassle of writing UI automation mid-development.
How do you or your organization integrate best TDD practices with web application development?
Unit testing will be achievable if you separate your layers appropriately. As Rob Cooper implied, don't put any logic in your WebForm other than logic to manage your presentation. All other stuff logic and persistence layers should be kept in separate classes and then you can test those individually.
To test the GUI some people like selenium. Others complain that is a pain to set up.
I layer out the application and at least unit test from the presenter/controller (whichever is your preference, mvc/mvp) to the data layer. That way I have good test coverage over most of the code that is written.
I have looked at FitNesse, Watin and Selenium as options to automate the UI testing but I haven't got around to using these on any projects yet, so we stick with human testing. FitNesse was the one I was leaning toward but I couldn't introduce this as well as introducing TDD (does that make me bad? I hope not!).
This is a good question, one that I will be subscribing too :)
I am still relatively new to web dev, and I too am looking at a lot of code that is largely untested.
For me, I keep the UI as light as possible (normally only a few lines of code) and test the crap out of everything else. At least I can then have some confidence that everything that makes it to the UI is as correct as it can be.
Is it perfect? Perhaps not, but at least it as still quite highly automated and the core code (where most of the "magic" happens) still has pretty good coverage..
I would generally avoid testing that involves relying on UI elements. I favor integration testing, which tests everything from your database layer up to the view layer (but not the actual layout).
Try to start a test suite before writing a line of actual code in a new project, since it's harder to write tests later.
Choose carefully what you test - don't mindlessly write tests for everything. Sometimes it's a boring task, so don't make it harder. If you write too many tests, you risk abandoning that task under the weight of time-consuming maintenance.
Try to bundle as much functionality as possible into a single test. That way, if something goes wrong, the errors will propagate anyway. For example, if you have a digest-generating class - test the actual output, not every single helper function.
Don't trust yourself. Assume that you will always make mistakes, and so you write tests to make your life easier, not harder.
If you are not feeling good about writing tests, you are probably doing it wrong ;)
A common practice is to move all the code you can out of the codebehind and into an object you can test in isolation. Such code will usually follow the MVP or MVC design patterns. If you search on "Rhino Igloo" you will probably find the link to its Subversion repository. That code is worth a study, as it demonstrate one of the best MVP implementations on Web Forms that I have seen.
Your codebehind will, when following this pattern, do two things:
Transit all user actions to the presenter.
Render data provided by the presenter.
Unit testing the presenter should be trivial.
Update: Rhino Igloo can be found here: https://svn.sourceforge.net/svnroot/rhino-tools/trunk/rhino-igloo/
There have been tries on getting Microsoft's free UI Automation (included in .NET Framework 3.0) to work with web applications (ASP.NET). A german company called Artiso happens to have written a blog entry that explains how to achieve that (link).
However, their blogpost also links an MSDN Webcasts that explains the UI Automation Framework with winforms and after I had a look at this, I noticed you need the AutomationId to get a reference to the respecting controls. However, in web applications, the controls do not have an AutomationId.
I asked Thomas Schissler (Artiso) about this and he explained that this was a major drawback on InternetExplorer. He referenced an older technology of Microsoft (MSAA) and was hoping himself that IE8 will do this better.
However, I was also giving Watin a try and it seems to work pretty well. I even liked Wax, which allows to implement simple testcases via Microsoft Excel worksheets.
Ivonna can unit test your views. I'd still recommend moving most of the code to other parts. However, some code just belongs there, like references to controls or control event handlers.