Can a UI application be unit tested? Lets consider a UI application which has a simple login form. What would form part of unit test for this UI application. Should the backend response validation form part of unit test for login form?
In my opinion, It should include;
appropriate rendering of UI components in the form.
enabling/Disabling of components based on user actions (password can not be empty message when password is not entered and form is submitted).
Are there any guidelines/rule of thumbs which should be used while devising unit test cases for UI application? Should an application behavior considered in unit testing?
In general, testing UI components is not an easy task - you need a good architecture on the tested code, quite some tools (which may come with some license costs...), and at least one virtual machine.
The most important part is to have some 'Model-View-younameit' architecture in place, because only this allows you to isolate the UI stuff from the rest of the application. A test for the login form can then be as follows:
A tool (e.g. White or Ranorex) mimics a user typing in some user/password combination.
The test verifies (by the help of e.g. a mocking framework or Typemock), that the correct methods are called (or not called under certain circumstances).
What you have to avoid for UI testing in any case is calling some business logic underneath. If you can't isolate the UI this way (which is often the case especially for legacy projects...), then automating UI testing may not be beneficial at all, because it often comes with way too much effort in such a situation. Sometimes you better go with manual testing...
HTH!
Thomas
UI testing is not unit testing in the strict sense, however there are tools/frameworks to help that. I assume you mean a web UI - for this you can try e.g. Selenium or HttpUnit.
Note that "appropriate rendering of UI components" is a very loose term - it can mean things which can not be verified automatically, only by human eye.
Testing functionality via the UI is always more cumbersome than testing the code directly via classic unit tests. Hence it is always recommended to separate the business logic from the UI as much as possible - then you can thoroughly test all the nooks and crannies of the logic itself via unit tests, and need only integration tests on the UI to verify that the app as a whole works as expected.
In your example, ideally the input validation and the authentication would be separated from the GUI, so you could unit test these (preferably by mocking out the DB). Thus you could verify that the login works as expected for all sorts of normal/abnormal inputs (empty userid/password, strange characters, very long input, logging in the same user twice, ...). Then in the UI tests you need only test a few common test cases to verify that the app indeed is functional.
If your UI is a native graphical application, you may want to consider AutoIt (Windows only), which allows you to programmatically click any place on the screen and simulate text typed into an widget or just keyboard presses. It also has the ability to read the color of pixels at specified positions. Or you can use it to take screen shots and do an image comparison.
If it's web based, you may also want to consider using xpath to determine (non-graphically) whether elements exist in your generated HTML. It doesn't help you with how pages are rendered though. AutoIt may help with that as well, but you won't be able to perform the same type of test on a different platform, assuming you would like all users from all platforms to be able to reasonably access your page.
If your UI is console based, consider using Expect. I prefer the Perl based expect, which is based on the TCL Expect.
I know I'm not answering directly about rules or guidelines, but I think knowing what tools are available out there will open up the possibilities that you think may not exist or had not considered. Just as an example, you can use AutoIt to determine whether a component is disabled or enabled because usually the widget is of a different color from the enabled version when it is disabled. So by using AutoIt to obtain the color of a pixel or region of the screen, you can programmatically test whether that widget is enabled or not.
Related
We're working on a project that has very little business logic. Essentially, we gather information from the user and call a REST API with that information. We get back information that we display.
Is there any point in trying to create unit tests for this kind of class?
I guess it depends on the importance of this application and this specific functionality. If it is business critical, then most probably it is a good idea to wrap it by unit tests. Even with a very little business logic you may run into exceptional cases, which your application should be ready for:
No network
Unreliable network
Incorrect / malicious user input
Incorrect server response
Alternatively, it may make sense to replace unit tests by UI tests or to have both. In this case it will be possible to test the user flow rather that a very simple bits and bytes.
Ember's Acceptance Test Helpers provide a nice abstraction layer for writing pure acceptance tests based on user actions, but isn't always enough.
In Ruby/Rails testing with Capybara the underlying test driver (selenium-webdriver, phantomjs, etc), can be accessed in a test case by calling page.driver, then its API used directly to do things not provided by the abstract API, like handling cookies and window sizing.
For example:
page.driver.browser.manage.window.resize_to(width, height)
visit '/foo/bar'
page.driver.browser.action.key_down(:alt).send_keys("c").key_up(:alt).perform
Can this be done in an Ember-CLI project, with default everything (qunit, testem)?
The context for this problem is we have a feature that forces the user to scroll to the bottom before clicking OK (like clickwrap license agreements sometimes make you, but it's for something else).
An acceptance test for this is working fine in Chrome, but failing in phantomjs, and we think it's probably because of the window or viewport size.
http://phantomjs.org/api/webpage/property/viewport-size.html
There may be other ways to fix this, but I'm still interested in accessing the test drivers directly as I think it's a useful tool to have.
what is the different purpose of those both? I mean, in which condition I should do each of them?
as for the example condition. if you have the backend server and several front-end webs, which one you'll do?do-unit testing the backend server first or do-UI testing in the web UI first?
given the condition, the server and the front-end webs already exist, so it's not an iterative design to build along with (TDD)...
Unit testing aims to test small portions of your code (individual classes / methods) in isolation from the rest of the world.
UI testing may be a different name for system / functional / acceptance testing, where you test the whole system together to ensure it does what it is supposed to do under real life circumstances. (Unless by UI testing you mean usability / look & feel etc. testing, which is typically constrained to details on the UI.)
You need both of these in most of projects, but at different times: unit testing during development (ideally from the very beginning, TDD style), and UI testing somewhat later, once you actually have some complete end-to-end functionality to test.
If you already have the system running, but no tests, practically you have legacy code. Strive to get the best test coverage achievable with the least effort first, which means high level functional tests. Adding unit tests is needed too, but it takes much more effort and starts to pay back later.
Recommended reading: Working Effectively with Legacy Code.
Unit test should always be done. Unittests are there to provide proof that each UNIT (read: object) of your technical solution delivers the expected results. To put it very (maybe too) simple, user testing is there to verify that your system fulfills the needs and demands of the user.
Test pyramid [1] is important concept here, well described by Martin Fawler.
In short, tests that run end-to-end through the UI are: brittle and expensive to write. You may consider test recording tools [2] to speed recording and re-recording up. Disclaimer - I'm developer of such tool.
[1] https://martinfowler.com/articles/practical-test-pyramid.html
[2] https://anwendo.com
In addition to the accepted answer, today I just came up with this question of why not just programmatically trigger layout functions and then unit-test your logic around that as well?
The answer I got from a senior dev was: programmatically trigger layout functions will not be an absolute copy of the real user-experience. In the real world, the system will trigger many callbacks, like when the user of an app backgrounds or foregrounds the app. Obviously you can trigger such events manually and test again, but would you be sure you got all events in all sequences right?!
The real user-experience is one where user makes actual network calls, taps on screens, loads multiple screen on top of each other and at times you might get system callbacks. Callbacks which you forgot to mock that you didn't properly mock. In unit-tests you're mainly testing in isolation. In UI test, you setup the app, may have to login, etc. That stack you build is much more complex vs a unit-test. Hence it's better to not mix unit-testing with UI testing.
Now I'm experiencing a problem about in testing,
We have developed a platform with C++, and this platform include 2 layers, one is engine layer, the other one is widget layer, and I'm a dev in widget layer,
The widget layer used by client app (those app is our product), now the problem is this:
1, We want to add some unit test for widget layer to enhance our whole platform quality and reliability, but in fact, this platform has been release for several versions, so now, what's kind of unit test could be most suitable for our project? To add some unit test in kinds of validation/bad input/path cover/stress or to add some unit test that simulate the client scenario? I think the former one will enhance code coverage very well, but i also think about that if we can simulate how our client using those api in their application, those unit test maybe more make sense for our platform, right?
2, just as i mentioned before, i'm in widget layer which is above and depended on engine layer, but we has no authority for engine layer code, and almost all widget API could call in engine layer, so it's difficult for us to implement good path cover unit test for those api, because we have no idea(and no document)about engine layer code, but i still want to ask, is there any ways, tools or frameworks could make someone write high quality unit test to cover path for those black box API?
Since you work in the widget API, and do not have access to the engine code, write your own engine instead.
Use this engine to test the widget code. The engine can be very thin, almost not functional at all, doing just the minimum to fool the widget layer into believing it works.
The test engine can even be a shim on top of the real engine. How you implement this varies a lot depending on the environment you work in, but in Windows it could be that a DLL forwards all its calls to the "real" DLL, but logs calls to it first.
Another technique: since you do have access to the Widget layer, and the engine API is very big, you can reprogram the Widget layer to use the "test layer" in only some places, but the real layer in remaining places. This can be combined with the shim technique.
Suitable (or available) tools for generating stubs or stub engines varies a lot depending on which language and environment you are working in.
Also see this DirectDraw hack for inspiration, it started life as a thin, logging shim upon the original DirectDraw implementation.
You do not mention what widget framework you are using but Qt has a test framework that allows the test code to trigger the same Qt signals that normally get generated by user input (for example mouse clicks or key pressed). See this for more detail.
If you are using a different framework, there might be similar functionality that you could use.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am working on the issue of testing my GUI and I am not entirely sure of the best approach here. My GUI is built using a traditional MVC framework so I am easily able to test the logic parts of the GUI without bringing up the GUI itself. However, when it comes to testing the functionality of the GUI, I am not really sure if I should worry about individually testing GUI components or if I should mainly just focus on functional testing the system. It is a pretty complex system in which testing the GUI frequently involves sending a message to the server and then observing the response on the GUI. My initial thoughts are that functional testing is the way to go here since I need a whole system running to really test the UI. Comments on this issue would be appreciated.
Other GUI-testing tools I can offer are:
Thoughtworks White,
PyWinAuto,
AutoIt,
AutoHotKey.
One thing to keep in mind when trying to automate GUIs is that the only way you can do that is to build the GUI with automation in mind. Crush devs that think their GUIs should not support testability early on in the project and happily expose all the hooks that can help in automation on demand as your testing needs require that.
You have (at least) 2 issues - the complexity of the environment (the server) and the complexity of the GUI.
There are many tools for automating GUI testing. All of them are more or less fragile and require pretty much constant maintenance in the face of changing layout. There is benefit to be gained from using them, but it's a long term benefit.
The environment, on the other hand, is an area that can be tamed. If your application is architected using the Dependency Injection/Inversion technique (where you 'inject' the server component into the application), then you can use a 'mock' of the relevant server interfaces to enable you to script test cases.
Combining these two techniques will allow you to automate GUI testing.
Depending on where in the spectrum of MVC (that's an overused term) you sit, testing the view could be a mechanical process of ensuring that the correct model methods are called in response to the correct inputs to the view to testing some client side validation to who knows.
A lot of the patterns that have been evolved out of MVC (I'm thinking passive view, supervising controller) are striving to make the view require very little testing because it's really just wiring user inputs to the presenter or model (depending on the exact variant of the pattern you're using).
"testing the GUI frequently involves sending a message to the server and then observing the response on the GUI" This statement worries me.
I'm immediately thinking that the GUI should be tested using a mock or stub of the server to test that the correct interactions are occurring and the GUI responds appropriately.
If you need automated functional tests of the server, I don't see the need to have the GUI involved in those.
Mercury QuickTest Pro, Borland SilkTest, and Ranorex Recorder are some GUI testing tools.
If your application is web-based you can write tests using tools like WatiN or Selenium.
If your application is Windows .NET based, you could try White.
My advice: forget the traditional GUI testing. It's too expensive. Coding the tests takes a lot of time, the tools aren't really stable so you will get unreliable test results. The coupling between the code and the test is very strong and you'll spend a lot of time with the maintenance.
The new trend is to ignore the GUI tests. See the ModelViewPresenter pattern from Fowler as a guideline link text
The clearest way I can say this is:
Don't waste your time writing automated GUI tests.
Especially when your working with an MVC app - in your case, when you send a message to the server, you can make sure the right message number comes back and be done. You can add some additional cases - or another test completely to make sure that the GUI is converting the message id's into the right strings, but you just need to run that test once.
We do incorporate GUI testing in our project, and it has its side effects. The developers however have one critical design principle: Keep the GUI layer as thin as possible!
That means no logic in the GUI classes. Separate this in presentation models responsible for input validation etc.
For testing on a Unix machine we use the Xvfb server as the DISPLAY when running the tests.
Try the hallway usability test. It's cheap and useful: go to the nearest hallway, grab the first person that passes, make them sit at your computer and use your software. Watch over their shoulder, you will see what they try to do, what frustrates them, and so on. Do this a few times and notice the patterns.
What you're looking for is "acceptance testing." How you do it depends on the frameworks you're using, what type of application you are creating and in what language. If you google your particular technology and the above phrase, you should find some tools you can use.
I've found WinTask to be a very good way to do GUI testing. Provided you don't constantly change the way the OS refers to each element of the UI, WinTask addresses the UI elements by name, so even if the layout changes, the UI elements can still be pressed / tweaked / selected.
Don't miss the 'U' in 'GUI'
I mean: if what you're trying to test is all works right and works as it was planned to work, then you may follow Seb Rose's answer.
But please, don't forget a USER interface has to be made thinking about USERS, and not ANY user but the TARGET USER the application was made for. So, after you are sure all works like it have to work, put every single view/screen/form in a test with a team made of users representing every group of different users that may use your application: advanced users, administrators, MS Office users, low computer profile users, high computer profile users... and then, get the critiques of every user, make a mix, re-touch your GUI if it's neccesary and back again to GUI user's test.
For SIMPLE Web based GUI testing try iMacros ( a simple Firefox plug-in , has a cool feature to send the entire test to another person )
Note that SIMPLE was spelled with Initials ...