We develop custom survey web sites and I am looking for a way to automate the pattern testing of these sites. Surveys often contain many complex rules and branches which are triggered on how items are responded too. All surveys are rigorously tested before being released to clients. This testing results in a lot of manual work. I would like to learn of some options I could use to automate these tests by responding to questions and verifying the results in the database. The survey sites are produced by an engine which creates and writes asp pages and receives the responses to process into a database. So the only way I can determine to test the site is to interact with the web pages themselves. I guess in a way I need to build some type of bot; I really don't know much about the design behind them.
Could someone please provide some suggestions on how to achieve this? Thank you for your time.
Brett
Check out selenium: http://selenium.openqa.org/
Also, check out the answers to this other question: https://stackoverflow.com/questions/484/how-do-you-test-layout-design-across-multiple-browsersoss
You could also check out WatiN.
Sounds like your engine could generate a test script using something like Test::WWW::Mechanize
Usual test methodologies applies; white box and black box.
White box testing for you may mean instrumenting your application to be able to make it go into a particular state, then you can predict the the result you expect.
Black box may mean that you hit a page, then consider of the possible outcomes valid. Repeat and rinse till you get sufficient coverage.
Another thing we use is monitoring statistics for our service. Did we get the expected number of hits on this page. We routinely run a/b tests, and I have run a/b tests against refactored code to verify that nothing changed before rolling things out.
/Allan
I can think of a couple of good web application testing suites that should get the job done - one free/open source and one commercial:
Selenium (open source/cross platform)
TestComplete (commercial/Windows-based)
Both will let you create test suites by verifying database records based on interactions with the web app.
The fact that you're Windows/ASP based might mean that TestComplete will get you up and running faster, as it's native to Windows and .NET. You can download a free trial to see if it'll work for you before making the investment.
Check out the unit testing framework 'lime' that comes with the Symfony framework. http://www.symfony-project.org/book/1_0/15-Unit-and-Functional-Testing. You didn't mention you language, lime is php.
I would suggest the mechanize gem,available for ruby . It's pretty intuitive to use .
I use the QEngine(commerical) for the same purpose. I need to add a data and check the same in the UI. I write one script which does this and call that in a loop. the data can be passed via either csv or excel.
check that www.qengine.com , you can try Watir also.
My proposal is QA Agent (http://qaagent.com). It seems this is a new approach because you do not need to install anything. Just develop your web tests in the browser based ide. By the way you can develop your tests using jQuery and java script. Really cool!
Related
So i have this huge SF2 project, which is luckily pretty 'OK' written. Services are there, background jobs are there, no god classes, it's testable--but, i never gotten any further than just unit-testing stuff, so the question is basically, where do i start taking this further.
The project consists of SF2 and all the yada yada, Doctrine2, Beanstalkd, Gaufrette, some other abstractions--its fine.
The one problem it has is some gluecode in controllers here and there, but i don't see it as a big problem since functional tests are going to me the main focus.
The infrastructure is setup pretty ok as well, its covered by docker so CI is going to work out well also.
But it has basically gotten too large to manually test any longer, so i want full functional coverage on short notice, and let the unit-testing grow over time. (Gonna dive into the isolated objects as they need future adjustments and build test for them in due course)
So i got the unit-testing covered, thats going to need to grow over time, but i want to make some steps towards the functional testing to get some quick gains on the testing dep. YESTERDAY.
My plan as of now is use Behat and Mink for this, the tests are going to be huge, so i might as well want to have it set as stories instead of code. Behat also seem to have a extension for Symfony' BrowserKit.
There are plenty of services and external things happening, but they are all isolated by services, so i can mock them through the test environment service config i guess.
Please some advice here if there is as better way
I'm also going to need fixtures, i'm using Alice for generating some fixtures so far, seems nice together with the doctrine extension, don't think there are "better" options on this one.
How should i test external services? Im mocking things as a Facebook service, but i also want to really test it to some test account, is this advisable? I know that this goes beyond its scope, the service has to be mocked and tested in every way possible to "ensure its working" according to the purist. But in the end of the day it still breaks because of some API key or other problem in the connection, which i cant afford really. So please advice here also
All your suggestions to use other tools are welcome ofcourse, and especially if there is a good book that covers my story.
I'm glad you brought up behat, I was going to suggest the same thing.
I would consider starting with your most business critical pieces; unit test the extremely important business logic and use behat on the rest.
For the most part, I would create stubs for your services that have expected output for expected input. That way you can create failures based on specific input. You can override your services in your test config.
Another approach would be to do very thin functional testing where you make GET requests to all of your endpoints and look for 200's. This is a very quick way to make sure that your pages are at least loading. From there, you can start writing tests for your POST endpoints and expanding your suite further with more detailed test cases.
I am Unit testing on the client side of a GWT+SmartGWT application.
First I tested with GwtTestCase. Too long for unit testing a huge application. GwtTestSuite doesn't help. It still takes too much time to execute. (more, it asked me to start a browser when it's testing)
Then gwt-test-utils : Great framework. Sadly, my javassist version is 3.5 and need at least the 3.11. Javassist is used by gilead so I can't touch this. So, no gwt-test-utils...
I saw Selenium. That's just great. With htmlunit driver, it's fast and useful. Simplest way to test a webapp. Problem here is SmartGWT generates it's own IDs when it generates the web page. So I can't get the TextItems and fill them since those IDs are constantly changing. I found that it could be solved by using setID() before the initialization of the widget. But that's the ID of the scLocator and not an HTML ID. Selenium doesn't want to work with scLocator.
=> Is there a simple way to accept scLocator with Selenium ?
(And when I say simple, it must be simple... I'm not a specialist in Java...)
Could someone help me to unit test the application ? It's coded in Java, it's huge and I have to cover ~70% of the code (25k lines of code)
Some more specs :
Maven is used to compile.
I'm not touching at the server side.
Tests must be faster than GwtTestCase and Suite :/
I hope my problem is clear, I'm not a native english so sorry for mistakes I may do :x
We provide Selenium extensions and a user guide for them in the SDK, under the "selenium" directory at top level.
If you download 3.1d (from smartclient.com/builds) there's even more documentation including some samples for JUnit.
Do not use ensureDebugId() (won't have an effect at all). Never try to work with DOM IDs (won't work).
Best practices information in the Selenium user guide explains when you should use setID().
I found that it could be solved by using setID() before the
initialization of the widget. But that's the ID of the scLocator and
not an HTML ID.
Why don't you try:
widget.ensureDebugId("some-id");
From the Java docs for ensureDebugId():
Ensure that the main Element for this UIObject has an ID property set,
which allows it to integrate with third-party libraries and test tools
< defaultUserExtensionsEnabled>true< /defaultUserExtensionsEnabled>
< userExtensions>[path to user-extensions.js]< /userExtensions>
There we go. I managed to make it work. (With the selenium-maven-plugin in the < configuration> tag)
Thanks for your help though.
I have been writing a lot of unit tests for the code I write. I've just started to work on a web project and I have read that WatiN is a good test framework for the web.
However, I'm not exactly sure what I should be testing. Since most of the web pages I'm working on are dynamic user generated reports, do I just check to see if a specific phrase is on the page?
Besides just checking if text exists on a page, what else should I be testing for?
First think of what business cases you’re trying to validate. Ashley’s thoughts are a good starting point.
You mentioned most pages are dynamically generated user reports. I’ve done testing around these sorts of things and always start by figuring out what sort of baseline dataset I need to create and load. That helps me ensure I can get exactly the appropriate set of records back in the reports that I expect if everything's working properly. From there I’ll write automation tests to check I get the right number of records, the right starting and ending records, records containing the right data, etc.
If the reports are dynamic then I’ll also check whether filtering works properly, that sorting behaves as expected, etc.
Something to keep in mind is to keep a close eye on the value of these tests. It may be that simply automating a few tests around main business use cases may be good enough for you. Handle the rest manually via exploratory testing.
You essentially want to be testing as if you are a user entering your site for the first time. You want to make sure that every aspect of your page is running exaclty the way you want it to. For example, if there is a signup/login screen, automate those to ensure that they are both working properly. Automate the navigation of various pages, using Assertions just to ensure the page loaded. If there are generated reports, automate all generations and check the text on the generations to ensure it is what you specified by the "user" (you). If you have any logic saying for example when you check this box all other boxes should check aswell. There are many assertions that can be applied, I am not sure what Unit-Testing software you are using but most have a very rich assortment.
After learning about TDD and unit testing, I'm really looking to write tests when I code, preferably before I code because I see the benefits coding in Python. I'm developing a website, and I'm trying to write tests according to the requirements, but its proving more difficult than expected.
I can see the benefits of writing tests when you're producing a library of code with a public interface for others to use. Developing a website, where there is really not much logic, and mostly Reading and Writing against a database seems a little harder to unit test. Mostly, I have to create/edit/delete rows in the database.
I'm using a framework (Kohana 3 for php), so 99% of all the libraries and helpers that I'm going to be using have already been tested (hopefully), so what else is their to write tests for?
I'm mostly talking about scripting languages, not about CSS or HTML, I already know about cross-browser testing.
How much can you really test when developing a web site, and how should you go about it?
Edit: Is the lack of activity on this question a sign? I understand that certain things MUST be tested, like security and the like, but how much can be written using unit tests and TDD is the question.
Thanks.
Developing a website, where there is really not much logic, and mostly Reading and Writing against a database seems a little harder to unit test. Mostly, I have to create/edit/delete rows in the database.
Not completely true.
You have data model processing. Does the validation work? Do the calculations on the reported rows from the database work?
You have control, sequence and navigation among pages -- do the links work? The test setup will provide a logged-in-user. The test will (1) do a GET or a POST to fetch a page, then (2) confirm the page actually loaded and has the right stuff.
You have authorization -- who can do what? Each distinct test setup will provide a different logged-in-user. The tests will (1) attempt a GET or POST to process a page. Some tests will (2) confirm they got directed to error-response pages. Some tests will (2) confitrm that the GET or POST worked.
You have content on the page -- what data was fetched? The test setup will provide a logged-in-user. The test will (1) do a GET or a POST to fetch a page, then (2) confirm the page actually loaded and has the right stuff.
Have you tried Selenium? It allows you to automatically do almost anything in a web browser. For example, you could have it go through and click all of the links and make sure that they go to the correct url.
It works with multiple languages, including python and allows for testing in chrome, firefox, ie, and other browsers.
If your site contains many forms, how do you write them? Do you write each view using plain HTML? Or do you write your own form helpers that generate forms just the way you want them? If you do that, you may find that unit-testing your form generators makes it easier to write them.
In general, if your program is mostly CRUD, look out for ways to automate CRUD management; write your own custom CRUD generator. Which does not mean write the CRUD framework that will end all frameworks; that would be too much work. Just write a generator for the small things you need for your current application. TDD will help you there.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
The company I'm currently working for is using Selenium for Uniting-Testing our User Interface. What do you use to Unit-Test your Web UI and how effective do you find it?
I'm a huge fan of Selenium. Saying 'unit-testing your web ui' isn't exactly accurate as some of the comments have mentioned. However, I do find Selenium to be incredibly useful for performing those sort of acceptance and sanity tests on the UI.
A good way to get started is using Selenium IDE as part of your development. Ie, just have the IDE open as you're developing and write your test as you go to cut down on your dev time. (Instead of having to manually go through the UI to get to the point where you can test whatever you're working on, just hit a button and Selenium IDE will take care of that for you. It's a terrific time-saver!)
Most of my major use case scenarios have Selenium RC tests to back them up. You can't really think of them as unit-tests along the lines of an xUnit framework, but they are tests targetted to very specific functionality. They're quick to write (especially if you implement common methods for things like logging in or setting up your test cases), quick to run, and provide a very tight feedback loop. In those senses Selenium RC tests are very similar to unit-tests.
I think, like anything else, if you put the effort into properly learning a test tool (eg, Selenium), your effort will pay off in spades. You mention that your company already uses Selenium to do UI testing. This is great. Work with it. If you find Selenium hard to use, or confusing, stick with it. The learning curve really isn't all that steep once you learn the API a little bit.
If I'm working on a web app, its rare for me to write a significant amount of code without Selenium RC tests to back it up. That's how effective I find Selenium. :) (Hopefully that'll answer your question..)
We use Watin at my place of employment, we are a .net shop so this solution made a lot of sense. We actually started with Watir (the original ruby implementation) and switched after. It's been a pretty good solution for us so far
Well, if you've designed your application properly, you won't have scads of logic inside the UI anyway. It makes much more sense to separate the actual work getting done into units separate from the UI, and then test those.
If you do that, then the only code in the UI will be code that invokes the backend, so simply testing the backend is sufficient.
I have used NUnit ASP in the past (at my job), and if you insist on unit testing your UI, I would strongly advise you to use ANYTHING but NUnit ASP. It's a pain to work with, and tests tend to be invalidated (needing to be revised) after even the most minor UI changes (even if the subjects of the tests don't actually change).
We are using QuickTestPro. So far it is effective, but the browser selection is limited. The nicest part is the ability to record your browser's activity, and convert it into a scriptable set of steps. There is also a nice .Net addin so if you have any validation code you need to do for the different stages of your test, you can write methods in an assembly and call them from your script.
We use Visual Studio 2008 Tester Edition.
Pros:
Very good at capturing user interaction
Captures Ajax calls
It is very easy to map user input to a database, XML or CSV file
The captured test can be converted to C# for more control
The same tests can be used for load testing and code coverage
Cons:
VS2008 Tester Edition is a seperate SKU from the normal Developer Edition, which means extra cost
You may be alergic to Microsoft ;-)
We have used it very effectively on projects, however there a lot of effort involved in keeping tests up to date, every time you change a screen the test may need to be re-recorded
We tend to keep the tests short and sharp, do one thing and get out instead of recording 10 minutes worth of clicking around in a single test.
We have a few standard UI test types:
Menu Test: Log in as a specific user (or user type/role) and make sure all the required menu items are available
Validation Test: Open a page and click save without entering any data, ensure that all the validation warnings appear. Complete required fields one at a time and check that the warning messages disappear when they are supposed to.
Search Test: Search using data from your database or a data file and ensure the correct data is returned by the search
Data Entry Test: Create new recrords from a data file, cleanup the database to allow tests to run multiple times
UI Testing is quite time consuming but the comfort feeling you get when a few hundred tests pass before you release a new version is priceless.
We have been using JSunit for a while to do unit tests... it may not be the same kinds of tests you are talking about, but it is great for ensuring your JavaScript works as you expect.
You run it in the browser, and it can be set in an Ant build to be automatically run against a bunch of browsers on a bunch of platforms remotely (so you can ensure your code is cross-browser as well as ensure the logic is correct).
I don't think it replaces Selenium, but it complements it well.
We use Selenium Core, but are switching gradually to Selenium RC which is much nicer and easier to manage. We have written lots of custom code to make the tests run on our Continuous Integration servers, some of them in parallel suites to run faster.
One thing you'll find is that Selenium seems to restart the browser for each test (you can set it not to do this, but we got memory problems when we did that). This can be slow in Firefox, but is not too bad in IE (one time I'm thankful for Bill Gates's OS integraion).
I've used WATIR, which is pretty good. I liked it because it's Ruby and allows for testing interactivity, available elements and source code parsing. I haven't used it for a while but I assume it's gotten better.
It's supposedly being ported to Firefox and Safari, but that's been happening for a while now.
Check out Canoo Web Test. It is open source and built on the ANT framework.
I spent some time working with it for a graduate course on Software QA and it seems to be a pretty powerful testing tool.
Selenium Grid can run your web tests across multiple machines in parallel, which can speed up the web testing process
I mostly use CubicTest, which is an eclipse plugin that lets you define tests graphically. It can export/run tests through several libraries, including watir and selenium. Most people just use the Selenium runner though.
Full disclosure: I'm one of the developers, so I'm kind of biased :)
Take a closer look here: cubictest.openqa.org
-Erlend
Selenium is for Integration testing, not Unit testing. It's a subtle, but important difference. The usage I usually see is for sanity checking a build. i.e., have a test that logs in, a test that (for example) submits a story, makes a comment, etc.
The idea is that you're testing to see if the whole system is working together before deployment, rather than have a user discover that your site is broken.
We currently use Silk4J - a Java centric approach to testing Web UI. It can test Flash, Flex, AIR, Silver Light, Win32, HTML, and a few other applications.
Since Silk4J can control Win32 apps it can control browser dialogs directly, which is a step above what Selenium can control and is especially useful for download prompts.
We use WatiN for system testing, and QUnit for JavaScript unit testing.
Molybdenum is built over Selenium and has some additional features.