We're looking at creating Selenium tests to help automate some of our testing.
Is there a tool/service/etc. for helping to manage those tests. Specifically, is there any kind of tool that helps keep them organized and documented?
If there isn't anything automated, are there any resources out there to help with this?
Ideally we'd like our testers to do the bulk of the work when it comes to these tests. Most of our testers have no programming background, so anything too intensive would be handled by the developers.
Thanks.
Generally for setting up automation framework a good rule of thumb is to use what your developers are using for unit and integration testing. Following the same model eliminates the need to maintain/learn multiple tools, plus developers will be happier helping with those tests if they are in their normal environment.
To begin with, you will need 4 components:
Source control (SVN, Git, etc.) for storing the source code of the tests. Some companies may not want the test code to be in the same repository as production code, so they may setup a separate server.
A project (or projects), that contain the test code. Starting from the typical bare-bone development project, adding selenium libraries/dependencies to it will give a good starting point.
Then you need an IDE to develop the tests. Again, ask developers what do they use.
And finally you need something that executes those tests on schedule or on single button click. This is where development's CI product (e.g. Jenkins, Bamboo or whatever they are using) can be useful. This tool will not only run tests for you, but will also give a fairly well-organized results report and history. And it can deploy, say, latest build, or reset test environment.
But to achieve all this, I think you need to accept that testers will deal with writing and debugging the code. And test code has to be of good quality, so you would have to hire or develop 1-2 good automation developers in your team. Or have some developers committed to creation of the tests, which testers will only draft with Selenium IDE.
Related
So after much hunting I failed to find a continuous testing tool for IntelliJ 14.
I stumbled across a post that references uses eclipse and Ant in order to simulate this. On save, Ant then runs the tests for any tests that were modified.
I've tried to replicate this but, alas! I've never used Ant before and am finding it extremely difficult. I've setup and configured a generic Ant build file in Intellij but simply cannot figure out how to achieve my task.
Any help, pointers in the right direction is very much appreciated. I've searched but only found information that needs to be decrypted first.
Eclipse has the builder feature, you create an AntBuilder for your project, see also https://stackoverflow.com/a/15075732/130683.
IntelliJ has a trigger feature that might serve the purpose.
Also Infinitest , which provides a Continous Testing Plugin for Eclipse and IntelliJ might be helpful.
Ant is a build tool. Although IntelliJ does that for you, you need IntelliJ to do this which means you can't distribute your application without IntelliJ.
Ant uses a dependency matrix for building. This is sometimes difficult for developers to understand, but it basically means that you define the steps, how the steps are dependent upon each other, and let the build tool figure out exactly how to do its job. Ant is for Java like Make is to C and C++ applications.
Ant uses targets which are the steps you specify to do. For example, you might have a target called package that will build your jar or war. That target might depend upon another target called compile to compile the code. That target might depend upon a code generation phases (like if you had WSDL files).
Each target is a set of tasks. For example, the compile target is likely to have the <javac> task in it. It might also need the <mkdir> task to create the work directories where you classfiles are stored.
There are plenty of books on Ant, and there's a tutorial on the Ant Website. You didn't explain the issues you were having, so it's hard to be more specific than this.
Ant can also run your unit tests too. There's a <junit> target which can run the tests, and you specify whether or not you want to run almost all of your <junit> tests via the <batchtest> sub-entity or if you have a program driver you specify via the <test> entity.
Once you get an Ant script that can build and run your tests outside of IntelliJ, you can now get a Continuous Integration tool like Jenkins. A continuous integration tool watches your repository for changes, and if a change occurs, will then build your application. It's a great way to catch errors early on.
What does this have to do with Continuous Testing? Well, if you have your Ant script able to run unit tests, the Continuous Integration engine not only can build your app, but then run the unit tests with each and every change that occurs.
Jenkins is nice because it's very simple to use. You download a jenkins.war and you can launch the Jenkins webpage via the java -jar jenkins.war command. This brings up a web server on port 8080 on your machine. Obviously, Jenkins can be configured to run on different ports and under Tomcat if you so desire. It can integrate with Windows Active Directory, LDAP, and many other user verification systems.
Jenkins will show you charts and graphs of your tests, let you know which tests failed or passed, and will notify you of any problems via email, tweets, IM, Jabber, and even Facebook posts. People have even setup a traffic light in their offices that turns red when builds or tests fail.
Take it one step at a time. Get a good book on Ant. Read the tutorial on the Ant website. Then try to get a working Ant script to just to build your app. If you are having specific issues, you can ask for help.
Once you have the build going, extend the script to run your unit tests. Once that is done, download Jenkins and try to get that up and running.
This may seem like a very broad question, but i am really interested to know about possible approaches. Our team has a Django Web app and we have huge amount of unit tests for our features. Now in github, we have master branch, develop branch, and individual feature/bug branches. Now the problem i want to solve is,
Every time some code is merged into develop branch, i want to run all(or subset) of unit tests against that branch. It would be cool to have it automated, i-e i do not have to trigger the test run.
I have read and heard about Jenkins - http://michal.karzynski.pl/blog/2014/04/19/continuous-integration-server-for-django-using-jenkins/. Currently one of the approaches i am leaning towards.
But i wanted to know if there are better approaches or tools which i can use.
Appreciate all your help.
For what it's worth, you can't really go wrong with Jenkins for the functionality you are looking to achieve.
Although Travis CI may be a better option given that it's meant to work seamlessly with Github and it appears all of your repositories have been moved to Github.
Really depends on your business needs though.
Getting Jenkins up and running, from past experiences, has always gone very smoothly and it gives you the benefit of keeping all data in house as you have the option to host Jenkins on your own private servers but probably doesn't scale or run as efficiently as Travis CI does depending on your setup.
Travis CI will probably allow for an even more seamless approach because it's already being hosted for you and tied directly into Github, but you won't get the privacy as running Jenkins on your own servers. There is a paid option though it appears for Travis CI which again, depending on your business needs, may be a better option.
We're looking into automating our web UI using either Microsoft's coded UI tests or Telerik TestStudio framework. I would love to run our tests using a service like Sauce Labs however, Sauce Labs requires Selenium tests. Is anyone aware of some service similar to Sauce Labs to run web tests other than Selenium?
TFS in the cloud looks promising http://tfspreview.com/
Regardless of TFS service or Sauce Lab, be aware that a service only frees you the burden of maintaining a CI/Test server in house. It doesn't magically generate tests for you. It's still your responsibility to design and code many tests. That is where huge effort and learning curve will take place.
(Speaking of learning curve, Selenium is much more manageable than Ms coded ui test. Wait for chances to influence company decision...)
Sauce Labs in fact is able to run other (Non-Selenium & Appium) frameworks. There is a testrunner that is super easy to use called saucectl. You can test with CYpress, Test Cafe, Espresso, XCUITest and more. You can see the info in the docs.
TLDR
Install saucectl in your project with cURL or npm install -g saucectl
Setup the runner in your test project with saucectl init. choose your framework flavor and preferences
Run your tests with the
command saucectl run
TFS is not an alternative to Sauce. In fact their is now an integration to run Sauce test from Visual Studio Online formerly TFS. Sauce is a testing grid. The closest competitive solution is BrowserStack. But they are not as widely adopted. But it also is selenium based. Telerik TestStudio is nice, but it also is not the cloud you run on. Maintaining your own grid is a huge part of the value of a cloud-based solution. And selenium really is the best and most supported language for doing it.
Development teams are often plagued by builds in version control being transiently broken. The entire team's productivity can come to a halt while trying recover from a build broken by one person.
Is there software that would allow hosting Git in a way that prevents breaking builds in version control by not accepting commits that fail to pass tests in the first place? The usage scenario could for example look as follows:
The software runs on a server that continuously pulls revisions from Git repositories that developers have published.
For each pulled revision, the software builds the revision and tests if it passes the unit tests.
If it passes the tests, the revision is merged into the "stable" branch.
If it doesn't pass the tests, it is rejected and the revision is not merged into the "stable" branch. The developer is forced to correct the revision and resubmit it.
Developers by default pull from the "stable" branch that should never be broken—in the sense that tests do not fail—and are more productive as they spend less time being blocked by broken builds. And the usefulness of such a system grows with the team size.
A few notes:
Git's pre-commit hooks and similar are not satisfactory in this case. The solution should be automatic and enforced on the server side for each commit.
Looking for a solution that has been implemented and thought out as far as possible, instead of writing a system like this myself from scratch.
I think this is more a build server feature that ties into a VCS such as Git. TeamCity does have support for this, but I've not tried it so I can't comment on how good it actually is.
http://www.jetbrains.com/teamcity/features/delayed_commit.html
The Hudson guys have been discussing it for a while, but I've yet to see it in a release.
http://wiki.hudson-ci.org/display/HUDSON/Designing+pre-tested+commit
We use gerrit and hudson. It is what android and Cyanogenmod use as well (along with many others).
Gerrit allows for code review and automatic building of every commit with automatic rejection of those that fail tests.
Hudson runs the tests.
Hudson: http://wiki.hudson-ci.org/display/HUDSON/Designing+pre-tested+commit
Gerrit: http://gerrit.googlecode.com
This system works well with the repo tool to have a large number of small repositories, this will reduce merge conflicts which have to be handled manually via a rebase.
Note: it is quite a bit of work to get up and running if you have a large existing code base, but totally worth it.
It's a really good idea. Bamboo supports this quite naturally. Several Bamboo customers as well as teams at Atlassian have this exact methodology setup and working to great effect. Bamboo has event listeners which can tell (without polling) when a commit is pushed to a 'test verfication' repo and then verify it by running the tests before pushing to the stable branch. www.atlassian.com/bamboo
Joel seems to think highly of daily builds. For a traditional compiled application I can certainly see his justification, but how does this parallel over to web development -- or does it not?
A bit about the project I'm asking for --
There are 2 developers working on a Django (Python) web app. We have 1 svn repository. Each developer maintains a checkout and thier own copy of MySQL running locally (if you're unfamiliar with Django, it comes bundled with it's own test server, much the way ASP apps can run inside of Visual Studio). Development and testing are done locally, then committed back to the repository. The actual working copy of the website is an SVN checkout (I know about SVN export and it takes too long). The closest we have to a 'build' is a batch file that runs an SVN update on the working copy, does the django bits ('manage.py syncdb'), updates the search engine cache (solr), then restarts apache.
I guess what I don't see is the parallel to web apps.
Are you doing a source controlled web app with 'nightly builds' -- if so, what does that look like?
You can easily run all of your Django unit tests through the Django testing framework as your nightly build.
That's what we do.
We also have some ordinary unit tests that don't leverage Django features, and we run those, also.
Even though Python (and Django) don't require the kind of nightly compile/link/unit test that compiled languages do, you still benefit from the daily discipline of "Don't Break The Build". And a daily cycle of unit testing everything you own is a good thing.
We're in the throes of looking at Python 2.6 (which works perfectly for us) and running our unit tests with the -3 option to see which deprecated features we're using. Having a full suite of unit tests assures us that a change for Python 3 compatibility won't break the build. And running them nightly means that we have to be sure we're refactoring correctly.
Continuous integration is useful if you have the right processes around it. TeamCity from JetBrains is a great starting point if you want to build familiarity:
http://www.jetbrains.com/teamcity/index.html
There's a great article that relates directly to Django here:
http://www.ajaxline.com/continuous-integration-in-django-project
Hope this gets you started.
Web applications built in dynamic languages may not require a "compilation" step, but there can still be a number of "build" steps involved in getting the app to run. Your build scripts might install or upgrade dependencies, perform database migrations, and then run the test suite to insure that the code is "clean" w.r.t. the actual checked-in version in the repository. Or, you might deploy a copy of the code to a test server, then run a set of Selenium integration tests against the new version to insure that core site functionality still works.
It may help to do some reading on the topic of Continuous Integration, which is a very useful practice for webapp dev teams. The more fast-paced and agile your development process, the more you need regular input from automated testing and quality metrics to make sure you fail fast and loud on any broken version of the code.
If it's really just you and one other developer working on it, nightly builds are probably not going to give you much.
I would say that the web app equivalent of nightly builds would be staging sites (which can be built nightly).
Where nightly builds to a staging area start paying real dividends is when you have clients, project managers, and QA people that need to be able to see an up to date, but relatively stable version of the app. Your developer sandboxes (if you're like me, at least) probably spend a lot of time in an unusable state as you're breaking things trying to get the next feature implemented. So the typical problem is that a QA person wants to verify that a bug is fixed, or a PM wants to check that some planned feature was implemented correctly, or a client wants to see that you've made progress on the issue that they care about. If they only have access to developer sandboxes, there's a good chance that when they get around to looking at it, either the sandbox version isn't running (since it means ./manage.py runserver is up in a terminal somewhere) or it's in a broken state because of something else. That really slows down the whole team and wastes a lot of time.
It sounds like you don't have a staging setup since you just automatically update the production version. That could be fine if you're way more careful and disciplined than I (and I think most developers) am and never commit anything that isn't totally bulletproof. Personally, I'd rather make sure that my work has made it through at least some cursory QA by someone other than me before it hits production.
So, in conclusion, the setup where I work:
each developer runs their own sandbox locally (same as you do it)
there's a "common" staging sandbox on a dev server that gets updated nightly from a cronjob. PMs, clients, and QA go there. They are never given direct access to developer sandboxes.
There's an automated (though manually initiated) deployment to production. A developer or the PM can "push" to production when we feel things have been sufficiently QA'd and are stable and safe.
I'd say the only downside (besides a bit of extra overhead setting up the nightly staging builds) is that it makes for a day of turnaround on bug verification. ie, QA reports a bug in the software (based on looking at that day's nightly build), developer fixes bug and commits, then QA must wait until the next day's build to check that the bug is actually fixed. It's usually not that much of a problem since everyone has enough stuff going on that it doesn't affect the schedule. When a milestone is approaching though and we're in a feature-frozen, bugfix only mode, we'll do more frequent manual updates of the staging site.
I've had great success using Hudson for continuous integration. Details on using Hudson with Python by Redsolo.
A few months ago, several articles espousing continuous deployment caused quite a stir online. IMVU has details on how they deploy up to 5 times a day.
The whole idea behind frequent builds (nightly or more frequent like in continuous integration) is to get immediate feedback in order to reduce the elapsed time between the introduction of a problem and its detection. So, building frequently is useful only if you are able to generate some feedback through compilation, (ideally automated) testing, quality checks, etc. Without feedback, there is no real point.