I just started to develop (from scratch) a new project in Erlang. I use Erlide, a plugin for Eclipse. I wrote some unit tests (using EUnit) but I find running each of them (one by one) by hand quite annoying. Is there any way to automate this process, so that - with just a couple of clicks - all of my tests are run?
Thanks in advance for your help.
To run all test in a directory you can use the primitive:
{dir, Path::string()}
This tests all object files in the specified directory, as if they had been individually specified using {file, FileName}.
You can also use the fileand application primitive and others to define sets of tests.
You'll find documentation to all these under 1.5 EUnit test representation
Related
I have been developing an application in my free time using Qt.
As the size of code is increasing I am finding it difficult to contain new bugs for older code. I have been testing my application manually.
Since the target is an exe I cannot test it automated with C++ tests without injecting some extra code into my application.
So my question is, what is the best QA technique for a GUI application if you are a single developer & wont be earning money from the project as it will be released for free?
Thank You.
EDIT:
I would like to have a set of simple tests, each testing for specific functionalities of my software. I would like them to run automatically one after another. Finally they should create a report of which tests failed. This can possibly be done by creating new functions in the same classes + adding some checks in existing functions I want to test & then create a new class which will have all the tests. So I wanted to know whether is this the best way or is there a better alternative? Because everytime I will build a release target, I will be commenting/deleting this QA code, which may create some bugs for that build.
Currently I am not worried about documentation & comments as I have maintained that from the beginning. It is only about source code QA.
Unit tests by-the-book will only give you assurance for your methods, not for the entire application. But you can also use the same unit-test framework to write acceptance tests for specific capabilities of the application.
The easiest way to go would be to extract the GUI from the application, and to make the GUI dependent of an API/library. The API will make it easy to write functional tests. Be sure to make the GUI as thin as possible.
I wouldn't add test code to your class and remove it to release, I think this is as risky as shipping with the test code. You're better off have separated source as already advised here.
If your project is getting large enough, you'll probably want to create some unit tests for it (I like the free CppUnit library, which is similar to JUnit; also Jo Are By suggested QtTest, which presumably is available with Qt).
Even if you have to make some changes to your production code, it will be worth your time in the end.
You may also wish to look into automated GUI testing frameworks for Qt applications; I'm not familiar with any of these.
Test code go to its own source file.
You may split your exe into library and one main.cpp which simply call your library.
That way, you may use any unitTest Framework with extra test files to generate a executable which only tests your library.
For code testing you will use Junit testcase
You may split your exe into library and one main.cpp which simply call your library.
For GUI testing you have do it Manually because there is not Tool Available to Test GUI interface of any application.
In Manual testing GUI is check complete and GUI image or text is not displayed clearly or text is missing is not all this is will not be test by automation.
I'm a part of a big (sort of...) C++ application written mainly in Qt.
I was wondering if this is the right / common approach:
whenever I make a change to a certain / several source files, I compile it (QtCreator) in debug mode, and then launch and test it.
the problem is, each compilation takes a couple of minutes (usually a 1 - 3 minutes), I hate that, and I guess I'm doing something wrong here, maybe compiling the whole project for each minor change is not the right path to go?
thanks,
Try to use unit test with QTest as much as possible, then you can verify the parts first and then top it of with some test on the complete application. This saves a lot of time and can also help to produce more robust code if done right.
This do need some kind of modular approach, so the code needs to be grouped in some way.
Most likely you need to tweak your build settings to ensure you are doing minimal rebuild or incremental build where it only compiles the files that have changed and doesn't update or rebuild anything not directly affected by the changes. This still doesn't help when you are changing a header file that's included heavily throughout the project but with a well laid out project that shouldn't happen.
In general there are several approaches to testing as you go but here are the main two things I'd recommend:
Don't rebuild the entire project (no clean, no rebuid all) just do an incremental build and test as you go. Great for testing gui changes and little things in projects that don't need to link against a million things or have a long start up.
Develop it as separate project with a console app or simple test application which you don't include in the final integrated version but can keep for independent testing later. This is better for libraries like say you are making a new encryption algorithm or file manager to replace some old archaic portion of a bigger project.
Of course there is always the approach of coding with overwhelming confidence like a crazy person and crossing your fingers when you compile and run it which is very popular but not quite as effective.
Do you have multiple SUBDIRS targets within your project ? If the answer is yes, You could try tweaking the project file by first removing all "ordered" keywords from project files and then if one subdir is depending on another, declare those as dependencies. And finally, make sure you pass -jX value to make (that is, if your build rules use make) so that all cpu cores are taken into use while compiling.
Also answered in: Qt automated testing
I along with my team recently developed TUG, an open-source framework for Qt GUIs Unit Testing. Is uses Qt Test. Maybe it can help you.
A video better than a thousand words:
https://www.youtube.com/watch?v=tUis6JrycrA
Hope we can make it better together. Github repo: http://pedromateo.github.io/tug_qt_unit_testing_fw/
I'm just getting started with a project that combines GWT, Google App Engine and the Google Eclipse plugin. Where is the best place to store my tests? I normally keep my code organized Maven-style, with src/main/java, and tests in src/test/java. The default setup I get from the plugin dumped my source directly into src, which I'm not too fond of, but I'd prefer not to fight against the tools. What's the "standard" place to put unit tests in such a project?
Solution:
create src/main/java, move the existing code under there
create src/test/java, add your tests here
go to Project -> Properties -> Java Build Path, add the new locations as Source Folders.
I've faced a kind of problem woth GAE testing: Some tests require an appengine-testing.jar wich conflicts with the main appengine-api-xxx.jar of the poject. That way, I was able to run tests for GAE but it conflicted with a normal run/debug launch. To be able to run the app in my local machine, I had to remove the appengine-testing.jar and then, a lot of compilation errors appeared in my test/ clases.
If you want an advice, set your test clases in another project (where you can use the jars without conflict)
Otherwise, if you got make it work, please, tell me how did you do.
Thanks a lot.
Put it where it pains you least.
GWT on Google App Engine is pretty new at this point; you are
optimistic to expect there is a "standard" place, especially since
you've already found an inconsistency in what the tools do.
Since you've already accepted the source starting at "src/", why not
put the test source in "test/"? This is certainly standard in many
contexts.
I think everyone here would agree that in order to be considered a professional software house there are number fundamental things you must have in place.
There is no doubt that one of these things is a build server, the question is, how far do you need to go.
What are the minimum requirements for the build server? (Somewhere to just compile?)
What is the ultimate goal for your build server? (Scheduled, source control integration, auto deployment to test / live servers)
Where is a good place to start assuming you have nothing at the moment?
It would be great if we could list out a few simple tasks that an amateur developer could take on board in order to set them on the right track to a fully functional build server.
It would also be good to hear about people that feel they have a "complete" system setup that performs all the functionality they require and how they went about setting it all up from scratch.
You can start by looking into Cruise Control.
There's also CruiseControl.net if that's your poison.
Essentially though, you need the following ingredients:
A dedicated environment (Virtual Machine/server. Don't use a developer's machine, unless it's just you. Even then, run a VM if you can. Much easier to move it to a server when/if one becomes available in your organisation)
A source control system that supports labelled/tagged revisions (for example, Subversion+TortoiseSVN)
Build scripts. These can be batchfiles that start the devenv.exe or msbuild.exe applications with a command line, or you can use something like Ant or NAnt.
In this scenario, CruiseControl acts as the Continous Integration server, and can make sure that you have builds done as you check in your code. This means you know whether the build is broken quicker than if you just had nightly builds. You should probably also have nightly builds, though.
Hudson is a great CI.
We run farm locally, but we started by downloading hudson.war and doing
java -jar hudson.war
It integrates with SCM, bug trucking systems it is really awesome.
You'll need some disk space if you want to keep old build.
Enjoy it is most straightforward CI solution so far.
HTH,
Hubert.
If you're using Cruise Control, the place to start is an Ant build.xml that does the job manually.
You need a version control system that can do labeled check-outs.
You need JUnit tests to run using the Ant task and generate HTML reports.
Id say you'd have to start by implementing a build strategy so you can build your code in a structured way - I use NANT.
For a basic build server - use one of the CI offerings out there that monitors your source control and triggers a build whenever a change is detected. eg: cruiseControl.
Once you get the basic build together - add the running of your unit tests after a successfuly build.
The most successful system i've had in place had 3 different builds :-
- one that fired on a check in - all this did was build the code.
- an on demand one that would build the application, generate the installer and then put
the installer into a shared drive for the testers to pick up
- a daily build that fired at 10pm. This:
- ran some code generation to build DB and C# code from a UML model
- build the code
- created a new build verification test user on a test oracle instance
- ran the application schema into the db
- fired off a bunch of unit tests
- cleaned up the db user (if the tests were successful)
- ran coverage analysis to build a report of the unit code coverage
Software we used for this was NANT, CruiseControl.NET, a custom code generation system, custom app to build an oracle schema, and NCover for the code analysis.
Start by having a read of Martin Fowler's excellent paper on Continuous Integration.
We built such a system for a major project >2,000 kSLOC and it proved itself to be invaluable.
HTH
cheers,
Rob
Cruise, Maven, Hudson etc are all great but its always worth having a stopgap solution.
You should have a batch file, shell script or simply written instructions that will allow you to run a build from any machine. We have had build servers unavailable in the past and the ability to switch quickly to another machine was invaluable!
The spec of the build machine need not be important unless you have a monster project. We try and keep our build times down to 10 minutes (including unit tests) and we have a pretty big project.
Don't be tempted to create or write your own build system because "none of the tools out there are good enough". All modern build systems allow you to write plugins to do custom stuff.
I'm using Cruisecontrol.NET and an msbuild buildscript.
I can use the buildscript manually so that I can get the latest version of the codebase, built the codebase very easily using the commandline. (This is very interesting if you are working on an application that consists of multiple solutions).
Next to that, my CruiseControl.NET buildserver uses this buildscript as well. It checks on a regular interval if there have been changes committed to the source-control.
If that happens, CC.NET performs the 'get-latest' task that I've defined in the buildscript, builds everything, executes unit-tests and performs a statical code analysis (fxcop).
My 'buildserver' is just an old workstation. It's a PIV, 3Ghz with 1gb RAM, and it does its job perfectly.
One additional thing that I would find interesting, is to have the ability to automatically deploy a new version, or build a setup.
I haven't done that yet, since I'm not sure whether it is a good idea, nor have I found a good strategy yet to do so ...
I mean; is deploying a new version of some components into production for a mission-critical application a good idea ? I don't think so ...
I think this is a good place to start:
[http://confluence.public.thoughtworks.org/display/CC/Home;jsessionid=5201DA7E8D361EB164C40E519DA0F0DE][1]
At least, that's where I started looking when setting up my build server. :)
[1]: Home of CruiseControl
Roughly in order - minimal/least sophisticated through more sophisticated
able to get a specific set of source onto any machine
able to build that source (with no problems)
able to (schedule) build each night/or some other defined period with no user intervention
One (or more) dedicated build server (not shared as qa or dev machine)
able to do a build after each check-in/commit
Notify interested parties of the build status after a build
Provide build status at any time
Create installers as part of the build
ability to deploy/live if build is good
Run unit tests
Run tests on the product
Report the results of those tests
Static code analysis and reporting
...
And the list goes on and on
Don't be afraid to just start with batch files or shell scripts or other ad-hoc means. People made perfectly good software before the CI craze. there were plenty of good processes before Hudson and Cruise Control - ( I am not knocking those or others - I use Hudson among others) - but don't miss the point - these things are here to help you - not become overbearing process)
I couldn't give you all the details about how we set our build server up (I was only involved at the start), but:
We started with an in-house system, implemented in ASP.NET and a .NET Windows Service, using NAnt to do the actual builds. Actually, most of the workflow was implemented in NAnt (e.g. emailing people, copying stuff around, etc.).
We moved to JetBrains TeamCity (there's a free cut-down version available), which is still serving us well.
We use it for builds triggered by a commit: these just build the binaries and run the unit tests. From here, we can do a complete build, which does the MSI as well. From there, we have system test builds that run more in-depth tests, across an environment built with virtual machines (with a separate domain controller, SQL Server box, etc.). When the system tests pass, the build is made available to our QA department for manual testing and some regression tests that we've not automated yet.
In the java space I've tested most of the available build environments. The issue with automatic build is that you quite often end up spending a fair amount of time following it up. After we switched to the commercial bamboo from atlassian, we found that we have to spend a lot less time pampering the build box, which in our case turns out to be very good economy. Bamboo also supports clustering, so you can add inexpensive boxes as needs evolves.
Try & find something that fits in with your existing practices in terms of building - e.g. it's not going to be a good fit to try & use an Ant-based buildserver if you're using Maven, for instance!
Ideally, it should just be able to monitor your source-control system, checkout the code, build, run some tests & publish the results without you being aware of it, or at least not 'till it's reporting a failure. Personally, I'd suggest Hudson (https://hudson.dev.java.net/) as a good starting point as it's easy to get installed & running & has a decent UI.
We start by writing batch scripts that will run on the developers machine. Once we have all the processes automated, we move them to the build server.
On the tools side we are currently moving from Cruise Control to TFS.
I mainly develop in native C++ on Windows using Visual Studio.
A lot of times, I find myself creating a new function/class or whatever, and I just want to test that piece of logic I just wrote, quickly.
A lot of times, I have to run the entire application, which sometimes could take a while since there are many connected parts.
Is there some sort of tool that will allow me to test that new piece of code quickly without having to run the whole application?
i.e.
Say I have a project with about 1000 files, and I'm adding a new class called Adder. Adder has a method Add( int, int );
I just want the IDE/tool to allow me to test just the Adder class (without me having to create a new project and write a dummy main.cpp) by allowing me to specify the value of the inputs going into Adder object. Likewise, it would be nice if it would allow me to specify the expected output from the tested object.
What would be even cooler is if the IDE/tool would then "record" these sets of inputs/expected output, and automatically create unit tester class based on them. If I added more input/output sets, it would keep building a history of input/outputs.
Or how about this: what if I started the actual application, feed some real data to it, and have the IDE/tool capture the complete inputs going into the unit being tested. That way, I can quickly restart my testing if I found some bugs in my program or I want to change its interface a bit. I think this feature would be so neat, and can help developer quickly test / modify their code.
Am I talking about mock object / unit testing that already exists?
Sidenote: it would be cool if Visual Studio debugger has a "replay" technology where user can step back to find what went wrong. Such debugger already exists here: http://www.totalviewtech.com/
It's very easy to get started with static unit testing in C++ - three lines of code.
VS is a bit poor in that you have to go through wizards to make a project to build and run the tests, so if you have a thousand classes you'd need a thousand projects. So for large projects on VS I've tended to organised the project into a few DLLs for independent building and testing rather than monolithic ones.
An alternative to static tests more similar to your 'poke and dribble' script could be done in python, using swig to bind your code to the interpreter, and python's doc tests . I haven't used both together myself. Again, you'd need a separate target to build the python binding, and another to run the tests, rather than it being just a simple 'run this class' button.
I would go with Boost.Test (see tutorial here)).
The idea would be to add a new configuration to your project, which would exclude from build all unnecessary cpp files. You would just have to add .cpp files to describe the tests you want to pass.
I am no expert in this area but i have used this technique in the past and it works !
I think you are talking about unit testing and mock objects. Here are couple of C++ mock object libraries that might be useful :-
googlemock which only works with googletest
mockpp
You are essentially asking how can I test one function instead of the whole application. That is what unit-testing is, and you will find many questions about unit-testing C++ on SO.