I've become responsible for cleaning up an old unit-testing environment. The existing environment contains a ton of executables (1000+ shell scripts, compiled binaries, etc) and each one returns code 0 or 1 and some output depending on the results. Tests can also time out. A set of PERL scripts goes through, runs each executable file, and collects the results into a big XML file which gets rendered into a web page. This system works great, but isn't very extensible or fast.
In addition to cleaning/speeding this up, I would like to implement concurrent testing. Right now the tests run one at a time. Many of the tests require resource locks (ports, files, etc) and there's no listing of which are safe to run simultaneously. An option here is to run each one in a VM of some kind.
Is there a framework or tool designed for this type of situation? What's the best way to approach it if I have to write my own brand new system? My limitation is that I cannot change the 1000+ test executables. I was thinking something like PyUnit with Python unit tests that use subprocess or similar to execute the existing tests and convert the results into a Pythonic format. Is this overkill? Could such a system support isolation of processes to avoid deadlocks and race conditions between tests? In a worst-case, I was thinking of using Python to SSH into one of several VM's to run the tests.
Any help or guidance would be appreciated.
Edit: Python isn't strictly necessary. Another solution might just break the test set into several M-sized chunks and run each test in an independent VM over SSH until all M-sized chunks are done. I'm hoping there's a tool out there that solves this problem.
There is no out-of-the-box or customize it for your needs solution of which I am aware to solve the problem you are facing.
Looking at your problem, there are several distinct needs which standout:
Test Tagging
Test Execution
Test Result Capture
The first issue you need to address is how are you going to identify and track the tests that you can execute in a given environment, concurrently, etc.
If you were using nose (i.e. is nicer testing for python), you would be able to use the Attribute selector plugin to tag the tests with various attributes.
nose would also assist you in your test execution and when coupled with test tagging, would allow you to run tests in parallel, etc. Using nose, you should be able to run an external executable and assert based on its status code.
The final problem you face is how to capture test output in a proprietary format and translate it to a format that can be ingested to readily available tools. Again, I believe nose could help you out here. You could build a nose plugin that would take your proprietary format and translate it to XUnit format and report results that way.
With all of the above in mind, here is how I would tackle this situation:
Create a test wrapper class based on nose which
Can be tagged
Execute a program and capture result output
Translate that output to XUnit
Create a wrapper for each test
Figure out how to automate this process, because it is going to be tedious
Build a test execution harness, which
Spins Up one or more VMs
Loads and runs a test wrapper
Capture the results
Related
SUMMARY
The software I am working is an engineering analysis tool for renewable energy systems. It tests power outputs and things of this nature.
My current job task has been to test different configurations of said systems (i.e. use a evaporative cooling condenser instead of the default radiative cooling condenser via a drop down menu on the user interface) to ensure that the release version of this software functions across all spectrums as it should.
So far the process has been very simple to test: changing one factor of the base model and testing if the code functions properly as it should via the outputs generated.
How I need to pivot
But now we want to start testing combinations of tests. This change of workflow will be very tedious and time consuming for we are considering a factorial design approach in order to map out my plan of attack for what configurations to test, to ensure that all the code functions properly when changing multiple things. This could create many different types of configurations I will need to manually test.
All in all, does anyone have any suggestions for an alternative approach? I've been reading up on alternative software testing methods, but am not entirely sure if things like: regression testing, smoke testing, or sanity checks are better for this scenario or not.
Thanks!
EDIT
The software I am testing is being used on Visual Studios where I am utilizing the Google Test framework to manually test my system configurations.
Currently, each test that I create for a certain concentrated solar power system configuration demands that I manually find the difference in code (via WinMerge) between the default configuration (no changes made) to the alternative configuration. I use the code differences in Google Test Framework to simulate what that alternative config. should output by testing it against the accepted output values.
It's only going to get more complicated, with an aspect of manual user interface needed ... or so it seems to me.
How can I automate such a testing suite, when I'm needed to do so much back end work?
As per what I understand, to avoid the manual effort of testing too many combinations, an automation testing tool is needed in this case. If the software that you are testing is browser based, then Selenium is a good candidate. But if the tool is run as an application on Windows or Mac, then some other automation testing tool that supports Win/Mac apps would be needed. The idea is to create test suites with the different combinations and set the expected results for one time. Once the suite is ready, it can be run after any change to the software to verify that all the combinations work as expected without doing any manual work. However there would be an effort involved to create the test suite in the first place and then maintain it if the new scenarios occur or the expected results need to be modified.
It would be a pain to test all the many combinations manually each time, automation testing can surely ease that.
I'm looking for a piece (or a set) of software that allows to store the outcome (ok/failed) of an automatic test and additional information (the test protocol to see the exact reason for a failure and the device state at the end of a test run as a compressed archive). The results should be accessible via a web UI.
I don't need fancy pie charts or colored graphs. A simple table is enough. However, the user should be able to filter for specific test runs and/or specific tests. The test runs should have a sane name (like the version of the software that was tested, not just some number).
Currently the build system includes unit tests based on cmake/ctest whose results should be included. Furthermore, integration testing will be done in the future, where the actual tests will run on embedded hardware controlled via network by a shell script or similar. The format of the test results is therefore flexible and could be something like subunit or TAP, if that helps.
I have played around with Jenkins, which is said to be great for automatic tests, but the plugins I tried to make that work don't seem to interact well. To be specific: the test results analyzer plugin doesn't show tests imported with the TAP plugin, and the names of the test runs are just a meaningless build number, although I used the Job Name Setter plugin to set a sensible job name. The filtering options are limited, too.
My somewhat uneducated guess is that I'll stumple about similar issues if I try other random tools of the same class like Jenkins.
Is anyone aware of a solution for my described testing scenario? Lightweight/open source software is preferred.
I've spent the last few days looking around for an existing solution to a functional testing problem, but I am out of ideas and would appreciate some SO help!
I've got a preexisting suite of functional networking tests currently written in C++ using Boost.Test and Google Test, but might become rewritten into Rust soon. These generally take the following form:
unit test fixture {
1. Start a thread representing "the server" which goes and listens on some localhost port for incoming network connections.
2. Do client stuff representing "the client" to that localhost port.
3. Join the server thread, fetching an errors or problems.
4. Exit with success or failure.
}
This is great, and it works well. However it only tests loopback and in the real world the server component is probably in its own process running behind a NAT routed network, so it's not particularly realistic and therefore not really testing the code. What I think I'm looking for is some method of splitting the server thread part off into its own process, and then some method of getting the server test process and the client test process to work together to run the functional tests. Ideally speaking the server and client processes would run in separate "machines", this is something I can automate using OpenVZ scripting fairly easily so consider that problem out of scope, though it makes forking the process non-ideal.
I had been thinking that in this age of Web 2.0 et al surely this is a very common functional test scenario, and therefore that established patterns and test frameworks would abound. I mean, I as an old timer thinks "DCOM" as my first thought on how to solve this, though that's a 1990s Microsoft only solution. And maybe there is some modern and portable equivalent, and I am not searching for the right terms here, so here is my first question:
Is there any standard functional testing library or framework which extends Google Test or Boost.Test etc which lets you easily choose at runtime whether the server and client parts of each functional test should run as threads or as processes or best of all, as processes inside their own virtual machine with its own network stack?
This test scenario is surely common as muck. But let's assume it isn't, and no such tool exists. If it doesn't, then we need to extend Boost.Test or Google Test with some extra support. Firstly, we need to associate with each test fixture a supporting "server" part test fixture, and for the threaded test scenario we need to always run the server and client test fixtures concurrently. So, my second question:
Is there any way of strongly associating two test fixtures in any of the popular C++ or Rust unit testing frameworks where the two fixtures are seen as two halves of the same test, and always executed concurrently?
This leaves the second part: how to get a unit test framework to execute only the client parts in one process and only the server parts in the other process, and to do both always concurrently and in sync with one another, and moreover to merge the junit XML output from both parts into a single test result. So:
Is there any alternative functional testing approach, methodology, or open source solution which is better suited for distributed network functional testing than unit test frameworks such as Google Test or Boost.Test? Preferably something libvirt aware so it can orchestrate virtual machines as part of the testing setup and teardown? For example, is there some Jenkins plugin or something which could use Jenkins slaves in each OpenVZ container to orchestrate the concurrent execution of the multiple parts of each of the functional tests? Or is old fashioned CORBA still the least worst solution here? Is there maybe some way of automatically wrapping up test fixtures into a REST HTTP API?
I did do a quick review of the major integration testing frameworks, so Citrus, STAF and Twister. I'll be honest in saying they all seem way overkill for what I want which is a quick and easy way of making the existing functional test suite use a more realistic network routing than loopback. That's all I really want essentially, and I don't care how it's done so long as the check and requires still appear in Jenkins. Over to you Stackoverflow!
My thanks in advance for any help.
I have had similar requirements but I am come from the Java side of the world. What you can easily do is having a distributed management of nodes / machines using jGroups.
Once you understand how it works you can build a distributed system of nodes by just using 100 lines of code. With this system you can span and control child processes on each of those systems and check output and everything yourself. Should only cost you a day to take a jGroup example and get this running.
Once you have the infrastructure copy code and execute it as independent process your control is easy. Then use some of those nodes and get Selenium up and use a number of browser windows and execute scripts (or use Sikuli) and do your testing. Since the Selenium process is again Java you can generate all kind of reports you print to console or send directly to the cluster since those processes can join the cluster using jGroups too.
Such a system can be implemented in a week and it is really under your control. Very simple to do and very extendable.
Also you can provide plugins for Jenkins, Jira or Quality Center to interact with it and trigger test execution and management.
How can I perform unit testing in Simulink, or preferably, Stateflow?
I'm a fan of agile software methods, including test driven development. I'm responsible for the development of safety critical control software and we're using Matlab/Simulink/Stateflow for the development of it. This toolset is selected because of the link with plant (hardware) models. (model-in-the-loop, hardware-in-the-loop)
I have found some links on Stackoverflow: Unit-testing framework for MATLAB: xunit, slunit and doctest.
Does anyone have experience in using those or different unit test frameworks?
How to link this to continuous integration systems (i.e. Hudson)?
EDIT: This is now much easier and getting easier all the time with the Jenkins plugin for MATLAB
ORIGINAL ANSWER:
As Craig mentioned there is indeed a framework in MATLAB introduced in R2013a. Furthermore, this framework added a TAPPlugin in R2014a which outputs the Test Anything Protocal. Using that protocol you can set up your CI build with a TAPPlugin (eg. Jenkins, TeamCity) so that the CI system can fail the build if the tests fail.
Your CI build may look like a shell command to start MATLAB and run all your tests:
/your/path/to/matlab/bin/matlab -nosplash -nodisplay -nodesktop -r "runAllMyTests"
Then the runAllMyTests creates the suite to run and runs it with the tap output being redirected to a file. You'll need to tweak specifics here, but perhaps this can help you get started:
function runAllMyTests
import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;
try
% Create the suite and runner
suite = TestSuite.fromPackage('packageThatContainsTests', 'IncludingSubpackages', true);
runner = TestRunner.withTextOutput;
% Add the TAPPlugin directed to a file in the Jenkins workspace
tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));
runner.run(suite);
catch e;
disp(e.getReport);
exit(1);
end;
exit force;
EDIT: I used this topic as the first two posts of a new developer oriented blog launched this year
Unit testing Simulink is not straightforward, unfortunately. Mathworks have the SystemTest. Alternatively, you can roll-your-own Simulink testing framework, which is the approach that we've followed and is not too difficult, but you may need to built test-harnesses programmatically.
In order to integrate with CI, you need to create a function/script that executes all the tests, then you can use the command-line parameters for MATLAB.exe to run a script on start-up. I'm not sure anyone has a good way of integrating the test reports with the CI software, though. Just look at the number of comments in Unit-testing framework for MATLAB.
With 2015a Matlab introduces a new product name "Simulink Test". Perhaps that'll simplify this mess.
http://www.mathworks.com/products/simulink-test/features.html#manage-test-plans-and-test-execution
R2016b introduces integration between Simulink Test and MATLAB Unit Testing framework. Tests created with Simulink Test using Test Manager (*.mldatx) are recognized by and can be run natively using the MATLAB Unit Test Runner and thus you can generate JUnit style XML test results or TAP test results facilitating Continuous integration workflows.
See this reference for more information: https://www.mathworks.com/help/sltest/ug/run-test-files-using-matlab-unit-test.html?s_tid=gn_loc_drop
The documentation shows an example of producing TAP results using matlab.unittest.plugins.TAPPlugin but you can use XMLPlugin (https://www.mathworks.com/help/matlab/ref/matlab.unittest.plugins.xmlplugin-class.html) instead just as easily.
This should open up a better integration within just MATLAB environment even without CI in the picture with the ability to have MATLAB and Simulink Tests together in the same test suite and have them run together seamlessly. For example if you have a directory MYDIR with both native MATLAB unit tests and Simulink Tests, you can do something as simple as the follows to execute both kinds of tests in one shot:
results = runtests(MYDIR)
If your system is complex, you should decompose it using Model Reference and test each of these independently.
An other solution (more "old school") is to put your main blocks in a library and to create small models.
To test these submodels and especially those with a State Machine (Stateflow), the best is to create temporal test cases
with the Signal builder block. You have a powerful function signalbuilder to interact with this block and load test cases. My method is to get for each case of each submodel an input file and an output file. Your outputs of the model are the "correct" output and the one from the blocks. The model is run with sim (no external inputs) and the 2 outputs are compared with a script the indicated which signal is different (and when).
You could use an existent system but I prefer to use my own to run each case (or some of them).
I don't have any public code for that but that's the way I use. I don't use a CIS so I can't answer the second part of your question.
Matlab (since 2013b) has built-in support for xUnit, in the form of the Unit Testing Framework.
I haven't used it but since it's possible to run simulink from Matlab with sim() then this framework can be used to test your simulink models. You libraries and possibly models will need a wrapper to execute it as the other answerers have noted.
There are plenty of examples on the Mathworks site, unfortunately non of them run simulink models. I'd code an example for you, but I don't have ML2013b :-(
In order to initiate your tests from a CI (I use Jenkins) then you can call matlab to run a .m file that runs your test suite, this example cmd script will call Run_Tests.m from Matlab:
IF EXIST "C:\Program Files (x86)\MATLAB\R2013b\bin\win32\matlab.exe" (
REM WinXP
"C:\Program Files (x86)\MATLAB\R2013b\bin\win32\matlab.exe" -r "Run_Tests;exit" -logfile matlab.log
) ELSE (
REM Win7
"C:\Program Files\MATLAB\R2013b\bin\win32\matlab.exe" -r "Run_Tests;exit" -logfile matlab.log
)
Note that if a startup.m exists in the directory that you call Matlab from, then it'll be executed automatically beforeRun_Tests.m`.
I think you are searching for something like EZTEST. It is intended for your special purpose: Test driven development for Simulink and Stateflow on unit level. For safety critical software, there is also a Safety Manual included, which describes what is covered regarding ISO 26262.
Edit: I am a developer of this software, so my opinion may be biased. But I am not involved in the marketing or sale of the product. I am just posting this, because I know that there are little to none unit test frameworks out there, meeting the questioner's needs (as the answers might suppose).
Testing units using SIL and PIL is also supported. Unfortunately I am not familiar with Hudson, so I cannot address this part of the question.
I've seen different solutions to the problem of unit testing Simulink models. Simulink Verification & Validation which did not support xUnit concepts of test runners and test suites at the time of examination, TPT being overloaded with functionality, not easy to use and very hard in terms of changeability and maintainability.
Furthermore I've seen custom solutions with Matlab scripts and Excel tables which were lightweight but also difficult in terms of understandability and maintainability. I'd still not recommend using any of these approaches, at least not for unit testing.
In the end, we ended up using a C unit testing framework (CUnit) testing the generated code. While this definitely has the disadvantage that you have to generate code before testing, it also has a lot of advantages, like easy integration into CI systems, high flexibility of writing unit tests, fast execution of unit tests and last but not least refactoring capabilities in terms of switching from Simulink to another model-based environment or to hand-written code. Especially the last point should not be underestimated since I have seen many Simulink models that should have been hand-written modules in the first place. Nowadays, I'd recommend using GoogleTest instead of CUnit.
I've been asked to help out with a project which has made extensive use of VBScript to process a whole bunch of text files and generate certain outputs - sanitized files, SQL entries etc.. The script is going to be scheduled to run as a Scheduled Task with operation based on the parameters passed to the script. There's no user interface.
Are there any tools out there that I can use to automate the testing?
Can I write unit tests that target specific functions within the script without executing the script's start up code etc..?
It sounds like you should be looking at tools at the acceptance/functional/system level rather than unit level.
A good match for what it sounds like you're trying to achieve might be (I've never used it in production) TextTest. It will allow you to run your scripts and analyse the text that being returned, the documentation is fairly thorough and there are decent tutorials.
It's impossible to answer whether or not you can run the scripts without the startup code, but it should be possible to refactor that code away into separate files/routines (either scripts or ini files or a combination of both) and get test specific versions returning canned answers which will allow you to isolate the methods under test. This is the same principle as with any test setup.