SUMMARY
The software I am working is an engineering analysis tool for renewable energy systems. It tests power outputs and things of this nature.
My current job task has been to test different configurations of said systems (i.e. use a evaporative cooling condenser instead of the default radiative cooling condenser via a drop down menu on the user interface) to ensure that the release version of this software functions across all spectrums as it should.
So far the process has been very simple to test: changing one factor of the base model and testing if the code functions properly as it should via the outputs generated.
How I need to pivot
But now we want to start testing combinations of tests. This change of workflow will be very tedious and time consuming for we are considering a factorial design approach in order to map out my plan of attack for what configurations to test, to ensure that all the code functions properly when changing multiple things. This could create many different types of configurations I will need to manually test.
All in all, does anyone have any suggestions for an alternative approach? I've been reading up on alternative software testing methods, but am not entirely sure if things like: regression testing, smoke testing, or sanity checks are better for this scenario or not.
Thanks!
EDIT
The software I am testing is being used on Visual Studios where I am utilizing the Google Test framework to manually test my system configurations.
Currently, each test that I create for a certain concentrated solar power system configuration demands that I manually find the difference in code (via WinMerge) between the default configuration (no changes made) to the alternative configuration. I use the code differences in Google Test Framework to simulate what that alternative config. should output by testing it against the accepted output values.
It's only going to get more complicated, with an aspect of manual user interface needed ... or so it seems to me.
How can I automate such a testing suite, when I'm needed to do so much back end work?
As per what I understand, to avoid the manual effort of testing too many combinations, an automation testing tool is needed in this case. If the software that you are testing is browser based, then Selenium is a good candidate. But if the tool is run as an application on Windows or Mac, then some other automation testing tool that supports Win/Mac apps would be needed. The idea is to create test suites with the different combinations and set the expected results for one time. Once the suite is ready, it can be run after any change to the software to verify that all the combinations work as expected without doing any manual work. However there would be an effort involved to create the test suite in the first place and then maintain it if the new scenarios occur or the expected results need to be modified.
It would be a pain to test all the many combinations manually each time, automation testing can surely ease that.
Related
I'm looking for a piece (or a set) of software that allows to store the outcome (ok/failed) of an automatic test and additional information (the test protocol to see the exact reason for a failure and the device state at the end of a test run as a compressed archive). The results should be accessible via a web UI.
I don't need fancy pie charts or colored graphs. A simple table is enough. However, the user should be able to filter for specific test runs and/or specific tests. The test runs should have a sane name (like the version of the software that was tested, not just some number).
Currently the build system includes unit tests based on cmake/ctest whose results should be included. Furthermore, integration testing will be done in the future, where the actual tests will run on embedded hardware controlled via network by a shell script or similar. The format of the test results is therefore flexible and could be something like subunit or TAP, if that helps.
I have played around with Jenkins, which is said to be great for automatic tests, but the plugins I tried to make that work don't seem to interact well. To be specific: the test results analyzer plugin doesn't show tests imported with the TAP plugin, and the names of the test runs are just a meaningless build number, although I used the Job Name Setter plugin to set a sensible job name. The filtering options are limited, too.
My somewhat uneducated guess is that I'll stumple about similar issues if I try other random tools of the same class like Jenkins.
Is anyone aware of a solution for my described testing scenario? Lightweight/open source software is preferred.
I'm working on a project to create a simulator (for modeling biological systems) in C++. The simulator takes an input file of parameters and then generates an output file with hundreds of molecule counts at different time points from the simulation. I'm using Google Test for all of my unit testing. I also want to include some higher level tests where I supply an input file with various model parameters and then check that the output file matches some reference file. Someone recommended using bash-tap for these higher level tests, but I'd prefer to stick to Google Test if possible. Is it possible to use Google Test for the higher level tests that I've described here?
We write CAE software (simulators) and use Google Test. We face similar issues, so hopefully you'll find the answers practical.
You can write the higher-level tests, but you will often have to do more than just "EXPECT_EQ()" for checking pass/fail. For example, if you had to test the connectivity of two abitrary graphs, it can be difficult if the algorithms are allowed to vary the order of nodes. Or, if you are comparing a matrix, sometimes you can have cases where the matrix rows and columns can be switched with no problem. Perhaps round-off error is ok. Be prepared to deal with these types of problems as they will be much more of an issue with a full simulator than with a unit test.
A more practical issue is when your organization says "run all tests before you check in." Or, maybe they run every time you hit the build button. If that's the case, you need to differentiate these unit tests from the higher level tests. We use Google Test Runner in Visual Studio, and it expects to run everything where the filename is "*Test*". It is best to name the higher level tests something else to be clear.
We also had to turn our entire executable into a DLL so that it could have tests run on top of it. There are other approaches (like scripting) which could be used with Google Test, but we've found the executable-as-a-dll approach to work. Our "real" product executable is simply a main() function that calls app_main() in the dll.
And, one final tip when using the Runner: If your app gets the --gtest_list_tests argument, don't do a bunch of expensive setup:
// Don't run if we are just listing tests.
if (!::testing::GTEST_FLAG(list_tests))
{
// Do expensive setup stuff here.
}
int result = RUN_ALL_TESTS();
if (!::testing::GTEST_FLAG(list_tests))
{
// Do expensive shutdown stuff here.
}
Unit and integration testing is usually performed as part of a development process, of course. I'm looking for ways to use this methodology in configuration of an existing system, in this case the Asterisk soft PBX.
In the case of Asterisk, the configuration file is as much a programming language as anything else, complete with loops, jumps, conditionals, etc., and can get quite complex. Changes to the configuration often suffers from the same problems as changes to a complex software product - it can be hard to foresee all the effects without tests in place. It's made worse by the fact that the nature of the system is to communicate with external entities, i.e. make phone calls.
I have a few ideas about testing the system using call files (to create specific calls between extensions) while watching the manager interface for generated events. A test could then watch for an expected result, i.e. dialling *99# should result in the Voicemail application getting called.
The flaws are obvious - it doesn't test the actual result, only what the system thinks is the result, and it probably requires some modification of the system under test. It's also really hard to write these tests robustly enough to only trigger on the expected output, especially if the system is in use (i.e. there are other calls in progress).
Is what I want, a testing system for Asterisk, impossible? If not, do you have any ideas about ways to go about this in a reasonable manner? I'm willing to put a fair amount of development time into this and release the result under a friendly license, but I'm unsure about the best way to approach it.
This is obviously an old question, so there's a good chance that when the original answers were posted here that Asterisk did not support unit / integration testing to the extent that it does today (although the Unit Test Framework API went in on 12/22/09, so that, at least, did exist).
The unit testing framework (David's e-mail from the dev list here) lets you execute unit tests directly within Asterisk. Tests are registered with the framework and can be executed / viewed through the CLI. Since this is all part of Asterisk, the tests are compiled into the executable. You do have to configure Asterisk with the --enable-dev-mode option, and mark the tests for compilation using the menuselect tool (some applications, like app_voicemail, automatically register tests - but they're the minority).
Writing unit tests is fairly straight-forward - and while it (obviously) isn't as fully featured as a commercial unit test framework, it gets the job done and can be enhanced as needed.
That most likely isn't what the majority of Asterisk users are going to want to use - although Asterisk developers are highly encouraged to check it out. Both users and developers are probably interested in integration tests, which the Asterisk Test Suite provides. At its core, the Test Suite is a python script that executes other scripts - be they lua, python, etc. The Test Suite comes with a set of python and lua libraries that help to orchestrate and execute multiple Asterisk instances. Test writers can use third party applications such as SIPp or Asterisk interfaces (AMI, AGI) or a combination thereof to test the hosted Asterisk instance(s).
There are close to 200 tests now in the Test Suite, with more being added on a fairly regular basis. You could obviously write your own tests that exercise your Asterisk configuration and have them managed by the Test Suite - if they're generic enough, you could submit them for inclusion in the Test Suite as well.
Note that the Test Suite can be a bit tricky to set up - Leif wrote a good blog post on setting up the Test Suite here.
Well, it depends on what you are testing. There are a lot of ways to handle this sort of thing. My preference is to use Asterisk Call Files bundled with dialplan code. EG: Create a callfile to dial some public number, once it is answered, hop back to the specified dialplan context and perform all of my testing logic (play soundfiles, listen for keypresses, etc.)
I wrote an Asterisk call file library which makes this sort of testing EXTREMELY easy. It has a lot of documentation / examples too, check it out here: http://pycall.org/. That may help you.
Good luck!
You could create a set of specific scenarios and use Asterisk's MixMonitor command to record these calls. This would enable you to establish a set of sound recordings that were normative for your system for these tests, and use an automated sound file comparison tool (Perhaps something from comparing-sound-files-if-not-completely-identical?) to examine the results. Just an idea.
Unit testing as opposed to integration testing means your code is supposed to be architectured so the logic itself is insulated from external dependencies. You said "the configuration file is as much a programming language as anything else" but that's the thing --- real languages has not just control flow but abstraction capabilities, which allow you to write the logic in a way that can be unit tested. That's why I keep logic outside of asterisk as much as possible.
For integration testing, script linphonec to drive your application, and grep the asterisk console to see what it's doing.
You can use docker, and fire up temporary asterisk instances for each test.
I'm working on a very large, data-intensive legacy application. Both the code base & database are massive in scale. A great deal of the business logic is spread across all of the tiers including in stored procedures.
Does anybody have any suggestions on how to begin applying "unit" tests (technically integration tests because they need to test across tiers for a single aspect of almost any given process) into the application in an efficient way? The current architecture does not easily support any type of injection or mocking. New code is being written to facilitate testing, but what about the legacy code? Because of the strong dependency on the data itself and business logic in the database, I'm currently using inline sql to find data to use for testing but these are time consuming. Creating views and/or stored procedures will not suffice.
What approaches have you taken (if applicable)? What worked? What didn't & why? Any suggestions would be appreciated. Thanks.
Get a copy of Working Effectively with Legacy Code by Michael Feathers. It is full of useful advice for working with large, untested codebases.
Another good book is Object Oriented Reengineering Patterns. Most of the book is not specific to object-oriented software. The full text is available for free download in PDF format.
From my own experience: try to...
Automate the build and deployment
Get the database schema into version control, if it isn't yet. Usually databases include reference data that the transactional code needs to exist before it can work. Get this under version control too. Tools like dbdeploy can help you easily rebuild a schema and reference data from a sequence of deltas.
Install a version of the database (and any other infrastructure services) onto your development workstation. This will let you work on the database without continually having to go through DBAs. It's also faster than using a schema on a shared server in a remote datacentre. All major commercial database servers have free (as in beer) development versions that work on Windows (if you're stuck in the unenviable situation of developing on Windows and deploying on Unix).
Before starting work on an area of the code, write end-to-end tests that roughly cover the behaviour of the area you're working on. An end-to-end test should exercise the system from outside -- by controlling its user interface or interacting through network services -- so you won't need to change the code to put it into place. It will act as an (imperfect) regression test and give you more confidence to refactor the internals of the system towards a structure that is easier to unit test.
If there are manual test plans, read them and see what can be automated. Most manual test plans are almost entirely scripted and so are low-hanging fruit for automation
Once you've got end-to-end tests coverage, refactor the code into more loosely coupled units as you modify and/or extend it. Surround those units with unit tests.
Things to avoid:
Copying data from the production database into the environment you use for automated testing. This will make your tests unpredictable. Sure, run the system against a copy of production data, but use that for exploratory testing, not regression testing.
Rolling back transactions at the end of tests to isolate tests from one another. This will not test behaviour that only happens when transactions are committed, and will throw away data that is valuable for diagnosing test failures. Instead, tests should ensure the database is in a known initial state when they start.
Creating a "tiny" data set for tests to run against. This makes tests hard to understand because they cannot be read as a single unit. The "tiny" data set soon grows very large as you add tests for different scenarios. Instead, tests can insert data into the database to set up the test-fixture.
“Testing Legacy Application Modernization,” highlights:
High level overview of how tests are created in AscentialTest
Ways to convert the legacy objects to the new platform Components of Object definition
How to ensure that the modernized version of the application produces the same results
For more details on usage of testing legacy application, do check here:
http://application-management.cioreview.com/whitepaper/testing-legacy-application-modernization-wid-529.html
As mentioned before, there are some very good books out there. Highly recommended to take a look at Working Effectively with Legacy Code.
Something you could do is following a data driven approach, observe your application and introduce tests where you have more “pain”. A semi-deterministic approach you might find useful: https://link.medium.com/zY9Tysfne9
i have project in .net , i want to test it.
But i dont know anything about testing and its method.
how can i go ahead with testing.
which method is better for me for begining?
Is there anything to decide which testing method is taken into account for better result?
There is no "right" or "wrong" in testing. Testing is an art and what you should choose and how well it works out for you depends a lot from project to project and your experience.
But as a professional Tester Expert my suggestion is that you have a healthy mix of automated and manual testing.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
Since it is not clear about the scale of the project you have, all you need to do is make sure:
Your tests are trustworthy - you should know they are telling u the truth.
Repeatable
Consistent - If you repeat test with same test data it should provide same output.
Proves you are covering all the problem areas.
To get this you can use:
Standard way : NUnit, MbUnit (myFav) or xUnit (havent got around to working with it) or MSTest
Quick and Dirty : Console app (not cool, not so flexible)
If you are using .Net, I'd recommend checking out NUnit. It's a great testing framework to use.
As far as learning about the "testing method", there are many different ways to test an application. When using a tool like NUnit, for example, you are writing automated tests which run without user interaction. In these types of tests, you typically write tests for each of the public methods in your application, and you ensure that given known inputs, these methods produce the expected outputs. Over time as the application changes (via enhancements, bug fixes, etc.) you have a core set of tests that you can re-run to ensure nothing breaks as a result of the changes. You can also do failure testing to ensure that given an invalid set of inputs to a method, it throws the proper exceptions, etc.
Besides automated testing with a tool like NUnit, it's also important to ensure that your end users test the product. "End users" here could be a Quality Assurance group in your company, or it could be the actual customer. The point is that you need to ensure that someone actually uses your application to make sure it works as expected, because no matter how good the automated tests are, there will still be many things you won't think of that your users will discover. One way to approach this type of testing is to write test scenarios, and have your users execute them to make sure the scenario results in the correct behavior.
I think the best testing approach combines both of the above, namely automated testing and user testing (with documented test scenarios).