I have two different Environments (Say) QA and QA2 Setup for Karate API Testing.
I wanted to know if it is possible to run Tests on both for same feature files and have a comparative result. Like How many Tests are failing on QA vs QA2.
How can I achieve this?
This is a very specific need which is more a responsibility of your continuous integration and 3rd party reporting than Karate (in my opinion).
You can look at this thread for ideas on 3rd party report servers etc: https://github.com/intuit/karate/issues/619
You can probably take the JSON report files (or the Results data mentioned in the above thread) and write your own custom solution which might be the best approach.
Related
We are working on billing system (java based module)for that we would like to have a testing framework. That testing framework should be able to adoptable for any type of billing scenarios(eg: utility bill payments, water/electricity/or any other type billing) Normally the billing entity have common attributes like customer name/usage/ etc..I would like to pick a suitable testing platform to test our billing module.
It can be opensource/licensed software.
Can anybody suggest such a framework/engine?
If you wanna go for open source tool, then I'd recommend Selenium Webdriver with TestNG Framework. You can get lot of documentation and help on web.
You can go for Cucumber for describing various test scenarios (and their dependencies).
These scenarios will be backed by jUnit for gluing the description to executable code. jUnit will typically be used to write low level tests as well (for use by developers).
Cucumber has the benefit of giving you reports and can serve well for discussion with users and their representatives.
I would suggest following based on your given little description about your need :
1 - For functionality testing and to make all your scenarios automate use Selenium WebDriver
2 -Then if you want to priorities your testes , want to run tests through XML , want to run multiple tests then you can use TestNG Framework
Above are totally open source tool and you will get real benefit of those by scripting. They allow you to do scripting using programming languages like Java , Python , ruby and a little more. You will get all details once you visit my given links in above 2 points.
According to your given requirements I think above 2 tools are enough to make everything automate for testing.
All,
I'm trying to build up our pairwise coverage analysis for current and future tests on our project. In particular, we want to make sure that our existing NUnit tests are covering the most important pairwise cases.
To that end, I want to create a helper method that will take a list of all the different test cases that we have and forward them to Hexawise (UPDATE: I posted on their internal support site, and Hexawise doesn't have an API for this) or some similar pairwise analysis software. I want to do this directly in the code, probably via a POST command to a rest client.
Any suggestions?
Thanks!
Pairwiser (developed by Inductive) is a pairwise tool that has an API for doing exactly what you want. It also supports other common tasks related to pairwise testing.
I am trying to build a test program in c++ to automate testing for a specific application. The testing will involve sending requests which have a field 'CommandType' and some other fields to a server
The commandType can be 'NEW', 'CHANGE' or 'DELETE'
The tests can be
Send a bunch of random requests with no pattern
Send 100 'NEW' requests, then a huge amount of 'CHANGE' requests followed by 200 'DELETE' requests
Send 'DELETE' requests followed by 'CHANGE' requests
... and so on
How can I design my software (what kind of modules or layers) so that adding any new type of test case is easy and modular?
EDIT: To be more specific, this test will be to only test one specific application that gets requests of the type described above and handles them. This will be a client application that will send the requests to the server.
I would not create your own framework. There are many already written that follow a common pattern and can likely accomodate your needs elegantly.
The xUnit framework in all incarnations I have seen allows you to add new test cases without having to edit the code that runs the tests. For example, CppUnit provides a macro that when added to a test case will auto-register the test case with a global registry (through static initialization I assume). This allows you to add new test cases without cracking open and editing the thing that runs them.
And don't let the "unit" in xUnit and CppUnit make you think it is inappropriate. I've used the xUnit framework for all different kinds of testing.
I would separate out each individual test into it's own procedure or, if it requires code beyond a function or two, it's own source file. Then in my main routine I'd do something like:
void main()
{
run_test_1();
run_test_2();
//...
run_test_N();
}
Alternatively, I'd recommend leveraging the Boost Test Library and following their conventions.
I'm assuming you're not talking about creating unit tests.
IMHO, Your question is too vague to provide useful answers. Is this to test a specific application or are you trying to make something generic enough to test as many different applications as is possible? Where do these applications live? Are they client server apps, web apps, etc.?
If it's more than one application that you want your tool to test, you'll need an architecture that creates a protocol in between the testing tool and the applications such that you can convert the instructions your tool and consumers of your tool can understand, into instructions that the application being tested can understand. I've done similar things in the past but I've only ever had to worry about maybe 5 different "applications" so it was a pretty simple matter of summing up all the unique functionality of the apps and then creating an interfact that supports them all.
I wouldn't presume that NEW, CHANGE, and DELETE would be your only command types either. A lot of testing involves data cleanup, test reporting, etc. And applications all handle this their own special ways.
use C++ unit testing framework , Read this for Detail and examples
I want to stress test a web service method by calling it several thousand times in quick succession. The method has a single string parameter that I will vary on each call.
I'm planning on writing a Powershell script to loop and call this method a number of times.
Is there a better way to do this?
If you run call after call - it's not going to help you too much, as it will not show you how the service behaves under a heavy load of many simultaneous connections.
Go with some multi-threaded solution (I do not know if powershell has this).
Some opensource testing tools are listed here. Just set your web service to accept GET requests as well, not only SOAP(default), so you can form the urls.
For these situations I would use JMeter. You need to play around with it first, but it is very flexible, it will run requests in different threads, it will display the results graphically and it also allows you to script your jobs.
I would also recommend to start it not in the same machine as the server and if possible start two or more instances in different machines to simulate the load.
Personally, I'd use something like openSTA.
This allows a session with a web site to be recorded and then played back via a relatively simple script language.
You can also easily test web services and write your own scripts.
It allows you to put scripts together in a test in any way you want and configure the number of iterations, the number of users in each iteration, the ramp up time to introduce each new user and the delay between each iteration. Tests can also be scheduled in the future.
It's open source and free.
It produces a number of reports which can be saved to a spreadsheet. We then use a pivot table to easily analyse and graph the results.
It really depends on if there are other requirements. Such as logging, history, etc.
If quick and dirty is all you need, then you're good.
If you need something more robust then you can look at either custom building a test harness in the language of your choice or using things such as Mercury, MS Team Tester, nUnit or the like.
For those that stumble upon this and don't want to use the 3rd party options already mentioned. As the accepted answer mentions, a multi-threaded solution is best, this can be achieved in PowerShell:
ForEach -Parallel ($item in $collection) { }
https://learn.microsoft.com/en-us/powershell/module/psworkflow/about/about_foreach-parallel
Although the number of threads could be limited to 5: Does powershell's parallel foreach use at most 5 thread?
I am introducing automated integration testing to a mature application that until now has only been manually tested.
The app is Windows based and talks to a MySQL database.
What is the best way (including details of any tools recommended) to keep tests independent of each other in terms of the database transactions that will occur?
(Modifications to the app source for this particular purpose are not an option.)
How are you verifying the results?
If you need to query the DB (and it sounds like you probably do) for results then I agree with Kris K, except I would endeavor to rebuild the DB after every test case, not just every suite.
This helps avoid dangerous interacting tests
As for tools, I would recommend CppUnit. You aren't really doing unit tests, but it shouldn't matter as the xUnit framework should give you the set up and teardown framework you'll need to automatically set up your test fixture
Obviously this can result in slow-running tests, depending on your database size, population etc. You may be able to attach/detach databases rather than dropping/rebuilding.
If you're interested in further research, check out XUnit Test Patterns. It's a fine book and a good website for this kind of thing.
And thanks for automating :)
Nick
You can dump/restore the database for each test suite, etc. Since you are automating this, it may be something in the setup/teardown functionality.
I used to restore the database in the SetUp function of the database related unit test class. This way it was ensured that each test runs under the same conditions.
You may consider to prepare special database content for the tests, i.e. with less data than the current production version (to keep the restore times reasonable).
The best environment for such testing, I believe, is VMWare or an equivalent. Set up your database, transaction log and so on, then record the whole lot - database as well as configuration. Then to re-test, reload the image and database and kick off the tests. This still requires maintenance of the tests as the system changes, but at least the tests are repeatable, which is one of your greatest challenges in integration testing.
For test automation, many people use Perl, but we've found that Perl programs grow like Topsy and become convoluted. The use of Python as a scripting language (we run C++ tests) is worthwhile if you're trying to build a series of structured tests.
As #Kris K. says dumping and restoring the database between each test will probably be the way to go.
Since you are looking at doing testing external to the App I would look to build the testing framework in a language where you can take advantage of better testing tools.
If you built the testing framework in Java you could take advantage of JUnit and potentially even something like FitNesse.
Don't think that just because the application under test is C++ that means you are stuck using C++ for your automated testing.
Please try AnyDbTest, I think it is the very tool you are finding. (www.anydbtest.com).
Features:
1.Writing test case with Xml, not Java/C++/C#/VB code. Not need those expensive programming tools.
2.Supports all popular databases, such as Oracle/SQL Server/My SQL
3.So many kinds of assertion supported, such as StrictEqual, SetEqual, IsSupersetOf, Overlaps, and RecordCountEqual etc. Plus, most of assertions can prefix logic not operator.
4.Allows using an Excel spreadsheet/Xml as the source of the data for the tests. As you know, Excel spreadsheet is to easily create/edit and maintain the test data.
5.Supports sandbox test model, if one test will be done in sandbox, all database operations on each DB will be rolled back meaning any changes will be undone.
6.Allows performing data pump from one database/Excel into target database in testing initialization and finalization phase. This is easy way to prepare the test data for testing.
7.Unique cross-different-type-database testing, which means target and reference result set can come from two databases, even one is SQL Server, another is Oracle.
8.Set style comparison for recordset. AnyDbTest will tell you what is the intersection, or surplus or absence between the two record sets.
9.Sequential style comparison for recordset or scalar values. It means the two result set will be compared in their original sequence.
10.Allow to export result set of SQL statement into Xml/Excel file.