I googled and found the below helpful references. Currently I want to run all from the command-line (for easy of execution & quickness) in cases:
A specific test (ie. a test written by a method marked [TestMethod()])
All tests in a class
All impacted tests of the current TFS pending change of mine.
All tests
All tests except the ones marked as category [TestCategory("some-category")]
I'm not sure how can I write a correct command for my needs above.
References:
the MSTest.exe http://msdn.microsoft.com/en-us/library/ms182487.aspx
the MSTest.exe's detailed options http://msdn.microsoft.com/en-us/library/ms182489.aspx
obtaining the result http://msdn.microsoft.com/en-us/library/ms182488.aspx
[Edit]
After a while, I found the below useful tips.
run Visual Studio unit tests by using MSTest.exe, located at %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe in my case.
using /testcontainer:Path\To\Your\TestProjectAssembly.dll to indicate where your tests are coded. You can specify multiple '/testcontainer' options if required.
using /test:TestFilter to filter the tests to run. Note that this filter is applied to the full test method name (ie. FullNamespace.Classname.MethodName)
Currently I can have some answers for my needs:
A specific test (ie. a test written by a method marked [TestMethod()])
Use MSTest.exe /container:TheAssemblyContainingYourSpecificTest /test:TheSpecificTestName
All tests in a class
Use MSTest.exe /container:TheAssemblyContainingYourClass /test:TheClassNameWithFullNamespace
Note that the /test: is the filter which uses the full name of the class when filtering.
The others are still left unknown. Please disscuss if you know how.
For number 4. To run all tests in an assembly it's simply:
mstest /testcontainer:YourCompiledTestAssembly.dll
For question
5 All tests except the ones marked as category
[TestCategory("some-category")]
Use
mstest.exe /testcontainer:yourTests.dll /category:"!some-category"
If you need to exclude more than one category, use
mstest.exe /testcontainer:yourTests.dll /category:"!group1&!group2"
Reference: /category filter
You might be interested by the Gallio bundle. It provides a free common automation platform to run your tests (MSTest, MbUnit, NUnit, xUnit, etc.) with various test runners (GUI, command line, PoSh, plugins for 3rd party tools, etc.)
In particular you may want to use Gallio.Echo which is a nice command line test runner:
The Gallio test runners have also filtering capabilities to run a subset of your unit tests only (e.g. per category, per fixture, etc.)
** adding this due to errors I've encountered.
To run all just use '''vstest.console.exe .\x64\Release\UnitTesting.dll'''
vstest.console.exe is not deprecated so you will not need the /nologo suppression.
If needed it also has --TestCaseFilter|/TestCaseFilter:
Related
I'm a new NUnit user, using NUnit 3.9 under Visual Studio Community 2017. I'm using it on a pet open source library project, and it's going well once I got the hang of it.
The library accesses a publicly available government website via a documented API. Most of my tests use local data, so that I have a stable bed to compare against, and so that I can test without going out to the website every time.
I would like to set it up so that normally, the tests that hit the server do not run. I run the tests over and over as I tweak the code, and just as a matter of courtesy, don't want to bang on the server. Also, I'd like to be able to test even when the remote system is down or when I don't have Internet access.
Is there any way to group or tag my tests so that normally only the ones using local data run, but that I can still, when necessary, run the ones that exercise the server access? Either specifying "run these" or "exclude these" would be fine.
I've grouped the tests into two different classes, UnitTestOffline.cs and UnitTestOnline.cs, and was hoping I could somehow run the tests on a class-by-class basis, but haven't found a way to do that.
You'll get better answers if you say specifically how you run your tests, since there are a number of ways to do it. Since you mention VS2017, I'm going to assume that you are using the NUnit 3 VS Adapter, but let us know if you are using some other approach.
In the VS adapter, use the dropdown to display your tests by class. Right click on the class for which you want to run tests and run them.
If you decide to categorize tests using the CategoryAttribute, you can display tests by "trait" in Visual Studio. As before, right click on the group you want to run tests for and run them.
If you get a lot of tests, you might want to put your unit tests in one assembly and your integration tests in another. In that case, display the tests by project, right click on the project you want and run them.
All of this can also be done using the nunit3-console command-line runner as well. To select by class or category, you use the --where option. To select by assembly, you merely enter the name of the assembly you want on the command-line.
Seems like you want to categorize your tests (unit test, integration tests...) and run only the unit tests... you could use [Category] for that.
In the nunit GUI you could /include /exclude category after that and run only the one you want.
And probably that the filtering of Visual Studio could work.
Try to see one of the solution suggested here as well
I'm trying to only run a certain subsection of xunit tests using the tfs Test Filter Criteria:
The problem is that this is not doing anything, it still runs every test regardless of it's name, what might I be missing here? Is there something else I need to be doing since I'm using Xunit? Or is there another method of excluding tests in the test unit step I can use.
Test filter criteria: Filters tests from within the test assembly files. For example, “Owner=james&Priority=1”. This option works the
same way as the console option /TestCaseFilter for vstest.console.exe
For more information, see
https://msdn.microsoft.com/en-us/library/jj155796.aspx
Please double check your property of FullyQualifiedName in your assembly files. I'm not sure if /TestCaseFilter is also support in xUnit. You could give a try with directly running the test using command line (vstest.console.exe). If it's not work, then this should also not work in TFS build task.
Add a related link talk about test filter for your reference: VSTS/TFS VISUAL STUDIO TEST TASK – FILTER CRITERIA
We've customized a product which includes their own phpunit test suite. In Jenkins, I have two jobs setup: the first runs our own test suite that covers our customizations, and the second job runs the existing core unit tests.
The core unit tests were not designed to be run on a customized version, so failures are expected. Out of the ~5000 tests, 81 fail. What I'd like to setup in Jenkins, is have the build marked as a failure only if the number of failed tests changes from the previous build.
I've looked at the Performance plugin but the documentation seems sparse and I'm trying to find something that matches our use case.
Any suggestions?
You should have a look at the plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin
It handle a threasolding mechanism (I specified this requirement for the xunit plugin when my team developed it )
expect this helps..
But you want to associates the failure to a change ....
Hum maybe more complex .. have to ask .. if such thing should be developped.
I have searched for a while and can't find an answer on here for the first time ever.
I have a solution which contains multiple C# projects. Two of these projects are unit test projects. I am building this solution with Jenkins, attempting to run all the unit tests, then packaging it for an internal NuGet server.
The issue I am having is that I am running the tests for one unit test project after the other, I think this is causing the second set of tests to save their results over the first set because I can see in the console output that both sets of tests ran and passed, however the Test Results in Jenkins only shows the second set of tests.
I am running these with a windows batch command using nunit-console like this:
nunit-console.exe MyFirstProject.UnitTests/bin/Debug/MyFirstProject.UnitTests.dll
nunit-console.exe MySecondProject.UnitTests/bin/Debug/MySecondProject.UnitTests.dll
Is there a better way I can run these so that all test results are recorded?
Thanks!
Figured it out. For anyone who runs into this it's quite simple, you just have to put both (or however many) assemblies in the same command like this:
nunit-console.exe MyFirstProject.UnitTests/bin/Debug/MyFirstProject.UnitTests.dll MySecondProject.UnitTests/bin/Debug/MySecondProject.UnitTests.dll
Now all my test results show up!
As an addition to anyone else who has this type of issue:
To get one report from two groups of tests run from the same assembly, define two Windows Batch Command build steps, each with its own category and result file :
"%nunit%\nunit-console.exe" "%WORKSPACE%\MyTests.sln" /include:TestCategory1 /xml=nunit-result1.xml
and
"%nunit%\nunit-console.exe" "%WORKSPACE%\MyTests.sln" /include:TestCategory2 /xml=nunit-result2.xml
Add a Post Build action to Publish the NUnit Test result report and define the XML filename as *.xml.
The Nunit plugin will concatenate the different test results into one result that can be viewed and emailed.
This way will also solve the OP's question.
I have a big mess with 100 tests in one class and running all of them by clicking "Test project (...). They run in a random order and I would like them to run in a specific order - from beginning to the end, the same order that I wrote them. In eclipse it's not a problem because eclipse just works like that, how to do it in netbeans?
Any help will be appreciated.
Edit (due to answers): Tests order is required for the clearance of the log. They are independent.
If your tests needs to run in a specific order, something is wrong with your design.
2 test that needs to run one after another are 1 test. Consider this before searching for a solution.
check this https://blogs.oracle.com/mindless/entry/controlling_the_order_of_junit
Having tests depending on other tests 99.9% of the time a very bad idea. Unit tests should be independent from each other, as otherwise you might have a cascade of errors, or (even worse) one test failing because something another test did sometime before.
If you still want to go through this pain, you'll need to use a different unit testing framework (such as TestNG - see dependsOnMethods) which supports test dependencies.
Junit doesn't support this feature because it's seen by many as a bad practice (for very good reasons).
The next JUnit release will support ordering of test methods. The standard Maven Surefire Plugin supports ordering of test methods already.
Netbeans has good integration with ant build files. You could write a specific ant target that could execute each test in order.