This question already has answers here:
Using Post-Build Event To Execute Unit Tests With MS Test in .NET 2.0+
(3 answers)
Closed 8 years ago.
i use VS2013. I have a "normal project" and a testproject with some testcases to the "normal project". I can run the tests in test project - so far so good. But now, I want to run the tests automatically, when I rebuild my "normal project". Any ideas?
Thanks.
As was pointed out in the comments to my other answer, it is possible to add a postbuild step which will invoke the tests from the command line. The details can be found in this question.
Whilst this might be a suitable solution, I believe it is inferior to the continuous test runner approach for several reasons.
feedback is not integrated nice into the IDE in a test runner window, allowing for simple visual feed back and/or running of individual or groups of tests
Tests only run at build time. Continuous runners will run when changes are detected
All tests will be run on every save. Continuous runners can identify only the tests that have been affected by the change and run only those tests.
running individual tests will require using a different runner
running specified subsets of tests (ie excluding integration tests) will require changing the postbuild step. NCrunch can be configured to run filtered sets of tests.
Having said all that some of the runners are commercial products that are not cheap, (although some are Open Source and free). In the end you pays your money, you takes your choice.
I don't know of anyway to do this in natively in VS but you should look into a tool like NCrunch, a continuous test runner, which will run your tests not only when you build your 'normal' project but also when you make any change to the project.
There are other continuous test runners out there, like MightyMoose, the source of which is here I believe
I'm in no way affiliated with NCrunch (or mighty moose), but I think its awesome and can't imagine life without it now.
Its not free though so whether its useful for you I can't say.
Related
I'm a new NUnit user, using NUnit 3.9 under Visual Studio Community 2017. I'm using it on a pet open source library project, and it's going well once I got the hang of it.
The library accesses a publicly available government website via a documented API. Most of my tests use local data, so that I have a stable bed to compare against, and so that I can test without going out to the website every time.
I would like to set it up so that normally, the tests that hit the server do not run. I run the tests over and over as I tweak the code, and just as a matter of courtesy, don't want to bang on the server. Also, I'd like to be able to test even when the remote system is down or when I don't have Internet access.
Is there any way to group or tag my tests so that normally only the ones using local data run, but that I can still, when necessary, run the ones that exercise the server access? Either specifying "run these" or "exclude these" would be fine.
I've grouped the tests into two different classes, UnitTestOffline.cs and UnitTestOnline.cs, and was hoping I could somehow run the tests on a class-by-class basis, but haven't found a way to do that.
You'll get better answers if you say specifically how you run your tests, since there are a number of ways to do it. Since you mention VS2017, I'm going to assume that you are using the NUnit 3 VS Adapter, but let us know if you are using some other approach.
In the VS adapter, use the dropdown to display your tests by class. Right click on the class for which you want to run tests and run them.
If you decide to categorize tests using the CategoryAttribute, you can display tests by "trait" in Visual Studio. As before, right click on the group you want to run tests for and run them.
If you get a lot of tests, you might want to put your unit tests in one assembly and your integration tests in another. In that case, display the tests by project, right click on the project you want and run them.
All of this can also be done using the nunit3-console command-line runner as well. To select by class or category, you use the --where option. To select by assembly, you merely enter the name of the assembly you want on the command-line.
Seems like you want to categorize your tests (unit test, integration tests...) and run only the unit tests... you could use [Category] for that.
In the nunit GUI you could /include /exclude category after that and run only the one you want.
And probably that the filtering of Visual Studio could work.
Try to see one of the solution suggested here as well
I understand that I should use the sure-fire plugin for unit tests, and failsafe for integration. I can run unit tests with mvn test and integration tests with mvn verify but this annoys me for 2 reasons:
I'd prefer to be able to select any test class (or method in that class) and run it individually by a simple click, rather than typing it into terminal every time.
The terminal returns the test results in ugly black/white paragraphs, requiring me to sift through them. I'd much prefer to have the results returned in a visually organized manner, similar to if I right-clicked on the test class in IntelliJ and click 'RunDemoTest`. This produces:
I find the error results much easier to sift through, for example it shows red/green #Test results on the left, and on the right it cleanly organizes the error into
Expected : 3
Actual :1
I'm sure there are advantages to using terminal for automated test runs later into production, but during development I don't find the terminal conducive to my tinkering.
How do I benefit from IntelliJ's visual feedback of test results, while simultaneously ensuring unit & integration tests are run separately, and preserving my freedom to pick and choose which test classes and test methods I can run at any time?
I'm assuming I can't have my cake and eat it too. Please explain.
If you are using the IntelliJ view "Maven Projects" you can very easy toggle on/off the exection of maven integrated tests.
Via "Run/Debug Configurations" you can create test executions that match your reqirement for a comfortable UI.
After these steps, there is a new entry in the drop down list "Run/Debug configurations". When you start the new JUnit Test configuration, the defined tests are executed and the results are presented exactly in the same manner as the screenshot in your question.
The options in my second screenshot allow a very flexible definition of the scope. You don't have go to every java file and click on the green arrows in the editor view.
This configuration isn't related to any maven configuration, and you can use them at any time in your coding process.
Our team has Visual Studio 2012 Professional licenses (not Test Professional). We are developing a smallish web application, and we have both true unit tests which mock everything needed, and tests for the data layer. Each class of data layer tests creates the whole database from scratch and fills it with a prepared set of test data, so running them takes a long time. As a result, we are reluctant to do a "run all", and our unit tests (which are quick) are only used rarely.
We are looking for a low-friction solution which will allow us to run all quick tests with 2-3 clicks (similar to the existing Run all) frequently, and easily run all tests when needed.
We tried making a playlist of the quick tests only. But we are done with programming the data layer, so practically all new tests we write are quick tests, and adding each of them to the playlist is annoying and somewhat error-prone. We would prefer an approach where we somehow mark the tests we do not want in a "quick run" as excluded, and it automatically runs all other tests in the solution. Note that we don't want to permanently add an Ignore attribute to the slow tests, as we still want to run them at least once daily.
You could use the Traits feature in mstest to accomplish this. Take a look at this blog post:
https://devblogs.microsoft.com/devops/how-to-manage-unit-tests-in-visual-studio-2012-update-1-part-1using-traits-in-the-unit-test-explorer/
Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.
Well, Maven is too good, when talking about speed.
But I want something that is more acceptable.
Imagine I wrote a test org.fun.AbcTestCase
In such TestCase, I include some JUnit / TestNG tests
Now I want to run only this test case, said org.fun.AbcTestCase from command line.
How can I do that?
I know it's easy to do it within Eclipse or IDEA. However, I am learning Scala and IDE support is currently terrible, especially when it comes to run unit test.
Here is why I find it difficult:
The project would involve many dependencies. When I test my project as a Maven goal, surefire takes care of that. Mimic that with reasonable manual effort is important.
The test process need to be fast enough with real time compiler (well, recompile the whole bunch of scala code is a terrible night mare).
Use the test parameter in the surefire:test mojo
mvn test -Dtest=MyTest
will run only the test MyTest.class, recompiling only if necessary (if changes are found).
If you are free to switch (as I imagine you might be if you have a toy project you're using to learn Scala) you might consider using SBT instead of Maven. Its IDE integration is only rudimentary, but it is quite handy for running tests (it can watch the source tree and re-run tests when you save, or even just the tests that failed during the last run.) Check out the website at http://simple-build-tool.googlecode.com/ .