I'm a new NUnit user, using NUnit 3.9 under Visual Studio Community 2017. I'm using it on a pet open source library project, and it's going well once I got the hang of it.
The library accesses a publicly available government website via a documented API. Most of my tests use local data, so that I have a stable bed to compare against, and so that I can test without going out to the website every time.
I would like to set it up so that normally, the tests that hit the server do not run. I run the tests over and over as I tweak the code, and just as a matter of courtesy, don't want to bang on the server. Also, I'd like to be able to test even when the remote system is down or when I don't have Internet access.
Is there any way to group or tag my tests so that normally only the ones using local data run, but that I can still, when necessary, run the ones that exercise the server access? Either specifying "run these" or "exclude these" would be fine.
I've grouped the tests into two different classes, UnitTestOffline.cs and UnitTestOnline.cs, and was hoping I could somehow run the tests on a class-by-class basis, but haven't found a way to do that.
You'll get better answers if you say specifically how you run your tests, since there are a number of ways to do it. Since you mention VS2017, I'm going to assume that you are using the NUnit 3 VS Adapter, but let us know if you are using some other approach.
In the VS adapter, use the dropdown to display your tests by class. Right click on the class for which you want to run tests and run them.
If you decide to categorize tests using the CategoryAttribute, you can display tests by "trait" in Visual Studio. As before, right click on the group you want to run tests for and run them.
If you get a lot of tests, you might want to put your unit tests in one assembly and your integration tests in another. In that case, display the tests by project, right click on the project you want and run them.
All of this can also be done using the nunit3-console command-line runner as well. To select by class or category, you use the --where option. To select by assembly, you merely enter the name of the assembly you want on the command-line.
Seems like you want to categorize your tests (unit test, integration tests...) and run only the unit tests... you could use [Category] for that.
In the nunit GUI you could /include /exclude category after that and run only the one you want.
And probably that the filtering of Visual Studio could work.
Try to see one of the solution suggested here as well
Related
I'm having trouble finding documentation on how to do this. I have a Class Library project and a Unit Test Project. The Unit Test project tests units of functional code. Amazing, I know. But I also have a Console Application to perform special Integration Tests.
Granted, I could use some frameworks, or even another "Unit Test" project, where I'm not really writing Unit Tests, but hitting third party APIs and such. All I'd have to do is NAME the project "Integration Tests". That's all fine and well, but that's not this question.
My question is, how can I get VSTS to RUN my Console Application and retrieve the output as the Test Results. In running my Console Application, it will perform some integration testing, and output something (I don't care what) to indicate either failure(s), or success.
On Fail or Success, I want VSTS to act just as it would for a Unit Test run.
In response to comments for why: Because I like to push the limits of systems and understand all of my options. Also because I don't let Microsoft control me even though I use their products. So even though I don't plan to put this into production at this moment, I want to know the measure of friction of this solution so I can compare it against other frictions.
I'm trying to only run a certain subsection of xunit tests using the tfs Test Filter Criteria:
The problem is that this is not doing anything, it still runs every test regardless of it's name, what might I be missing here? Is there something else I need to be doing since I'm using Xunit? Or is there another method of excluding tests in the test unit step I can use.
Test filter criteria: Filters tests from within the test assembly files. For example, “Owner=james&Priority=1”. This option works the
same way as the console option /TestCaseFilter for vstest.console.exe
For more information, see
https://msdn.microsoft.com/en-us/library/jj155796.aspx
Please double check your property of FullyQualifiedName in your assembly files. I'm not sure if /TestCaseFilter is also support in xUnit. You could give a try with directly running the test using command line (vstest.console.exe). If it's not work, then this should also not work in TFS build task.
Add a related link talk about test filter for your reference: VSTS/TFS VISUAL STUDIO TEST TASK – FILTER CRITERIA
I understand that I should use the sure-fire plugin for unit tests, and failsafe for integration. I can run unit tests with mvn test and integration tests with mvn verify but this annoys me for 2 reasons:
I'd prefer to be able to select any test class (or method in that class) and run it individually by a simple click, rather than typing it into terminal every time.
The terminal returns the test results in ugly black/white paragraphs, requiring me to sift through them. I'd much prefer to have the results returned in a visually organized manner, similar to if I right-clicked on the test class in IntelliJ and click 'RunDemoTest`. This produces:
I find the error results much easier to sift through, for example it shows red/green #Test results on the left, and on the right it cleanly organizes the error into
Expected : 3
Actual :1
I'm sure there are advantages to using terminal for automated test runs later into production, but during development I don't find the terminal conducive to my tinkering.
How do I benefit from IntelliJ's visual feedback of test results, while simultaneously ensuring unit & integration tests are run separately, and preserving my freedom to pick and choose which test classes and test methods I can run at any time?
I'm assuming I can't have my cake and eat it too. Please explain.
If you are using the IntelliJ view "Maven Projects" you can very easy toggle on/off the exection of maven integrated tests.
Via "Run/Debug Configurations" you can create test executions that match your reqirement for a comfortable UI.
After these steps, there is a new entry in the drop down list "Run/Debug configurations". When you start the new JUnit Test configuration, the defined tests are executed and the results are presented exactly in the same manner as the screenshot in your question.
The options in my second screenshot allow a very flexible definition of the scope. You don't have go to every java file and click on the green arrows in the editor view.
This configuration isn't related to any maven configuration, and you can use them at any time in your coding process.
Our team has Visual Studio 2012 Professional licenses (not Test Professional). We are developing a smallish web application, and we have both true unit tests which mock everything needed, and tests for the data layer. Each class of data layer tests creates the whole database from scratch and fills it with a prepared set of test data, so running them takes a long time. As a result, we are reluctant to do a "run all", and our unit tests (which are quick) are only used rarely.
We are looking for a low-friction solution which will allow us to run all quick tests with 2-3 clicks (similar to the existing Run all) frequently, and easily run all tests when needed.
We tried making a playlist of the quick tests only. But we are done with programming the data layer, so practically all new tests we write are quick tests, and adding each of them to the playlist is annoying and somewhat error-prone. We would prefer an approach where we somehow mark the tests we do not want in a "quick run" as excluded, and it automatically runs all other tests in the solution. Note that we don't want to permanently add an Ignore attribute to the slow tests, as we still want to run them at least once daily.
You could use the Traits feature in mstest to accomplish this. Take a look at this blog post:
https://devblogs.microsoft.com/devops/how-to-manage-unit-tests-in-visual-studio-2012-update-1-part-1using-traits-in-the-unit-test-explorer/
I've got a bunch of unit tests built using Visual Studio 2005's built-in unit testing functionality. For the last little while, it's been taking absolutely forever to start the tests... Everything just sits there at "Pending" for two minutes or more. Now Visual Studio's decided to take things to a new level and never even start the tests. After two to three minutes, it aborts the run and barfs an error message into the log:
Failed to Queue Test Run '(blahblahblah)' with id {bfba05b1-afe5-499e-b452-29167f414f0f}: Microsoft.VisualStudio.TestTools.TestManagement.ExecutionException: Failed to establish communication environment for local run.
Anyone have any ideas? In the medium term, we are planning to switch to NUnit, but for now I'd prefer to stick to Visual Studio because the rest of the team already has that installed and that makes it easier to convince them to actually run the tests... ;-)
Is your computer name UPPERCASE? change it to lower case and try it again.
I've experienced this problem. However, for me the answer was to change my machine name from all lowercase to all UPPERCASE.
Some references for this...
http://teamfoundation.blogspot.com/2008/12/case-of-never-ending-unit-tests.html
http://social.msdn.microsoft.com/Forums/en-US/vststest/thread/fd6f2128-e248-4336-b8be-1eb5480e3de8/
Note, that if you're just changing the case of the machine name then you'll need to use the registry method of changing name as the dialog box will grey-out the OK button when it sees what looks like the same name.
Let me say this MS TEST is not a professional product and should be avoided whenever possible. If you want to use a good testing framework then use MBUnit with TestDriven.NET. MBUnit comes with many new attributes including RollBack and RowTest , Row.
Also, TestDriven.NET will allow you to debug your unit tests (How cool is that). You can also run the code coverage from TestDriven.NET which shows you how much of your code is covered under tests.
Give it a try I am sure you like it.
PS: IT is FREEE