I am using MSTest and running unit tests on Team Foundation Server. I am doing wildcard test execution as here; also see image.
I want to use an arbitrary test settings file for one specific dll / test assembly (let's call it foo.dll) and not use any test settings file/use the default settings for every other dll that is grabbed by the wildcard. What is the best way to do that?
I have an image that might help illustrate:
(The above is an image of the screen at Process tab-->Basic-->Automated Tests-->Test Assembly, within the Team Foundation Build template.)
With the standard, out of the box process templates you can't do what you're after. You can't wildcard assemblies and exclude some at the same time, and you can't attach a test settings file for just some assemblies. The reason being that under the hood it's doing a single test run and the settings apply for the whole test run.
As an idea, you could copy and customize the build process template, and specifically alter the section relating to running tests to do what you need. Add a second test run to it for example so you can do one set of tests without the run settings and another run with them. For both test runs use the wildcard specification and then use the criteria/arguments tab to decide which tests to run. In the DLL you want to exclude from the first run, tag the tests with an attribute and ignore those in your first test run. For the second include just tests with that attribute.
Related
I'm a new NUnit user, using NUnit 3.9 under Visual Studio Community 2017. I'm using it on a pet open source library project, and it's going well once I got the hang of it.
The library accesses a publicly available government website via a documented API. Most of my tests use local data, so that I have a stable bed to compare against, and so that I can test without going out to the website every time.
I would like to set it up so that normally, the tests that hit the server do not run. I run the tests over and over as I tweak the code, and just as a matter of courtesy, don't want to bang on the server. Also, I'd like to be able to test even when the remote system is down or when I don't have Internet access.
Is there any way to group or tag my tests so that normally only the ones using local data run, but that I can still, when necessary, run the ones that exercise the server access? Either specifying "run these" or "exclude these" would be fine.
I've grouped the tests into two different classes, UnitTestOffline.cs and UnitTestOnline.cs, and was hoping I could somehow run the tests on a class-by-class basis, but haven't found a way to do that.
You'll get better answers if you say specifically how you run your tests, since there are a number of ways to do it. Since you mention VS2017, I'm going to assume that you are using the NUnit 3 VS Adapter, but let us know if you are using some other approach.
In the VS adapter, use the dropdown to display your tests by class. Right click on the class for which you want to run tests and run them.
If you decide to categorize tests using the CategoryAttribute, you can display tests by "trait" in Visual Studio. As before, right click on the group you want to run tests for and run them.
If you get a lot of tests, you might want to put your unit tests in one assembly and your integration tests in another. In that case, display the tests by project, right click on the project you want and run them.
All of this can also be done using the nunit3-console command-line runner as well. To select by class or category, you use the --where option. To select by assembly, you merely enter the name of the assembly you want on the command-line.
Seems like you want to categorize your tests (unit test, integration tests...) and run only the unit tests... you could use [Category] for that.
In the nunit GUI you could /include /exclude category after that and run only the one you want.
And probably that the filtering of Visual Studio could work.
Try to see one of the solution suggested here as well
I understand that I should use the sure-fire plugin for unit tests, and failsafe for integration. I can run unit tests with mvn test and integration tests with mvn verify but this annoys me for 2 reasons:
I'd prefer to be able to select any test class (or method in that class) and run it individually by a simple click, rather than typing it into terminal every time.
The terminal returns the test results in ugly black/white paragraphs, requiring me to sift through them. I'd much prefer to have the results returned in a visually organized manner, similar to if I right-clicked on the test class in IntelliJ and click 'RunDemoTest`. This produces:
I find the error results much easier to sift through, for example it shows red/green #Test results on the left, and on the right it cleanly organizes the error into
Expected : 3
Actual :1
I'm sure there are advantages to using terminal for automated test runs later into production, but during development I don't find the terminal conducive to my tinkering.
How do I benefit from IntelliJ's visual feedback of test results, while simultaneously ensuring unit & integration tests are run separately, and preserving my freedom to pick and choose which test classes and test methods I can run at any time?
I'm assuming I can't have my cake and eat it too. Please explain.
If you are using the IntelliJ view "Maven Projects" you can very easy toggle on/off the exection of maven integrated tests.
Via "Run/Debug Configurations" you can create test executions that match your reqirement for a comfortable UI.
After these steps, there is a new entry in the drop down list "Run/Debug configurations". When you start the new JUnit Test configuration, the defined tests are executed and the results are presented exactly in the same manner as the screenshot in your question.
The options in my second screenshot allow a very flexible definition of the scope. You don't have go to every java file and click on the green arrows in the editor view.
This configuration isn't related to any maven configuration, and you can use them at any time in your coding process.
When using WebStorms as a test runner every unit test is run. Is there a way to specify running only one test? Even only running one test file would be better than the current solution of running all of them at once. Is there a way to do this?
I'm using Mocha.
not currently possible, please vote for WEB-10067
You can double up the i on it of d on describe and the runner will run only that test/suite. If you prefix it with x it will exclude it.
There is a plugin called ddescribe that gives you a gui for this.
You can use the --grep <pattern> command-line option in the Extra Mocha options box on the Mocha "Run/Debug Configurations" screen. For example, my Extra Mocha options line says:
--timeout 5000 --grep findRow
All of your test *.js files, and the files they require, still get loaded, but the only tests that get run are the ones that match that pattern. So if the parts you don't want to execute are tests, this helps you a lot. If the slow parts of your process automatically get executed when your other modules get loaded with require, this won't solve that problem. You also need to go into the configuration options to change the every time you want to run tests matching a different pattern, but this is quick enough that it definitely saves me time vs. letting all my passing tests run every time I want to debug one failing test.
You can run the tests within a scope when you have a Mocha config setting by using .only either on the describe or on the it clauses
I had some problems getting it to work all the time, but when it went crazy and kept running all my tests and ignoring the .only or .skip I added to the extra mocha options the path to one of the files containing unit tests just like in the example for node setup and suddenly the .only feature started to work again regardless of the file the tests were situated in.
I know this has been asked many times but I want to be specific.
I use to use selenium. After googling it looks like I can run it via console and it gives me a bunch of text output but I rather not parse that and I want a pass/fail type of thing
Every once in a while I like to run all of my unit test on the UI not code. I don't want to submit a form with certain values, I want to see if I click this img does the dropbox beside it pop out and if i select a name will it be in the form which I'll then submit after running a few other things.
The reason I'd like this is certain features MUST ALWAYS be working so i'm ok with adjusting the unit test everytime I modify the UI for those feature. For the rest the unit test in code which checks business logic will be enough as those UIs are always changing or not very important.
It be nice if something can kickoff firefox and chrome (or webkit) but thats not required.
Like I said I'd like pass/fail, some kind of easy text to parse. Complex test is ok as I know regex but I don't want to figure out when one unit test ends or starts.
If you're using java/maven - I wrote a maven plugin for selenium that should do what you want:
https://github.com/willwarren/selenium-maven-plugin. You generate the tests in firefox + selenium, then save the files to a directory in your maven project.
If you're not using maven you can use the project that I built upon:
http://code.google.com/p/selenium4j
From the Readme:
We use selenium IDE to record our tests. We then saved the test cases into our project in the following fashion: (Note: currently the code from selenium4j only suports one level, so don't nest your folders)
./src/test/selenium
|-signin
|-LoginGoodPassword.html
|-LoginBadPassword.html
|-selenium4j.properties
We didn't save the test suites as maven takes care of finding your tests.
The selenium4j.properties contains setup information about:
# the web site being tested
webSite=http://yourwebapp:8080
# A comma separated values of the WebDrivers being used. Accepted drivers:
# HtmlUnitDriver, FirefoxDriver, ChromeDriver, InternetExplorerDriver
driver=FirefoxDriver
# How many times we want to iterate and test
loopCount=1
The selenium maven plugin, which is bound to the process-test-resources phase, then converts these html files into junit 4 tests in your src/test/java folder.
So you end up with:
./src/test/java
|-signin
|-firefox
|-LoginGoodPasswordTest.java
|-LoginBadPasswordTest.java
I have a problem with executing tests in JUnit. Imagine you have one test case class with f.e. 100 tests, no test suite and no main program - test case class test the device on com port. JUnit project is in Netbeans. I want to run tests - but not all of them at the same time, i would like to choose tests to run before actual testing.
Once I saw something like that in eclipse - but it wasn't my project and I don't know how it was done and how to do the same thing in netbeans. It was a separate window, poping up before running tests. In this window there were checkboxes with names of methods with #Test annotation and you could choose tests you wanted to run and click run - so it let you to run what you wanted.
Does anyone know how to do it in netbeans? Is it any library or plugin?
Any help will be appreciated.
You can take a look at Run single test from a JUnit class using command-line. It does allow you to specify what test you want to run given a class with multiple test cases in it. Being command-line you can then script your own test suite that runs the specific ones you want.
I also noticed your other question Junit: changing sequence of test running. With the scripting approach you can actually control the order of your testing.
This approach does not take advantage of Eclipse's or NetBean's JUnit test runners though, so it is a very specific workaround.
Netbeans nowadays support running single tests: