In Visual Studio 2013, I like to run tests from Test Explorer by Trait as shown in the image below:
I'm now running these tests from a Visual Studio 2013 command prompt but only want to run for a specific trait and not sure how to do this. I can run all tests in a project with this command:
C:\Directory>MSTEST /testcontainer:Project.dll
but haven't been able to find a command to run by a specific trait only. Can anyone help with that command?
Traits represent the test category and you can filter that by using the /category:[test category filter] option
for example, assume I have a test method with the following TestCategory attribute
[TestCategory("IntegrationTest")]
To filter on that category the command would look like this
C:\Directory>MSTEST /testcontainer:Project.dll /category:IntegrationTest
Reference MSTest.exe command-line options : category
Related
I am unable to run vstest with a /TestCaseFilter that excludes certain tests by category. If I use the "Test Explorer" in Visual Studio 2017 and filter by "Traits" it is working correctly but ultimately I want to be able to edit the TFS build definition and exclude certain tests from running on the build server.
I used the example from the following to add a "TestCategory" attribute to certain tests:
http://www.ademiller.com/blogs/tech/2014/04/test-categories-for-visual-c/
I've tried various versions of the command line but cannot get vstest to recognize my "TestCategory" filter:
https://blogs.msdn.microsoft.com/vikramagrawal/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter/
https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md
https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019
vstest.console.exe /TestCaseFilter:"TestCategory=Slow" "D:\MyStuff\Project.Tests.dll" /Platform:x64 /InIsolation'
I get the following error:
No test matches the given testcase filter TestCategory=Slow in D:\MyStuff\Project.Tests.dll"
I have tried escape characters, different types of quotes, parentheses, but nothing works. If I set the filter to be "TestCategory!=Slow", all tests will execute, even the slow ones!
Any help is appreciated. Thanks.
It's been a little over two years but this is finally working.
In a header file, I declare the following:
#define EXPENSIVE_TEST_METHOD(name) \
BEGIN_TEST_METHOD_ATTRIBUTE(name) \
TEST_OWNER(L"Expensive") \
END_TEST_METHOD_ATTRIBUTE() \
TEST_METHOD(name)
I can then use EXPENSIVE_TEST_METHOD in place of TEST_METHOD in the unit test.
We're now using Azure DevOps Server 2019 so in the Visual Studio Test task, I can now add the following test filter on CI builds:
On command line, you can now do something like this:
vstest.console.exe "%MyTestProjectDll%" /Platform:x64 /InIsolation /TestCaseFilter:"Owner!=Expensive"
I can't use MSBuild to compile SSIS projects as it's not supported yet. Hence,
I'm trying to use VS2017 devenv.exe to compile SSIS projects in my TFS (2017) task via PowerShell. I don't see any errors but neither I see the compiled output from dtproj files i.e. dtsx or ispac.
However, if I run the same powershell command manually on the build agent machine then it works fine and produces the output files. This is the command I'm trying:
$VsIdeExe = 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\devenv.exe'
$BuildConfiguration = 'Development'
$PackageStagingLogsDirectory = "C:\temp\logs"
& "$VsIdeExe" "C:\temp\Source\SSIS\MyData.sln" /build "$BuildConfiguration" `
/log "$PackageStagingLogsDirectory\MyData.ide.log" `
/out "$PackageStagingLogsDirectory\MyData.sln.log"
What am I missing here?
I found an alternative and it works. So, instead of using PowerShell task, I used Command line task and it works like a charm! I don't know how the mechanics are different between these 2 tasks but my use case sufficed.
Thanks!
I'm looking for a solution to start my tests from command line.
I created a UnitTest Procjet in VisualStudio2017 for my .NET solution.
Added TestStack.White NuGet package to the project.
The test are running fluently when I start from the VisualStudio2017.
I would like to start it from Jenkins also. I think it is the easiest to do it from command line, so I add it to my pipeline configuration (Jenkinsfile)
stage('Run UI Tests') {
steps {
bat('"C:\\PATH_TO_MSTEST\\mstest" /testcontainer:PATH_TO_MY_TEST_PROJECT\\bin\\Debug\\MyTests.dll')
}
}
When I try to start it from cmd like I would do with with regular Unit Tests, it is not working.
It says:
Starting execution...
No tests to execute.
I build the project before I start 'Run UI Tests' stage.
Any ideas how to make it work? Could really find it on stackoverflow, github issues of TestStack nor other glory places on the web
Found a solution.
On my local developer machine it was working, the mstest version was 14
On the build agent machine the mstest version was 15, that was not working somehow (it had nothing to do with TestStack White, simply the unit tests were not working)
What I do is, calling vstest.console.exe instead of the mstest.
C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe
So, instead of
stage('Run UI Tests') {
steps {
bat('"C:\\PATH_TO_MSTEST\\mstest" /testcontainer:PATH_TO_MY_TEST_PROJECT\\bin\\Debug\\MyTests.dll')
}
}
My command in the Jenkinsfile was:
stage('Run UiTests') {
steps {
bat('"C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\TestAgent\\Common7\\IDE\\Extensions\\TestPlatform\\vstest.console.exe" PATH_TO_MY_TEST_PROJECT\\bin\\Debug\\MyTests.dll')
}
}
nunit3-console is great alternative to MSTEST. Please refer below link.
e.g.
nunit3-console \bin\Debug\Automation.dll --where "cat=Smoke-Tests"
https://github.com/nunit/docs/wiki/Console-Command-Line
I'm using tfs 2017 to run CodedUI ordered unit tests. These are my build steps:
These are my "Run Functional Tests" configurations:
And these are the "Publish Test Results" (I'm not sure they are correct):
The TestAgent is deployed and the tests are running fine, The problem is that the test results appear as only one result and I can't see detailed result for each test. This is how my tests results looks like (The attachments are screen shots I take for each test):
I can reproduce your situation and TFS will treat the tests as one in the log:
DistributedTests: Total Tests : 1, Passed Tests : 0
It's a know issue, please refer below link:
How to display test results of individual test in the ordered test
suite in TFS Web
This is a limitation of the run functional test task. You can
publish the .trx file using the "publish test results" task and it
will show you all the tests but you wont know which ".orderedtest"
they were associated with etc.
You need either open *.trx file in Visual Studio or use Publish Test Results task(need to check continue on error).
Besides also change the outcome from failed to all in the test result page.
Just installed Android Studio to give it a try. I'm trying to create a test module for my application and, whatever the method I use, I always get the following error: "Error: No AndroidManifest.xml file found in the main project directory...".
Methods I use:
1) from Android Studio: New > Module > Test Module
2) from command line: android create test-project -m "main_path" -n "project_name" -p "test_path"
Any ideas on how to solve this?
Edit: As of 0.1.8 this is now supported in the IDE.
According to the documentation for the new build system a separate project is no longer needed, which implies that a separate module may not be either. That said, getting test working within the IDE is still problematic. I've posted an answer here that at least works for running tests from the command line.