net core 7, dotnet test : continue after one test failed - unit-testing

I am using .net core 7.
From command line, dotnet test (I am using nunit) stops when a test failed.
Is there a way to NOT stop and continue with following tests ?
Thx !

Could you please provide some code to better illustrate your issue?
Typically dotnet test runs all tests and provides a report on which pass and which fail.
For example, on this minimal reproduction:
public class Tests
{
[Test]
public void FailingTest1()
{
Assert.Fail();
}
[Test]
public void PassingTest()
{
Assert.Pass();
}
[Test]
public void FailingTest2()
{
Assert.Fail();
}
}
When I run dotnet test, I see:
PS C:\Users\SeanK\Repositories\2023_01_NunitRepro> dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
Repro -> C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\bin\Debug\net7.0\Repro.dll
Test run for C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\bin\Debug\net7.0\Repro.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed FailingTest1 [27 ms]
Stack Trace:
at Repro.Tests.FailingTest1() in C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\UnitTest1.cs:line 8
1) at Repro.Tests.FailingTest1() in C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\UnitTest1.cs:line 8
Failed FailingTest2 [< 1 ms]
Stack Trace:
at Repro.Tests.FailingTest2() in C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\UnitTest1.cs:line 20
1) at Repro.Tests.FailingTest2() in C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\UnitTest1.cs:line 20
Failed! - Failed: 2, Passed: 1, Skipped: 0, Total: 3, Duration: 30 ms - Repro.dll (net7.0)

Related

How do I use dotnet test to run tests from multiple libraries with a single pass/fail summary

How do I run unit tests with dotnet test if I have multiple test libraries in a code base?
I can run dotnet test, and it will find and run all tests even across multiple libraries, but it runs and reports each test library run independently:
$ dotnet test
Test run for C:\Users\mark\Documents\Redacted.Test\bin\Debug\netcoreapp2.1\Redacted.Test.dll(.NETCoreApp,Version=v2.1)
Test run for C:\Users\mark\Documents\Redacted\Redacted.SqlAccess.Test\bin\Debug\netcoreapp2.1\Redacted.SqlAccess.Test.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 16.2.0-preview-20190606-02
Copyright (c) Microsoft Corporation. All rights reserved.
Microsoft (R) Test Execution Command Line Tool Version 16.2.0-preview-20190606-02
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Starting test execution, please wait...
Test Run Successful.
Total tests: 59
Passed: 59
Total time: 3.1779 Seconds
Test run for C:\Users\mark\Documents\Redacted\Redacted.RestApi.Tests\bin\Debug\netcoreapp2.1\Redacted.RestApi.Tests.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 16.2.0-preview-20190606-02
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Test Run Successful.
Total tests: 99
Passed: 99
Total time: 9.8155 Seconds
Test Run Successful.
Total tests: 25
Passed: 25
Total time: 21.2894 Seconds
In this example, there's two test libraries, so I get two test result outputs.
This may work OK if the code has already been compiled, but in a clean build, there's going to be a lot of output from the compiler. This can easily cause one of the test run summaries to scroll past the visible part of the screen.
That's a problem if that test run fails.
How can I collapse all the unit tests to a single pass/fail summary?
On .NET 4.x, I could, for instance, use xUnit.net's console runner to run all test libraries as a single suite:
$ ./packages/xunit.runner.console.2.4.0/tools/net461/xunit.console BookingApi.UnitTests/bin/Debug/Ploeh.Samples.Booking
Api.UnitTests.dll BookingApi.SqlTests/bin/Debug/Ploeh.Samples.BookingApi.SqlTests.dll
xUnit.net Console Runner v2.4.0 (64-bit Desktop .NET 4.6.1, runtime: 4.0.30319.42000)
Discovering: Ploeh.Samples.BookingApi.UnitTests
Discovered: Ploeh.Samples.BookingApi.UnitTests
Starting: Ploeh.Samples.BookingApi.UnitTests
Finished: Ploeh.Samples.BookingApi.UnitTests
Discovering: Ploeh.Samples.BookingApi.SqlTests
Discovered: Ploeh.Samples.BookingApi.SqlTests
Starting: Ploeh.Samples.BookingApi.SqlTests
Finished: Ploeh.Samples.BookingApi.SqlTests
=== TEST EXECUTION SUMMARY ===
Ploeh.Samples.BookingApi.SqlTests Total: 3, Errors: 0, Failed: 0, Skipped: 0, Time: 3.816s
Ploeh.Samples.BookingApi.UnitTests Total: 7, Errors: 0, Failed: 0, Skipped: 0, Time: 0.295s
-- - - - ------
GRAND TOTAL: 10 0 0 0 4.111s (5.565s)
Notice how this produces a single summary at the bottom of the screen, so that I can immediately see if my tests passed or failed.
Use dotnet vstest to run multiple assemblies.
PS> dotnet vstest --help
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation. All rights reserved.
Usage: vstest.console.exe [Arguments] [Options] [[--] <RunSettings arguments>...]]
Description: Runs tests from the specified files.
Arguments:
[TestFileNames]
Run tests from the specified files. Separate multiple test file names
by spaces.
Examples: mytestproject.dll
mytestproject.dll myothertestproject.exe
...
Note that this method requires that you point at compiled assemblies (as opposed to dotnet test, which wants you to point at project files, and will optionally build things first for you).

Azure DevOps Server: Why does Test Impact Analysis identify my tests both as "impacted" and as "not impacted"?

I have a .net framework (v4.8) class library project that contains 18 xUnit.net unit tests that target another project.
In my Azure DevOps Server (ADOS) pipeline I have a rather standard Visual Studio Test task that find the test assembly with the unit tests and runs them.
When I enable the "Run only impacted tests" feature on the pipeline task, ADOS runs all of them on the first build, which is normal as this becomes the baseline build.
On the second build of the pipeline, things get weird. ADOS shows the build as having 36 unit tests, of which 18 are "not impacted" (and skipped = normal) and 18 were executed and "passed".
My question is - Why are my unit tests executed when they've been recognized as "not impacted"? How can I fix this?
PS - I'm attaching a screenshot and the task log below and I've noticed two interesting things:
Looks like the test selection phase, at the very start, happened twice.
This error message appears 18 times:
TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : ... to test run.
Googling for it yields no useful information
Screenshot:
Visual Studio Test task log:
2019-07-31T13:25:33.3027955Z ##[section]Starting: Redacted tests
2019-07-31T13:25:33.3042189Z ==============================================================================
2019-07-31T13:25:33.3042843Z Task : Visual Studio Test
2019-07-31T13:25:33.3043811Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2019-07-31T13:25:33.3044138Z Version : 2.143.8
2019-07-31T13:25:33.3044323Z Author : Microsoft Corporation
2019-07-31T13:25:33.3044558Z Help : [More information](https://go.microsoft.com/fwlink/?LinkId=835764)
2019-07-31T13:25:33.3044783Z ==============================================================================
2019-07-31T13:25:35.5490506Z SystemVssConnection exists true
2019-07-31T13:25:35.5492434Z SystemVssConnection exists true
2019-07-31T13:25:35.6518950Z Running tests using vstest.console.exe runner.
2019-07-31T13:25:35.6519796Z ======================================================
2019-07-31T13:25:35.6520669Z Test selector : Test assemblies
2019-07-31T13:25:35.6521225Z Test filter criteria : null
2019-07-31T13:25:35.6521656Z Search folder : C:\_work\169\s
2019-07-31T13:25:35.6523219Z VisualStudio version selected for test execution : latest
2019-07-31T13:25:35.6888410Z Run in parallel : false
2019-07-31T13:25:35.6933473Z Run in isolation : false
2019-07-31T13:25:35.6934221Z Path to custom adapters : null
2019-07-31T13:25:35.6953986Z Other console options : null
2019-07-31T13:25:35.6954775Z Code coverage enabled : false
2019-07-31T13:25:35.6958141Z Diagnostics enabled : false
2019-07-31T13:25:35.7006840Z SystemVssConnection exists true
2019-07-31T13:25:35.7068611Z Run the tests locally using vstest.console.exe
2019-07-31T13:25:35.7069443Z ========================================================
2019-07-31T13:25:35.7090912Z Test selector : Test assemblies
2019-07-31T13:25:35.7101985Z Test assemblies : **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**
2019-07-31T13:25:35.7102753Z Test filter criteria : null
2019-07-31T13:25:35.7114572Z Search folder : C:\_work\169\s
2019-07-31T13:25:35.7121444Z Run settings file : C:\_work\169\s
2019-07-31T13:25:35.7129106Z Run in parallel : false
2019-07-31T13:25:35.7135940Z Run in isolation : false
2019-07-31T13:25:35.7185224Z Path to custom adapters : null
2019-07-31T13:25:35.7186782Z Other console options : null
2019-07-31T13:25:35.7193511Z Code coverage enabled : false
2019-07-31T13:25:35.7195285Z Diagnostics enabled : false
2019-07-31T13:25:35.7202436Z Rerun failed tests: false
2019-07-31T13:25:35.7213303Z VisualStudio version selected for test execution : latest
2019-07-31T13:25:35.9796440Z ========================================================
2019-07-31T13:25:48.9123895Z ======================================================
2019-07-31T13:25:49.1259524Z [command]C:\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.143.8\TestSelector\TestSelector.exe PublishCodeChanges
2019-07-31T13:25:51.5460715Z Is pull request flow: False
2019-07-31T13:25:51.5461897Z
2019-07-31T13:25:51.5462363Z ***************************** Section 'Get Files that were changed' Starting *****************************
2019-07-31T13:25:51.5462753Z Baseline build: http://redacted/tfs/Redacted/390eb9aa-e420-4922-8ef4-bb12632dda52/_build/results?buildId=26442
2019-07-31T13:25:51.5463129Z Baseline build id = 26442
2019-07-31T13:25:51.5463438Z Total files changed = 0
2019-07-31T13:25:51.6444164Z ***************************** Section 'Get Files that were changed' Ended ********************************
2019-07-31T13:25:51.6445042Z
2019-07-31T13:26:04.3435720Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" C:\_work\169\s\Redacted\Redacted.Tests\bin\Release\Redacted.Tests.dll /ListFullyQualifiedTests /ListTestsTargetPath:C:\_work\_temp\b76cadd0-b396-11e9-84b4-7144b5b4b30f.txt "/TestAdapterPath:\"C:\_work\169\s\""
2019-07-31T13:26:07.4913814Z Microsoft (R) Test Execution Command Line Tool Version 15.9.0
2019-07-31T13:26:07.4915501Z Copyright (c) Microsoft Corporation. All rights reserved.
2019-07-31T13:26:07.4916535Z
2019-07-31T13:26:07.4917502Z [xUnit.net 00:00:00.5867523] Discovering: Redacted.Tests
2019-07-31T13:26:07.4918503Z [xUnit.net 00:00:00.5867523] Discovering: Redacted.Tests
2019-07-31T13:26:07.4919413Z [xUnit.net 00:00:00.7084444] Discovered: Redacted.Tests
2019-07-31T13:26:07.4920310Z [xUnit.net 00:00:00.7084444] Discovered: Redacted.Tests
2019-07-31T13:26:07.4921380Z Microsoft (R) Test Execution Command Line Tool Version 15.9.0
2019-07-31T13:26:07.4922173Z Copyright (c) Microsoft Corporation. All rights reserved.
2019-07-31T13:26:07.4923076Z
2019-07-31T13:26:07.4924003Z [xUnit.net 00:00:00.5867523] Discovering: Redacted.Tests
2019-07-31T13:26:07.4924894Z [xUnit.net 00:00:00.5867523] Discovering: Redacted.Tests
2019-07-31T13:26:07.4925884Z [xUnit.net 00:00:00.7084444] Discovered: Redacted.Tests
2019-07-31T13:26:07.4926792Z [xUnit.net 00:00:00.7084444] Discovered: Redacted.Tests
2019-07-31T13:26:07.4927362Z
2019-07-31T13:26:07.4973829Z [command]C:\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.143.8\TestSelector\TestSelector.exe GetImpactedtests
2019-07-31T13:26:09.4002997Z
2019-07-31T13:26:09.4004545Z ***************************** Section 'Get Impacted Tests' Starting *****************************
2019-07-31T13:26:09.4005365Z Following tests are impacted.
2019-07-31T13:26:09.4005940Z Redacted.Redacted.Tests.RedactedTests.Test1
2019-07-31T13:26:09.4006587Z Redacted.Redacted.Tests.RedactedTests.Test2
2019-07-31T13:26:09.4007287Z Redacted.Redacted.Tests.RedactedTests.Test3
2019-07-31T13:26:09.4007967Z Redacted.Redacted.Tests.RedactedTests.Test4
2019-07-31T13:26:09.4008425Z Redacted.Redacted.Tests.RedactedTests.Test5
2019-07-31T13:26:09.4008946Z Redacted.Redacted.Tests.RedactedTests.Test6
2019-07-31T13:26:09.4009492Z Redacted.Redacted.Tests.RedactedTests.Test7
2019-07-31T13:26:09.4010065Z Redacted.Redacted.Tests.RedactedTests.Test8
2019-07-31T13:26:09.4010533Z Redacted.Redacted.Tests.RedactedTests.Test9
2019-07-31T13:26:09.4011052Z Redacted.Redacted.Tests.RedactedTests.Test10
2019-07-31T13:26:09.4011524Z Redacted.Redacted.Tests.RedactedTests.Test11
2019-07-31T13:26:09.4013893Z Redacted.Redacted.Tests.RedactedTests.Test12
2019-07-31T13:26:09.4015028Z Redacted.Redacted.Tests.RedactedTests.Test13
2019-07-31T13:26:09.4015617Z Redacted.Redacted.Tests.RedactedTests.Test14
2019-07-31T13:26:09.4016093Z Redacted.Redacted.Tests.RedactedTests.Test15
2019-07-31T13:26:09.4016626Z Redacted.Redacted.Tests.RedactedTests.Test16
2019-07-31T13:26:09.4017177Z Redacted.Redacted.Tests.RedactedTests.Test17
2019-07-31T13:26:09.4017718Z Redacted.Redacted.Tests.RedactedTests.Test18
2019-07-31T13:26:09.5412388Z ***************************** Section 'Get Impacted Tests' Ended ********************************
2019-07-31T13:26:09.5413181Z
2019-07-31T13:26:36.7905770Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" #C:\_work\_temp\adfd8492-b396-11e9-84b4-7144b5b4b30f.txt
2019-07-31T13:26:37.3869927Z Microsoft (R) Test Execution Command Line Tool Version 15.9.0
2019-07-31T13:26:37.3870828Z Copyright (c) Microsoft Corporation. All rights reserved.
2019-07-31T13:26:37.3871108Z
2019-07-31T13:26:37.3872406Z vstest.console.exe /Tests:"Redacted.Redacted.Tests.RedactedTests.Test1,Redacted.Redacted.Tests.RedactedTests.Test2,Redacted.Redacted.Tests.RedactedTests.Test3,Redacted.Redacted.Tests.RedactedTests.Test4,Redacted.Redacted.Tests.RedactedTests.Test5,Redacted.Redacted.Tests.RedactedTests.Test6,Redacted.Redacted.Tests.RedactedTests.Test7,Redacted.Redacted.Tests.RedactedTests.Test8,Redacted.Redacted.Tests.RedactedTests.Test9,Redacted.Redacted.Tests.RedactedTests.Test10,Redacted.Redacted.Tests.RedactedTests.Test11,Redacted.Redacted.Tests.RedactedTests.Test12,Redacted.Redacted.Tests.RedactedTests.Test13,Redacted.Redacted.Tests.RedactedTests.Test14,Redacted.Redacted.Tests.RedactedTests.Test15,Redacted.Redacted.Tests.RedactedTests.Test16,Redacted.Redacted.Tests.RedactedTests.Test17,Redacted.Redacted.Tests.RedactedTests.Test18"
2019-07-31T13:26:37.3874590Z "C:\_work\169\s\Redacted\Redacted.Tests\bin\Release\Redacted.Tests.dll"
2019-07-31T13:26:37.3874969Z /Settings:"C:\_work\_temp\b5e34eb0-b396-11e9-84b4-7144b5b4b30f.runsettings"
2019-07-31T13:26:37.3875446Z /logger:"trx"
2019-07-31T13:26:37.3875824Z /TestAdapterPath:"C:\_work\169\s"
2019-07-31T13:26:37.6710107Z Starting test discovery, please wait...
2019-07-31T13:26:39.6038466Z [xUnit.net 00:00:00.4368142] Discovering: Redacted.Tests
2019-07-31T13:26:39.6040363Z [xUnit.net 00:00:00.4368142] Discovering: Redacted.Tests
2019-07-31T13:26:39.7546522Z [xUnit.net 00:00:00.5937753] Discovered: Redacted.Tests
2019-07-31T13:26:39.7547684Z [xUnit.net 00:00:00.5937753] Discovered: Redacted.Tests
2019-07-31T13:26:43.7883423Z 3.1075
2019-07-31T13:26:50.1977477Z [xUnit.net 00:00:02.0480239] Starting: Redacted.Tests
2019-07-31T13:26:53.3389390Z 718.1859
2019-07-31T13:26:53.5472252Z Passed Test1
2019-07-31T13:26:53.5603376Z 56.5764
2019-07-31T13:26:53.6326306Z 54.9699
2019-07-31T13:26:53.7154878Z 64.2715
2019-07-31T13:26:53.8069205Z 73.892
2019-07-31T13:26:53.9014648Z 68.4937
2019-07-31T13:26:54.0023435Z 80.3846
2019-07-31T13:26:54.1314930Z 100.5241
2019-07-31T13:26:54.2364680Z 80.3842
2019-07-31T13:26:54.3291170Z 69.7939
2019-07-31T13:26:54.3428291Z Passed Test2
2019-07-31T13:26:54.3429229Z Passed Test3
2019-07-31T13:26:54.3429629Z Passed Test4
2019-07-31T13:26:54.3431016Z Passed Test5
2019-07-31T13:26:54.3431880Z Passed Test6
2019-07-31T13:26:54.3433438Z Passed Test7
2019-07-31T13:26:54.3434523Z Passed Test8
2019-07-31T13:26:54.3435127Z Passed Test9
2019-07-31T13:26:54.3435892Z Passed Test10
2019-07-31T13:26:54.4130603Z 59.9983
2019-07-31T13:26:54.4911591Z 60.3922
2019-07-31T13:26:54.5695728Z 54.5636
2019-07-31T13:26:54.6410513Z 55.6404
2019-07-31T13:26:54.7216412Z 62.6562
2019-07-31T13:26:54.7934494Z 53.8405
2019-07-31T13:26:54.9052845Z 92.4184
2019-07-31T13:26:55.0215033Z 90.1679
2019-07-31T13:26:55.0358313Z [xUnit.net 00:00:06.9133992] Finished: Redacted.Tests
2019-07-31T13:26:55.6290009Z Passed Test11
2019-07-31T13:26:55.6291164Z Passed Test12
2019-07-31T13:26:55.6292127Z Passed Test13
2019-07-31T13:26:55.6292638Z Passed Test14
2019-07-31T13:26:55.6292965Z Passed Test15
2019-07-31T13:26:55.6293222Z Passed Test16
2019-07-31T13:26:55.6293491Z Passed Test17
2019-07-31T13:26:55.6293753Z Passed Test18
2019-07-31T13:26:55.7055753Z Results File: C:\_work\169\s\TestResults\mossadmin_SRVSP2013D23_2019-07-31_16_26_53.trx
2019-07-31T13:26:55.7098705Z
2019-07-31T13:26:55.7103676Z Total tests: 18. Passed: 18. Failed: 0. Skipped: 0.
2019-07-31T13:26:55.7105019Z Test Run Successful.
2019-07-31T13:26:55.7126127Z Test execution time: 11.7080 Seconds
2019-07-31T13:26:55.8276786Z [command]C:\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.143.8\TestSelector\TestSelector.exe UpdateTestResults
2019-07-31T13:26:59.0886259Z
2019-07-31T13:26:59.0887559Z ***************************** Section 'Update test results' Starting *****************************
2019-07-31T13:26:59.0888142Z Test results remaining to be updated: 18
2019-07-31T13:26:59.0888434Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test1 to test run.
2019-07-31T13:26:59.0889274Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test2 to test run.
2019-07-31T13:26:59.0890060Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test3 to test run.
2019-07-31T13:26:59.0890582Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test4 to test run.
2019-07-31T13:26:59.0891197Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test5 to test run.
2019-07-31T13:26:59.0891663Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test6 to test run.
2019-07-31T13:26:59.0894448Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test7 to test run.
2019-07-31T13:26:59.0895370Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test8 to test run.
2019-07-31T13:26:59.0895890Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test9 to test run.
2019-07-31T13:26:59.0896359Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test10 to test run.
2019-07-31T13:26:59.0896879Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test11 to test run.
2019-07-31T13:26:59.0897955Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test12 to test run.
2019-07-31T13:26:59.0898632Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test13 to test run.
2019-07-31T13:26:59.0899585Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test14 to test run.
2019-07-31T13:26:59.0900632Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test15 to test run.
2019-07-31T13:26:59.0901513Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test16 to test run.
2019-07-31T13:26:59.0902139Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test17 to test run.
2019-07-31T13:26:59.0902833Z TcmTestResultPublisher.ConvertResults : No matching tcm test case result found for test case result given by agent to publisher. Adding test case : Redacted.Redacted.Tests.RedactedTests.Test18 to test run.
2019-07-31T13:26:59.0903437Z UnMatched Results found. Adding them to test run.
2019-07-31T13:26:59.0904228Z Publishing test results to test run '2490'.
2019-07-31T13:26:59.0905286Z TestResults To Publish 18, Test run id:2490
2019-07-31T13:26:59.0905702Z Test results publishing 18, remaining: 0. Test run id: 2490
2019-07-31T13:26:59.0906262Z Publishing test results to test run '2490'.
2019-07-31T13:26:59.0906568Z TestResults To Publish 18, Test run id:2490
2019-07-31T13:26:59.0907212Z Test results publishing 18, remaining: 0. Test run id: 2490
2019-07-31T13:26:59.0907563Z Total not impacted test: 18
2019-07-31T13:26:59.1801393Z ***************************** Section 'Update test results' Ended ********************************
2019-07-31T13:26:59.1802118Z
2019-07-31T13:26:59.2962424Z ##[section]Finishing: Redacted tests
You probably were running data driven tests, TIA(Test Impact Analysis) does not support data driven tests yet.
Please check https://devblogs.microsoft.com/devops/accelerated-continuous-testing-with-test-impact-analysis-part-1/ for currently supported scope.
If your tests are not data driven, Could you share some sample test code for more investigation?

How to Add Better Logging to Debug when a Previously Working VSTS Build Times out When Executing Unit Tests step

I have a VSTS build with Unit tests that was working previously in VSTS build. This week, it stopped working due to timing out.
The failure occurs due to timing out after 60 minutes.
Here's what we know:
The test library only contains 1 test. We run the test by pointing to the test assembly in the Unit Tests configuration of VSTS.
The test still works locally (runs in <1 second).
Even if I remove the actual test and replace it with a single trivial test that just checks Assert.IsTrue(true), the test still time out. So we conclude that it's something about the test configuration, rather than the test itself, causing the issue.
I tried adding additional Debug logging to the test, but it appears that the test execution never actually starts.
No trx log file appears to be generated. It doesn't appear to actually attempt to run the test at all.
We have tried this in a VSTS hosted VM and on our own VM with the same result.
We tried adding a new test step to the build and pointing that the the test DLL with no other modifications to the defaults, and it still gets stuck.
We pointed the original test step to a different test DLL, and the tests pass! So there is something related to this particular lib that is different (but again, even if I remove all the tests it still gets stuck).
Does anyone have alternate suggestions for other things to try to narrow down what could be causing this (e.g., trace logging in VSTS)? The log file below shows some of the specifics of our unit test configuration, but please let me know if other details would be helpful.
The failure log looks like this:
> ============================================================================== 2018-07-18T20:14:25.1963123Z Run the tests locally using
> vstest.console.exe 2018-07-18T20:14:25.1964566Z
> ======================================================== 2018-07-18T20:14:25.1965645Z Test selector : Test assemblies
> 2018-07-18T20:14:25.1966925Z Test assemblies : **\<PathToTestLib>.dll
> 2018-07-18T20:14:25.1968187Z Test filter criteria : null
> 2018-07-18T20:14:25.1968810Z Search folder : C:\vsts-agent\_work\4\s
> 2018-07-18T20:14:25.1969311Z Run settings file :
> C:\vsts-agent\_work\4\s 2018-07-18T20:14:25.1970327Z Run in parallel :
> false 2018-07-18T20:14:25.1970792Z Run in isolation : false
> 2018-07-18T20:14:25.1972913Z Path to custom adapters : null
> 2018-07-18T20:14:25.1973153Z Other console options : null
> 2018-07-18T20:14:25.1973381Z Code coverage enabled : false
> 2018-07-18T20:14:25.1974004Z Rerun failed tests: false
> 2018-07-18T20:14:25.1974423Z VisualStudio version selected for test
> execution : latest 2018-07-18T20:14:25.3811086Z
> ======================================================== 2018-07-18T20:14:30.6014691Z [command]"C:\Program Files
> (x86)\Microsoft Visual
> Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
> #C:\vsts-agent\_work\_temp\2aa40f41-8ac7-11e8-b190-a1cdff2b8b30.txt
> 2018-07-18T20:14:30.8198002Z Microsoft (R) Test Execution Command Line
> Tool Version 15.7.2 2018-07-18T20:14:30.8206458Z Copyright (c)
> Microsoft Corporation. All rights reserved.
> 2018-07-18T20:14:30.8206862Z 2018-07-18T20:14:30.8227447Z
> vstest.console.exe 2018-07-18T20:14:30.8228082Z
> "C:\vsts-agent\_work\4\s\partners\exooutlook\OutlookAnalysisSolution.Test\bin\Debug\OutlookAnalysisSolution.Test.dll"
> 2018-07-18T20:14:30.8228647Z /logger:"trx"
> 2018-07-18T20:14:31.0672644Z Starting test execution, please wait...
> 2018-07-18T21:14:05.8979923Z ##[error]The operation was canceled.
> 2018-07-18T21:14:05.9027909Z ##[section]Finishing: Unit Tests
We found the source of the hang. It was in a test class method marked with the "AssemblyInitialize" attribute.
[TestClass]
public class TestClassInitializer
{
[AssemblyInitialize]
public static void AssemblyInit(TestContext context)
{
LocalResourceDeployment.CopyResources();
}
}
There was a bug in CopyResources() that did not manifest locally but caused a powershell window in VSTS to wait for console input.
We have improved the trace logging in this initialization method so that if it fails again in the future we'll at least have a pointer to where the test got stuck. E.g.:
Trace.WriteLine("Starting test assembly initializtion.");

Cannot find element for dotnet test id: error in Resharper 2016.3

I just downloaded Resharper 2016.3 EAP 4 to check out the Unit test functionality with .NET Core. But when I run all unit tests, I get this error:
Cannot find element for dotnet test id:
MvcMovieTests.SimpleTests.TestMethodPassing
Cannot find element for dotnet test id:
MvcMovieTests.SimpleTests.TestMethodFailing
Here are my simple unit tests:
[TestClass]
public class SimpleTests
{
[TestMethod]
public void TestMethodPassing()
{
Microsoft.VisualStudio.TestTools.UnitTesting.Assert.IsTrue(true);
}
[TestMethod]
public void TestMethodFailing()
{
Microsoft.VisualStudio.TestTools.UnitTesting.Assert.IsTrue(false);
}
}
When I run all unit tests with the MSTEST Test Explorer, they run properly and I see the results. But with Resharper 2016.3 I get the two errors above showing up in the Unit Test Sessions window in Visual Studio 2015 Community.
Clearing up the ReSharper caches fixed it up for me. Open the Environment | General page of ReSharper options. Click Clear caches. More information here.

Problem running tests under Netbeans

I am running some Junit tests and Netbeans behaves strangely giving the report in "Output" window:
Testcase: warning(junit.framework.TestSuite$1): FAILED
No tests found in uk.ac.cam.ch.wwmm.chemicaltagger.ChemistryPOSTaggerTest
junit.framework.AssertionFailedError: No tests found in uk.ac.cam.ch.wwmm.chemicaltagger.ChemistryPOSTaggerTest
Test uk.ac.cam.ch.wwmm.chemicaltagger.ChemistryPOSTaggerTest FAILED (crashed)
test:
BUILD SUCCESSFUL (total time: 12 seconds)
The (5) tests are there. I have run mvn test which runs them but fails on OutOfMemoryError. Is this likely to be the cause of the Netbeans problem?
How did you created your test file? Manually, or using NB wizard? (Tools - Create JUnit test from Java file's popup menu)
If you are using JUnit 3, all test methods in your test file must start with "test", e.g.
public void testFoo() { //some testing here :) }
With JUnit 4, a '#Test' annotation is required, e.g.
#Test
public void myOwnTestFoo() { //...}
Otherwise JUnit does not recognize the test and throws AssertionFailedError error.