I have a Visual Studio 2012 Solution with a number of native c++ test projects.
I can run all of these correctly and successfully from within Visual Studio 2012 using the Test Explorer tab.
However, I cannot get the tests to run when running from the command line.
Following the documentation I have been running the following command line
mstest /testcontainer:PathToTestProject\Win32\Release\testproject.dll
I also need to run
mstest /testcontainer:PathToTestProject\x64\Release\testproject.dll
for the testing of the 64bit version of the code.
When I run these command lines I get the following error message.
Microsoft (R) Test Execution Command Line Tool Version 11.0.50727.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading PathToTestProject\Win32\Release\testproject.dll...
PathToTestProject\Win32\Release\testproject.dll
Unable to load the test container PathToTestProject\Win32\Release\testproject.dll' or one of its dependencies. If you build your test project assembly as a 64 bit assembly, it cannot be loaded. When you build your test project assembly, select "Any CPU" for the platform. To run your tests in 64 bit mode on a 64 bit processor, you must change your test settings in the Hosts tab to run your tests in a 32 bit process. Error details: Could not load file or assembly 'file:///c:\PathToTestProject\Win32\Release\testproject.dll' or one of its dependencies. The module was expected to contain an assembly manifest.
The code is native c++ and has two build configurations one on Win32 platform, and the other on x64 platform. I cannot have an AnyCPU platform configuration.
What am I missing here to be able to run the tests from the command line?
After a lot of searching, I finally discovered a very hidden msdn documentation page
here which states the compatibility of mstest with different test project types.
And it turns out the mstest is not compatible with native unit tests (nice of msdn to document this in an easy to find location).
Instead you need to use the visual studio test running (vstest.console.exe) instead of msbuild for native unit test projects.
for example
vstest.console.exe /Platform:x64 PathToTestProject\x64\Release\testproject.dll
Related
I'm trying to get Code Coverage for a dotnet core application through manual testing (I don't have any unit tests project)
As mentioned in this SO Question, I'm trying to use vsinstr for instrumentation and vsperfmon.
But when running the below command
vsinstr /coverage C:\Test\netcoreapp2.0\ConsoleAppCodeCoverageTest.dll
I get the below error.
Microsoft (R) VSInstr Post-Link Instrumentation 15.0.27128 x86
Copyright (C) Microsoft Corp. All rights reserved.
File to Process:
C:\Test\netcoreapp2.0\ConsoleAppCodeCoverageTest.dll -->
C:\Test\netcoreapp2.0\ConsoleAppCodeCoverageTest.dll
Original file backed up to
C:\Test\netcoreapp2.0\ConsoleAppCodeCoverageTest.dll.orig
Error VSP1019: Internal error. HRESULT=80004005
But if I use vsinstr on a .Net Framework application exe it works fine.
Any idea how to fix this of if there is any other approach to get code coverage for a dotnet core application through manual testing.
Currently you have to be running Visual Studio Enterprise and it only works on Windows.
Related github issue.
If that isn't a problem for you, note these instructions:
If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info.
More details here.
Using the C++ unit testing framework with Visual Studio 2013 Professional, one can write unit tests and run them from within the IDE, but in order to generate any coverage reports, apparently, one needs to have the Premium or Ultimate edition of Visual Studio.
Is it possible to get code coverage reports with the Professional edition, preferably without installing any third party tools? If not, what alternate options exist for people who are not using the more expensive editions of the IDE?
Please note that it's possible to generate the coverage data by using command line tools, but I am unable to find a way to look at the results. For the sake of reference, here are the steps for command line generation of the coverage statistics:
Build the code to test with with /PROFILE linker switch
Run vsinstr /coverage <binaryName> to instrument the code; make sure that you are inside VS 2013 command prompt
Run start vsperfmon -coverage -output:results to run the profiler
Run your unit tests
Stop the profiler by running vsperfcmd -shutdown
The above will give you a results.coverage file, with no way to view it without the Premium or Ultimate editions as far as I know.
With VS2013 Professional you are out of luck if you want to do it without third party tools Requirements: Visual Studio Ultimate, Visual Studio Premium (http://msdn.microsoft.com/en-us/library/dd537628.aspx). From what I understand you already managed to generate your *.coverage file and you are having problems opening it. Visual Coverage (https://github.com/jsargiot/visual-coverage) tool can help you with that, its very simple to use and it is opensource. If you would like to find more alternatives, see another SO thread: Viewing Code Coverage Results outside of Visual studio. The tools are meant for C# coverage files but from what I understand there should not be any difference.
If you are ready to use third-party libraries, you can use OpenCPPCoverage. It works for me like a command line app. But I couldn't run it like a visual studio plugin at Professional Visual Studio 2013.
One possible way you could get coverage data is by running an instrumentation toolkit on your Unit Test program. The instrumentation tool will tell you how much was covered during execution and you can easily relate this data back to how much coverage your unit tests give you. I did this with an XCode project and OpenPAT but you could do the same with any Visual Studio friendly toolkit.
I have a project upgraded from Visual Studio 2010 to 2012 and the .testrunconfig file was included in the upgrade process.
I noticed that it was possible to click "Analyze code coverage" on any of the unit tests that I had run and it would correctly display the result. However, my test run configuration (originally from VS 2010) had code coverage disabled.
After doing a bit of research I learned that the VS 2010 configuration files have been deprecated and replaced by .runsettings files. It would appear that VS 2012 enforces assembly instrumentation by default which has a massive overhead associated with it.
Therefore, I would like to know how I can disable code coverage in VS 2012. Based upon my current findings it does not seem to be a trival task. One recent article I read had me creating an XML file manually and naming it "MYCONFIGURATION.runsettings" and manually manipulating XML attribute values.
Does anyone know how this should be done?
This is what I understand from your post:
You have a Test project with .testsettings file. You have not enabled code coverage in the test settings.
Code coverage instrumentation is not enabled by default in your scenario. Binaries will be instrumented if you do 'analyze code coverage' from VS.
Additional Info:
You can confirm that .coverage file is not generated by running the following command from visual studio developer command prompt:
vstest.console.exe /Settings:<your test settings file> test.dll
A coverage file will only get generated if you have enabled coverage in test settings.
Code coverage is only enabled through the Test Explorer using data driven adapters. The metadata for tests ran through the test explorer is almost completely different than that of tests ran straight from the unit test session window. Have you tried simply running it straight from the code (the MSTest gui bubbles) or from the unit test session window?
I'm using TFS, it works fine. Now I want to add tests and code coverage. So I create a Local.testsettings. Running tests on my computer runs fine.
Configuration of the TFS Build is as follows:
- Automated test with test assembly file specification and the same test settings file (as recommended, instead of using a .vsmdi file)
When launching a build with TFS, my tests don't get executed :
Run MSTest for Test Assemblies
The MSTestActivity was invoked without a value for Platform or Flavor. The values Mixed Platforms and Debug were used.
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo /usestderr /testSettings:"C:\Builds\1\MyProject\Sources\MyProject\Main\Source\MyProject\Local.testsettings" /searchpathroot:"C:\Builds\1\MyProject\Binaries" /resultsfileroot:"C:\Builds\1\MyProject\TestResults" /testcontainer:"C:\Builds\1\MyProject\Binaries\MyProject.Tests.Module1.dll" /maxpriority:"1" /minpriority:"1" /publish:"http://tfsserver:8080/tfs/Test" /publishbuild:"vstfs:///Build/Build/433" /teamproject:"MyProject" /platform:"Mixed Platforms" /flavor:"Debug"
Loading C:\Builds\1\MyProject\Sources\MyProject\Main\Source\MyProject\Local.testsettings...
Loading C:\Builds\1\MyProject\Binaries\MyProject.Tests.Module1.dll...
Starting execution...
No tests to execute.
There are no results to be published.
So it finds the assembly, but MSTest doesn't actually runs the tests.
Any hint is welcome.
Looking at this:
/maxpriority:"1" /minpriority:"1"
It looks like your priority criteria may be the problem.
I installed the newest Gallio release 3.2.603 on a box without Visual Studio that will be our CI server. The NAnt script successfully built the .Net 4.0 projects but when it tries to run MSTest tests the following error occurs.
[gallio] Gallio NAnt Task - Version 3.2 build 601
[gallio] [error] Assembly XXXXXXXXXXXXXX
[gallio] Cannot run tests because the MSTest executable was not found
Am I wrong to expect that Gallio is able to run MSTest tests without the MSTest executable or any other Visual Studio components installed?
Ok, I got an answer from Gallio support that it actually requires MSTest.exe to be present on the box. I'll test the package that Aseem suggested whether it provides enough stuff to run the tests.
Update: I installed Microsoft Visual Studio Test Agent 2010 from the package. I had to add the string value InstallDir to [HKLM\SOFTWARE\Microsoft\VisualStudio\10.0] and set the value to something like
“C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE”
Now my CI box runs those tests perfectly.
I dont know anything about Gallio, but just wanted to share that with visual studio 2010, you dont really need visual studio to run MSTest tests. You can simply install test agent (a very light-weight sku) which additionally installs MSTest. This test agent installer can be installed from here.
Confirmed this worked for me. Running Sonar on a box with VS2010. Sonar could not find MSTest until I added this registry value.