I'm building ~600 projects to a flat file structure (this includes a bunch of unit test assemblies). I'm following a naming convention to easily identify unit test assemblies (.utest.dll).
TFS Build is finding the correct unit test projects to build, but it's not copying their respective app.config files to the flat file structure before trying to execute the unit tests.
What would be the easiest way of having every test use their own app.config file?
You can add the following attribute to your test method to have the app.config file deployed.
[DeploymentItem("app.config", "targetFolder")]
Check this MSDN article for the details: https://msdn.microsoft.com/en-us/library/vstudio/ms182475(v=vs.120).aspx
Related
I have a local nuget package which contains a single test. The test is decorated with [TestClass] and it's test method is [TestMethod].
The reason this is a nuget package is because this test will be brought in to any test projects in order to test that a t4 transform has occurred in the referenced assemblies. The t4 template generates classes dynamically and I want the test to run to verify the t4 template has been executed (using reflection to make sure the generated classes have the expected methods)
When I reference the nuget package, the Visual Studio Test Explorer never displays the test. I was under the impression that VS reflected over the types in the assemblies to build it's test list, but that seems to be an incorrect assumption.
Is there a configuration setting or something that I am missing in order to have the test discovered?
Thank you,
Jason
Visual Studio Test Explorer can only run tests from multiple test projects in a solution and from test classes that are part of the production code projects.
There are two workarounds for your scenario:
Run the test from command line.
Instead of placing the generated dll file in the nuget package, placing the test class file in the "content" folder of the nuget package. This will add the test class file to your project when install the nuget package and then Test Explorer will detect the tests after building the solution.
My application has several unit tests projects. All other projects and the unit tests projects get built to a common output directory. With an msbuild task I'm collecting all the unit tests assemblies and run vstest.console.exe to test these assemblies.
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\..\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "Assembly1.UnitTest.dll" "Assembly2.UnitTest.dll" /Platform:x64 /Framework:Framework40 /InIsolation /Logger:trx
This results in an exception: System.IO.FileNotFoundException: Could not load file or assembly 'someassembly.dll' or one of its dependencies. The specified module could not be found.
When I execute vstest.console.exe for a single unit tests assembly the above exception doesn't occur. Further investigation learned me that when testing multiple assemblies at once the vstest.console is copying the test assemblies and depended assemblies to an "out" directory in the "testresults" directory. However not all needed assemblies are referenced by a project but manually copied to the common output directory. Those assemblies are missing in the "out" directory in "testresults" and causing the System.IO.FileNotFoundException.
With MSTest I could use a config file an point to the common output directory by adding a DeploymentItem. For vstest.console this doesn't work anymore.
What can I do to get around this behavior? I don't want to work with an "out" directory. Running my unit tests from the common output directory is just fine.
PS. I have the same issue on TFS 2013 with build definitions. My build definitions are collecting *.unittest.dlls and executing these with the Test Runner.
The solution is using a .runsettings file. In the runsettings file it is possible to specify "DeploymentEnabled". By default this is true. Changing it to false doesn't copy all the assemblies to the out directory of the TestResults directory. More information on the runsettings file: http://msdn.microsoft.com/en-us/library/jj635153.aspx.
Example:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!--Info: http://msdn.microsoft.com/en-us/library/jj635153.aspx -->
<MSTest>
<IgnoreTestImpact>True</IgnoreTestImpact>
<MapInconclusiveToFailed>True</MapInconclusiveToFailed>
<CaptureTraceOutput>False</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>True</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>False</DeploymentEnabled>
</MSTest>
</RunSettings>
I have DLL plug-in, which I'd like to test and launch on TeamCity: it contains .h and .cpp files
What is the right strategy to test this DLL:
Create a new Test Project in the same solution, configure 'include directory' to see sources and copy DLL project files to Test console project. In this case I have the same solution but in console mode, which I can test in normal way. But if my DLL project will change I need to synchronize source files.
Create export function in my DLL, like 'runTests()' which will produce XML file with results. TeamCity will process this? And how should it run this? And some stuff function appears in release DLL...
To unit test our libraries, we create standalone unit testing console executables. so:
For each library , we create a console executable testing each method in the API.
Each source file is obviously added to a SCM so modifying files will automatically be reflected into the unit testing program;
All this (source updates, compilation, unit testing and doc generation) is added to our CI server (Jenkins) so that all libraries, for all unit testing programs are always recompiled from scratch;
The documentation for the library API is constructed with Doxygen using snippet from this program. This has a nice side-effect: changing the API will break your unit tests. So you have to fix your unit tests so your documentation is always up to date.
I have all my Unit Test project in a folder under my Solution Folder and would like to have the TestResults folder in the same folder as the Test projects instead in the solution directory.
I have found that this could be done via the test seeting file:
How to specify the location for the unit test results in VS 2010?
but I also read, that with VS2012 you should no longer use tests settings files. Actually VS2012 does not create one.
Is there another way?
To specify a different location for the "TestSettings" folder, add a .runsettings to your solution as explained in the Visual Studio documentation: http://msdn.microsoft.com/en-us/library/vstudio/jj635153.aspx
My .runsettings file looks like this:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<ResultsDirectory>.\Visual Studio Test Results</ResultsDirectory>
</RunConfiguration>
</RunSettings>
As far as I could tell, however, the ResultsDirectory location is not relative to the solution folder (like the example file from the doc suggests), but rather relative to the location of the .runsettings file itself. Also note that Visual Studio macros like $(SolutionDir) are not expanded here.
All in all, .runsettings files are not related to a particular project or solution.
The reason why they recommend using .runsettings files instead of .testsettings in newer version of Visual Studio is also found in the documentation: http://msdn.microsoft.com/en-us/library/vstudio/ee256991.aspx
If you use a .testsettings file, the MSTest test framework will be
used to run your tests. This runs more slowly and does not allow you
to run tests from third-party test frameworks.
You can create a small RunSettings file which looks like
<RunSettings>
<RunConfiguration>
<ResultsDirectory>e:\myResultsFolder</ResultsDirectory>
</RunConfiguration>
</RunSettings>
Select this setting file via top level menu Test->TestSettings->"Select Test Settings" before running your tests.
You can find more detail on http://msdn.microsoft.com/en-us/library/jj635153.aspx.
When unit testing with Ivy, is it best practice to put the test dependencies into the Ivy.xml file for the target binary you are building? Or should they be in a separate Ivy.xml file for the test project you are building?
I'm using Ant build files on my Jenkins build server.
Originally I planned to run the Unit test project after the target binary's build, but then my uploads to Artifactory got confused as the last ivy resolve was done for the test dependencies, not the target binary.
If I place the test dependencies in the actual binary's Ivy.xml, I get the following error:
"a module is not authorized to depend on itself"
...but my test depends on the target binary that I'm building.
Should the test project's Ivy.xml file not actually list the target binary as a dependency?
Update
I have the dependencies now in the target binary's Ivy.xml.
I've flagged the new depencencies as having a "test" configuration to set them apart from the dependencies needed for the target binary.
I'm now working out how to point my test project at the binary produced by the build of the target project. I'm not sure if this is technically an issue with Ivy.
After I build the target binary, should I have a relative path reference from the test project to the target binary?
Another idea is to publish the target binary to the local cache, and then reference it from there in the target project.
Update 2
I ended up using the ant copy task to copy the binary built from target project to the folder with the target project's dependencies in it. Since the test project dependencies are in this folder, the target project can locate the target binary.
Agreeing with Mark O'Connor's comment:
Best practice with unit testing in general is to run the tests on the compiled code before packaging it.
I would recommend including your call to your unit tests as a step after compilation, and before packaging. Most of the time, you don't want to package and upload code to an artifact manager if it has failing tests (unless you have some form of special circumstances).
All of this implies that the tests should occur in the same Jenkins job. You can build both DLL's, have them refer to each other using relative paths, and execute tests, all in the same job. Then you can upload your binary to Artifactory, confident that it passes tests.