I have a VS2010 unit test project set to using SpecFlow 1.8.1 and mstest. In order to get the SpecFlow unit tests working, I've done the following:-
I added the references to the following files in my project:-
Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
TechTalk.SpecFlow.dll
Note that the TechTalk.SpecFlow.dll has been added into my project and the reference points to that file.
I've set the "Copy Local" property of the TechTalk.SpecFlow.dll reference to True.
I've also added an App.Config that specifies "MsTest.2010" as the provider, and regenerated all code-behinds for the SpecFlow features.
Everything works in my VS2010, the tests run successfully in both the SpecFlow testrunner and the mstest test runner. BUT when I try to run the mstests in TFS 2008 (using a .vsmdi test list file), it failed with the following exception:-
Class Initialization method MyNamespace.MyTestFeature.FeatureSetup threw exception.
System.Configuration.ConfigurationErrorsException:
System.Configuration.ConfigurationErrorsException: An error occurred creating the
configuration section handler for specFlow: Could not load file or assembly
'TechTalk.SpecFlow' or one of its dependencies. The system cannot find the file
specified. (D:\Projects\TestProject\TestResults\administrator_MYPC 2012-06-27
18_30_05_Any CPU_Debug\Out\TestProject.DLL.config line 4) --->
System.IO.FileNotFoundException: Could not load file or assembly 'TechTalk.SpecFlow'
or one of its dependencies. The system cannot find the file specified.
Note that the TFS built the project fine and it runs other unit tests in the same project (normal mstests, not SpecFlow) without problems. It only failed for the SpecFlow test runs.
So what am I doing wrong?
Edit: The contents of my App.Config file looks like this:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section
name="specFlow"
type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow"/>
</configSections>
<specFlow>
<unitTestProvider name="MsTest.2010" />
<runtime detectAmbiguousMatches="true"
stopAtFirstError="false"
missingOrPendingStepsOutcome="Inconclusive" />
<trace traceSuccessfulSteps="true"
traceTimings="false"
minTracedDuration="0:0:0.1" />
</specFlow>
</configuration>
Following the instruction on this site and this site:
the command Tools > Library Package Manager > Package Manager Console allows you to type in PM> Install-Package SpecFlow
when the prompts returns "installed successfully", the SpecFlow Assembly now appears in the references of your project. And the MSTest project now compiles succesfully (at least for me).
I got this error as well, in my case the problem was that I was using the \...\obj\Debug||Release\ folder as target and not the \...\bin\Debug||Release\ folder. Looking in these folders I saw that the TechTalk.dll assembly was missing from the former. Simply switching in my .bat file the problem was fixed.
Sometimes VS2013 is looking for SpecRun dlls not in project folder, but in C:\Users\**YOUR_USER**\AppData\Local\Temp\VisualStudioTestExplorerExtensions\SpecRun.Runner.1.3.0\tools. So you just need to put all necessary SpecFlow libraries therel
One hack I found to get it working is to add another class for EVERY single SpecFlow feature that I created in the project. The class looks like this:-
[DeploymentItem(#"TechTalk.SpecFlow.dll")]
partial class MyTestFeature { }
// The above class-name needs to come from the auto-generated code behind
// (.feature.cs) for each SpecFlow feature.
I consider this as a very nasty hack, but it does provide a clue as to why it didn't work. It would be good if anyone comes up with a more elegant solution.
I finally found the more proper fix for this issue. I just need to add a post-build event to remove the .config file from the build output. (The App.config file is used only to generate the code-behind during design time. It is not used at all during runtime, so it can be removed.)
The command for the post-build event looks like this:-
del /f /q "$(TargetDir)$(TargetFileName).config"
Correction: The .config file is used for generating inconclusive results, so a better post-build event command is as follows:-
if "$(IsDesktopBuild)"=="false" del /f /q "$(TargetDir)$(TargetFileName).config"
Related
How can I set the path to my external binaries during test discovery in visual studio's Test Explorer? After that how to make sure, it uses the correct paths?
I use windows 10 and VS 2019. I have a solution that builds some binaries and some tests into different folders. Also, I have some 3rd party dependencies, each in its own folder.
Something like:
solutionDir/
-ownBinaries/
-testBinaries/
-externalBinaries/
I'd like to use the Test Explorer to run my tests. For this purpose, I use a .runsettings file. I installed Google Test adapter via NuGet (later it will run on CI, so this is the only option). The automatic runsetting discovery is disabled, and this file is selected as the runsettings file. It overrides the workingDir to my ownBinaries folder, and extend the PATH enviroment variable with the externalBinaries. The relevant parts are:
<SolutionSettings>
<Settings>
<AdditionalTestExecutionParam>-testdirectory=$(SolutionDir)</AdditionalTestExecutionParam>
<WorkingDir>$(SolutionDir)ownBinaries</WorkingDir>
<PathExtension>$(SolutionDir)externalBinaries</PathExtension>
</Settings>
</SolutionSettings>
This is works fine, after my tests are discovered, but I have problems when it tries to discover my tests.
I use google test and c++, so the test discovery tries to run those tests with the --gtest-list-tests argument, then populate the view with the test name, case, etc. The binaries are just fine, builds without error, I can run them from the debugger, and they produce the output I want.
But the test explorer won't show them, because it doesn't set the externalBinaries path.
This is what lead me to this situation.
First I copied every binaries next to my test exe, namely into the testBinaries folder. Then, I could run it in the cmd with the --gtest-list-tests argument. Everything was fine, all my test names showed up. Started VS, and Test Explorer discovered all my tests, it was able to run them.
Then I done a clean build, so the external stuff deleted from the testBin folder. The Test Explorer cached the test names, so it was able to run them.
Restart VS. Test Explorer tries to discover my tests. but it fails whit this helping message: (removed date+time)
Google Test Adapter: Test discovery starting...
Failed to run test executable 'D:\MySolution\testBinaries\SBCUnitTest.exe': One or more errors occurred.
Check out Google Test Adapter's trouble shooting section at https://github.com/csoltenborn/GoogleTestAdapter#trouble_shooting
In particular: launch command prompt, change into directory '..\ownBinaries', and execute the following command to make sure your tests can be run in general.
D:\MySolution\testBinaries\SBCUnitTest.exe --gtest_list_tests -testdirectory=
Found 0 tests in executable D:\MySolution\testBinaries\SBCUnitTest.exe
Test discovery completed, overall duration: 00:00:00.3022924
Have you noticed that -testDirectory= is empty despite it is set in the runsettings file?
I'm completely lost how I can proceed with it. This workaround is quite heavy to copy all files, then delete all but the test binaries each time when I start VS.
Here is the link for the Troubleshooting section mentioned in the error message.
I've read through the readme file on github, also the runsetting docs on Microsoft's website.
Edit
I made progress with the VsTest.console.exe, I can successfully run all my tests with the proper arguments as below:
& "VSTest.console.exe" *_uTest.exe /Settings:..\MySolution.gta.runsettings /TestAdapterPath:"..\packages\GoogleTestAdapter.0.18.0\build\_common\"
I use the same *.runsettings and *.gta_settings_helper files. Those files are used to get absolute paths for the dependencies. I could run this from different folders, but then I had to adjust the arguments (test discovery pattern, relative path to runsettings, and relative path to GTA).
Great news, that it successfully runs on Azure (it uses vstest.console).
Edit 2
Tried to merge the workingDir and pathExtension nodes, so only one needed (the pathExtension). No success.
Tried to install Test adapter for google test in the VS installer, delete the runsetting file, and set the properties in VS->Tools->Options then Test adapter for google test. Even the example pathExtension didn't worked for me.
Found the extended logs under %AppData%/Local/Temp/TestAdapter/someNumber/*.txt and in that log I've found one line as the runsettings file. I paste here the formatted version of the log
<RunSettings>
<GoogleTestAdapterSettings>
<SolutionSettings>
<Settings>
<WorkingDir>$(SolutionDir)</WorkingDir>
<PathExtension>$(SolutionDir)externalBinaries</PathExtension>
</Settings>
</SolutionSettings>
<ProjectSettings>
</ProjectSettings>
<GoogleTestAdapterSettings>
<SolutionSettings>
<Settings>
</Settings>
</SolutionSettings>
<ProjectSettings>
</ProjectSettings>
</GoogleTestAdapterSettings>
</GoogleTestAdapterSettings>
</RunSettings>
Does anybody know why is there an empty google test adapter setting? Where does it comes from? I think this is overwrites my settings.
It turned out, before first run the relative paths are not known.
Trivial solution
Add the full path to the PATH Extension under Visual Studio -> Options -> Test Adapter for Google Test settings. Meanwhile the custom *.runsetting file is not selected.
Using this method all my tests are discoverable, but it is a manual setting for each repo cloned.
I have this build :
It works fine. The only issue is that the Test Results are overridden. So I actually end up with the test results for the last test project executed.
This is executed by build engine;
C:\Program Files\dotnet\dotnet.exe test C:/agent/_work/4/s/test/Services.UnitTests/project.json --configuration release -xml ./TEST-tle.xml
C:\Program Files\dotnet\dotnet.exe test C:/agent/_work/4/s/test/Web.UnitTests/project.json --configuration release -xml ./TEST-tle.xml
What could help:
1) having "dotnet test" generate XML output file - did not find a way how to do that
2) Use a variable for -xml output file in Build Task. That variable could be a random string/number or just a project name being tested - like what Build engine feeds to "dotnet.exe test". No way how to do that.
Any ideas? Thanks.
I think that, although you're running the task against all of the projects in one go, as the .Net Core (Preview) task doesn't have a working directory, that the test results are being generated at solution root (or similar) and done for each project in turn.
I set mine up using simple command line tasks...
Tool: dotnet
Arguments: test -xml testresults.xml
Working folder: {insert the folder for the project to test here}
These work fine but I have one set up for each project. You could try creating a task for each library and adding the full path to the test results argument (or name them appropriately as starain suggested).
This feels like a minor bug to me.
Based on my test, it doesn’t recognize the date variable as Build Number.
To deal with this issue, you can add another .Net Core (Test) step to run xunit test with different result file.
For example:
I'v some DB facing unit tests that take their connection string from an app.config file. I'm grabbing the strings thru ConfigurationManager class.
var conString = ConfigurationManager.ConnectionStrings["DB"].ConnectionString;
The tests run fine on dev machine but not on TFS. There is a null refernce error. If I jsut replace the connection string in unit test with a hard coded string - identical to that in app.config - then the test works. Oh yeah, and I've made sure that app.config is set to "copy to output folder".
Is there a known issue with TFS and ConfigurationManager?
pom
I recently ran into the same problem with TFS 2015 (on premises with update 2). We have multiple test projects that access a connection string from the app.config file however one particular project failed to access the connection string which caused our build to fail.
I altered the .csproj file to match other working test projects but this didn't provide any success, instead the following worked for me:
Copied the contents of the App.config to notepad
Deleted the App.config file from the project
Added a new Application Configuration File (App.config) to the project
Pasted the contents from notepad into the new App.confg file
Right click the App.config file and altered its properties:
Build Action: Content
Copy to output directory: Copy always
Upon check-in all unit tests within the project passed successfully.
It's an odd solution but it got our tests running.
Note: Deleting the App.config file was a necessary step. Changing the properties on the file (prior to deleting the file) didn't make a difference.
Update:
After applying the above solution I noticed all unit tests were running twice and after some investigation I discovered the entire issue was down to our build configuration: -
On our Visual Studio Build task we use the following parameter on our MS Build Arguments:
/p:OutDir=$(Build.StagingDirectory)
Our Visual Studio Test task was configured to look for Test Assembly in the following location:
**\$(BuildConfiguration)\*test*.dll;-:**\obj\**
Altering the Test Assembly location to the following fixed everything:
$(Build.StagingDirectory)\*test*.dll;-:**\_PublishedWebsites\**
No more issues with App.config files and unit tests ran once only.
Actually, I made another test project in my solution and added the app.config there again with the conn strings and now it works also on TFS. I don't know what the cause was but I'll leave the question here for possible reference and solution for others. Thank you.
My Test project is a separate project from the main project. Therefore it needs its own copy of App.config. Follow the steps in previous response to add new Application Configuration file and give it the same contents as in your main project.
I am trying to set/override some settings in our TEST installation of TFS with regards to forcing Code Analysis and assosicated settings during the build process (regardless of the setting sin the project file)
We currently use in our TEST TFS installation:
Visual Studio 2012 Ultimate on our developer machines AND build server
Have TFS 2012 installed on one server (application and data layer)
Have TFS 2012 build service (controller and agent) installed on another server
We can compile sample .net 4.5 projects (class libraries (DLLs), web applications etc) as expected. This is solely to do with overriding associated Code Analysis settings (hopefully).
Scenario 1 - In our sample applications on our developer machines when you select the project settings (right click -> properties in solution explorer), go to the Code Analysis tab if I turn on the "Enable Code Analysis on build" and select a Rule set from the drop down is performs as exepcted, hence it will generate some warnings. This technical adds <RunCodeAnalysis>false</RunCodeAnalysis> to the *.csproj file if opened up in notepad. If the build is executed to compile the sample project/solution then Code Analysis is performed as expected. I do NOT want to do this on every project because a developer could turn it off (although I am looking to have check-in policies and/or private/gated checkins as well to force this anyway).
Scenario 2 - I can disable the "Enable Code Analysis on Build" checkbox and force code analysis in our TFSBuild.proj file (we (will) use the default upgradetemplate.xaml as our process definition because we will be upgrading from TFS 2008 on our LIVE TFS installation) by having:
<RunCodeAnalysis>Always</RunCodeAnalysis>
This works and this is how we will force (lessons still to be learned :-)) Code Analysis on our builds.
The problem then comes when setting other assosicated Code Analysis settings. For example which default rule set(s) to apply/use or treat CA warnings as errors. Some of these settings can be set either in VS or all of them by editting *.csproj in notepad. If i edit the *.csproj then these values are used in the build as expected (as well as locally on the developer machine). This is not ideal as I want to do it centrally in TFSBuild.proj without having to edit every project file. I believe I can use settings such as in my TFSbuild.proj file:
<PropertyGroup>
<RunCodeAnalysis>Always</RunCodeAnalysis>
<CodeAnalysisRuleSet>AllRules.ruleset</CodeAnalysisRuleSet>
<CodeAnalysisTreatWarningsAsErrors>true</CodeAnalysisTreatWarningsAsErrors>
</PropertyGroup>
But they don't appear to work or I am putting them in the wrong place? How do I fix/use them correctly?
FYI i build my solutions in TFSBuild.proj by:
<Project DefaultTargets="DesktopBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="4.0">
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets" />
<ItemGroup>
<SolutionToBuild Include="/some folder/some solution.sln" />
<ConfigurationToBuild Include="Debug|Any CPU">
<FlavorToBuild>Debug</FlavorToBuild>
<PlatformToBuild>Any CPU</PlatformToBuild>
</ConfigurationToBuild>
</ItemGroup>
</Project>
On the build server I did find reference to the target file for Code Analysis at c:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\CodeAnalysis but I dont want to change the default behaviour on the build server (although it does work when I do). The condition for example for CodeAnalysisTreatWarningsAsErrors must be getting evaluated as false. Its like my values are not read from TFSBuild.proj but are from the .csproj file.
Any questions feel free to ask and thanks in advance
I had what I think is a similar problem with Cruise Control not compiling using the CODE_ANALYSIS compilation symbol, even if "Enable Code Analysis on Build (defines CODE_ANALYSIS constant)" was checked in VS.Net.
It looks like whether it is check or not, CODE_ANALYSIS is actually not explicitly added to the list of compilation symbols in the csproj (even if it appears in the text box "Conditional compilation symbols"), only <RunCodeAnalysis>true</RunCodeAnalysis> is added.
When compiling through VS.Net, the CODE_ANALYSIS is automatically added, but not when using MSBuild, which is what Cruise Control uses.
I eventually changed in VS.Net the "Conditional compilation symbols" from "CODE_ANALYSIS;MySymbol" to "MySymbol;CODE_ANALYSIS". Doing that forced CODE_ANALYSIS to also appear in the csproj.
I remember having a similar problem - but not having the time to investigate it, I worked around it by calling FxCop directly using the exec task. I'll just give you the highlights, omitting the specification of some properties, I hope the names are clear.
I created an ItemGroup of the output dlls, FilesToAnalyze, and fed it to FxCop in a way similar to:
<PropertyGroup>
<FxCopErrorLinePattern>: error</FxCopErrorLinePattern>
<FxCopCommand>"$(FxCopPath)" /gac /rule:"$(FxCopRules)" /ruleset:="$(FxCopRuleSet)" #(FilesToAnalyze->'/file:"%(identity)"', ' ') /out:$(FullFxCopLog) /console | Find "$(FxCopErrorLinePattern)" > "$(FxCopLogFile)"</FxCopCommand>
</PropertyGroup>
<Exec Command="$(FxCopCommand)"
ContinueOnError="true">
<Output TaskParameter="ExitCode" PropertyName="FxCopExitCode"/>
</Exec>
<ReadLinesFromFile File="$(FxCopLogFile)">
<Output TaskParameter="Lines" ItemName="AllErrorLines"/>
</ReadLinesFromFile>
I could then determine the number of errors in the output using an extensionpack task:
<MSBuild.ExtensionPack.Framework.MsBuildHelper TaskAction="GetItemCount" InputItems1="#(AllErrorLines)">
<Output TaskParameter="ItemCount" PropertyName="FxErrorCount"/>
</MSBuild.ExtensionPack.Framework.MsBuildHelper>
and create a failing build step for each error:
<BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
Id="$(FxCopStep)"
Status="Failed"
Message="FxCop Failed: $(FxErrorCount) errors."/>
<BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
Status="Failed"
Message="%(AllErrorLines.Identity)"/>
By doing code analysis on the build server this way, we also avoided having to configure each project separately. We isolated all this in a separate .targets file, so adding code analysis to a solution was a matter of importing that file, and perhaps adjusting the behavior by setting appropriate properties.
We have been using TeamBuild and test for our continuous integration build for about 4 months and this issue just popped up the other day in one of our test assemblies when running tests on one of our test assemblies on the build server only.
{NameSpace}.Order_Accessor, Version=0.0.0.0, Culture=neutral, PublicKeyToken='{xxx}' or one of its dependencies. The system cannot find the file specified.
This is not being caused by problems with Publicize.exe noted elsewhere. The unit tests run fine locally and the generated assembly does get created.
The part that is confusing is that we use generated private accessor assemblies in numerous projects including the one with this issue above without any problems. Investigating this issue further, the generated assembly never gets copied to the TestResults//Out directory which appears to be the problem. I determined this was the issue by adding the assembly from SolutionRoot using the TestRunConfig deployment settings which results in an assembly manifest does not match error.
I am wondering if anyone has had issues with these assemblies being copied properly. We are running TeamSystem 2008 on Windows Server 2008 if that helps.
Below is the test run deployment error for the file in question as well
Run has the following issue(s):
TESTTOOLSTASK : warning : Test Run deployment issue: Failed to get the file for deployment item 'E-mail Templates\OrderConfirmation.txt' (output directory 'E-mail Templates') specified by the test 'EmailServiceTest.EnqueueTest': System.IO.DirectoryNotFoundException: Could not find a part of the path 'Continuous Integration Build\Binaries\Continuous Integration Test\E-mail Templates\OrderConfirmation.txt'.
TESTTOOLSTASK : warning : Test Run deployment issue: Failed to get the file for deployment item 'E-mail Templates\OrderConfirmation.htm' (output directory 'E-mail Templates') specified by the test 'Email.Tests.EmailServiceTest.EnqueueTest': System.IO.DirectoryNotFoundException: Could not find a part of the path 'Continuous Integration Build\Binaries\Continuous Integration Test\E-mail Templates\OrderConfirmation.htm'
TESTTOOLSTASK : warning : Test Run deployment issue: The assembly or module 'Services.Order_Accessor' directly or indirectly referenced by the test container '\continuous integration build\binaries\continuous integration test\services.order.supplierintegration.tests.dll' was not found.
Looks like your TestRunConfig deployment expects that the files are in "Continuous Integration Build\ Binaries\"
I've highlighted the "Binaries" part for a reason. This is a folder the $BinariesRoot property points to (unless overridden), and it is used by the compiler. Therefore, I'd suggest you double-check that:
The Publicize tool (or whatever tool you use to generate the assemblies) is configured to copy the output into ($BinariesRoot)Continuous Integration Test
There are provisions in the team build type to copy the .htm and the .txt files to ($BinariesRoot)Continuous Integration Test