I have recently started to try to set up unit testing for my project. However, it is not working properly for classes.
I set it up using the directions from the question Unresolved externals when compiling unit tests for Visual C++ 2012.
What will happen is that the tests will start fine, but then there will be a memory access error (C0000005). This does not happen all of the time though. It will generally fail every two or three attempts, but can fail as often as one in every 20. It does not seem to matter what other functions are in the code.
When I try to debug the test, the first time a variable is accessed or used, the test will fail.
I am trying out two different testing frameworks, Boost.test (using Boost Unit Test Adapter) and Native Unit Tests. Both of them are having the same problems.
I know that the dll and the test application both use the same calling convention (__cdecl).
What is bizarre is that the test will work for a while, then suddenly begin to fail.
The project is available here.
Reinstalling Visual Studios did not help.
Whatever is going on, it's almost certainly in your code as many, many people use Boost.Test successfully with Visual Studio. Instead of a 100+ MB zip file on dropbox that noone is going to download, try to simplify your problem to the smallest possible bit of code that reproduces the problem and post it in your question.
Here's a more constructive answer. Try working through the 5-part tutorial I wrote on test-driven development with Boost.Test and Visual Studio:
C++ Unit Tests With Boost.Test, Part 1
C++ Unit Tests With Boost.Test, Part 2
C++ Unit Tests With Boost.Test, Part 3
C++ Unit Tests With Boost.Test, Part 4
C++ Unit Tests With Boost.Test, Part 5
You can also look at a workshop that I presented at C++ Now! 2014, which covers the use of the Turtle Mock library for mocking collaborators. I used CMake for the build system in the workshop as it was unreasonable to assume Visual Studio for that audience.
Working through either (or both) of those examples should give you enough step-by-step instructions that you should be able to identify what you are doing wrong in your project.
Related
When testing a Windows Universal class library (.NET 4.6) in Visual Studio 2015, the time required in the "red->green->refactor" cycle is quite long. My test project is just the standard MSTest "Unit Test App" project. Even in the simplest scenario of a brand new solution, brand new subject and test projects, and a basic int Add(int n1, int n2) method, it's taking about 8-15 seconds. This is the time (after making a small code change) from clicking "run test" till the pass/fail is shown.
On my machine (Win 10 pro) conducting the same experiment but with a WPF-based solution yields about 1-2 seconds.
In the UWP scenario, the actual test time itself is listed as 79 ms. The rest of the time is compiling and deploying the unit test app container.
Is there any way to significantly speed up TDD with UWP?
The reason the TDD cycle takes so long is because all UWP code has to run in an AppContainer, and that requires packaging and deployment which is slow.
The way around this problem is to separate as much of your code as you can into a different project that doesn't need to run in an AppContainer and then test that project instead.
The solution for this is to use a portable class library for your application logic instead of a UWP class library. You'll find the portable class library in the new project dialog:
The default settings should work:
You then need to add a reference from the UWP to the portable library so you can consume it. To test the portable library, use a regular non-UWP unit test project:
Note - to make this work you'll need to change the target framework of the unit test project from 4.5.2 (which is the default) to 4.6.
Any unit test you run from the regular unit test project will run as fast as possible because it doesn't need to run in the AppContainer. I tested this out and the inner loop speed was great. Hope that helps!
[I am very much new to the visual studio environment. After working with visual studio for 2-3 days I was able to get some knowledge about it.]
Well. I am working in a testing team. I just wanna setup a code coverage for cpp code developed in visual studio 2010. There are merely 2 doubts I wanna ask
1.My company has purchased visual studio 2010 ultimate edition and using that I was able to generate the code coverage for a sample test project (.dll) created using Test Project template in visual studio.
The problem here is that developers had written a test project using
cppunit as an .exe. So will I be able to get the code coverage for
an .exe too (with PROFILER enabled) using the usual method to get
the code coverage in visual studio ultimate edition. Or should I ask
them to write the unit tests using the Microsoft's unit test
framework(ie .dll). I just cant simply say do it, I need to present
them the genuine reason to do so.
2.The second thought I had was, there are lot more open source tools to get code coverage for cpp compiled using gcc or g++. So I thought of compiling the cpp code using g++ and use the open source tools to get code coverage. Is this a good idea?
When I shared this idea with a senior developer he said there are lot
more cpp code designed to be MSDN specific, you cant do by this way.If
you wanna do so you should make the code cross platform compatible or
portable[something I couldn't understand].
So here is the final thing, I am trying to present them a immediate solution of getting code coverage using the visual studio ultimate edition which we have. And later on search for the best way to achieve this without purchasing any paid tools like ncover, squish coco, bullseye coverage, etc only so I opted for the above mentioned second option.
My ultimate aim is to display the final report in the Jenkins so that everyone is able to see it without any manual intervention after the build process [which is not possible using the visual studio I guess,].
There is a plugin called MSTestRunner plugin for jenkins which can generate unit tests report but not the coverage report.
I think I made it clear. If this is a duplicate question please mention me the link to.Thanks in advance
I forgot to say this I was also able to find some tool called vcxproj2cmake I think it is clear from the name itself. So if i use this tool to generate a cmake corresponding to the actual vcxproj will i be able to the my second choice?
NOTE:
As for as now, I created a test project (lets say, SampleTestProject) as said in method 1 which has
only one test method (lest say, TestMethod) which in turn triggers all the unit tests written
using cppunit by the below mentioned method.
I asked the devs to make the .exe (which they have already with cppunit unit tests written) as .dll in a specific format like it
should contain a class called TestInitiator which contains the
CppUnit::TextUI::TestRunner (the runner instance which triggers the
cppunit tests, ideally this method should be present in order to trigger cppunit tests) with all the test methods defined in a suite.
This method is called by the above mentioned TestMethod in the SampleTestProject and thereby obtaining the code coverage
By this way the code coverage was obtained with very less change done to the code.Hope you like it!
Another day, another problem with VS2012 and unit tests: Our solution has three test projects, together containing 335 different unit tests. All of these tests successfully run locally on different machines, but when checked in to our TFS 2012 build server, each time only 235 tests are being discovered.
There are 15 different test classes containing the unit tests, some of them using the new Fakes framework, others MOQ, some none or both. All of the classes reside in test projects (assemblies) ending in Test, the classes themselves are all ending in Test.
Some classes are not being discovered at all during the test run as part of the build process, in other classes e.g. 8 out of 12 tests are discovered and run.
Taking the "8 out of 12" example, the [ProjectNameTest].dll that's being generated shows all 12 tests using .NET Reflector 7.0, it's just so that some of the tests aren't run. I also cannot find any similarities (e.g. tests using Fakes / MOQ / ...) among the tests not run / discovered.
The build settings being used define the Test Sources Spec as **\*.dll (it used to be **\*test*.dll, but I changed this just to make sure that all files are taken into account).
For the Run Settings I initially started of with no Run Settings File and CodeCoverageEnabled. As part of trying to solve the problems I also changed this to a user defined UnitTest.runsettings file, unfortunately with the same result.
Does anybody know why there are all these problems? What might be the reasons? VS2010 and Moles worked a lot better, VS2012 has been causing a lot of trouble recently.
Update
Months later we finally updated the TFS installation to Update 3, but this made things even worse. Unit tests on the build server don't run anymore at all. Enabling dumps and analysing them showed that a StackOverflowException is thrown, but there is no indicator about the test that might be causing it. Again, locally on different developer systems the tests run without any problems.
We are considering opening a call at Microsoft as we've spent far too much time getting the whole thing to work (again). Moles were a whole lot better... I will keep this updated as soon as there is some kind of progress.
Update 2
Another few weeks have passed, and no real solution yet. As we have moved from VS2012 to VS2013 (Premium each time) now, we discovered that all tests are now run as part of the build process (again). Unfortunately, we still have problems getting all tests to pass - locally, everything runs fine on multiple systems, on the build server about ten percent of test fail. Investigation to be continued...
Usually, when tests pass locally but not on the build server, you probably need to include your assembly dependencies or if your tests are using databases you are connecting to the database properly.
Finally an answer was found - the problems with test failing randomly on the TFS build server were caused by using the Microsoft Fakes Framwork. On our local systems, everything worked as expected - we had Fakes defined in our different unit test projects (generally one test project for each "normal" project).
On the build server all the fakes were generated in the same directory, therefore depending on the build order of the projects (which in parts may be different for each build), the fakes generated first were overwritten by the fakes from the following project etc.
Instead of having the fake definitions as part of every project in our solution, we changed it so that we have a solution wide definition of fakes (e.g. System.Web.Fakes.dll) that we link to from each test project where we need it (Add existing item => Add as link). This solved the problems with running the unit tests on the build server, all tests pass again.
I was able to execute the following code in VS 2010 but in 2012 the following does not work. Am i doing something wrong here.
[TestMethod]
public void UTestMethod()
{
TestContext.BeginTimer("t1");
Thread.Sleep(100);
TestContext.EndTimer("t1");
}
it throws NoSupportedException on method BeginTimer.
According to the following Microsoft blog, http://blogs.msdn.com/b/geoffgr/archive/2014/08/05/adding-timers-to-coded-ui-tests-running-in-load-tests.aspx, the ability to use timers in a Unit test was removed from Visual Stuido 2012. The blog states that the timer still works when a unit test is placed inside a load test, but standalone, the unit test will throw the System.NotSupportedException. The blog states that a workaround is to use the following line of code just before the beginTimer() call:
if (TestContext.Properties.Contains("$LoadTestUserContext")) //running as load
testContextInstance.BeginTimer("MyTimerName");
This approach is working on my installation of VS2012, and allows me to have the timers in my unit test and to run it as a single user without putting it in a load test. Removing the functionality in VS 2012 was NOT the best decision by Microsoft. I have found some cases where a timer in unit tests still does not work in VS 2013, but then sometimes it will work in 2013. In any case, in VS 2012, "unit test timers" are consistently broken but the above line of code provides a workaround.
IMHO, Microsoft should a) (preferred) restore the timer functionality to the way it was in VS 2010 or b) at least update the tooltip in intellisense to state that the timer functionality only works in a load test, and update the IDE to show a squiggly warning unless timers are associated with a $LoadTestUserContext variable as shown here. The change in VS 2012 gives Visual Studio load test scripting a confusing and irritating disadvantage in comparison with industry standard tools such as LoadRunner's Vugen.
The primary use of TestContext.BeginTimer is within loadtests. However I think you are running this within a standard unit test. So in this case yes this is a bug.
The workaround here is to use a .testsettings file while running your tests to allow you to get the functionality that you are familiar with.
Hope that helps
I have 80+ VS2010 solutions, each contains some unit test projects.
All solutions are merged into one big solution before build process.
All tests (or some test subset) are executed after the successful build.
I want to enable test code coverage for all tests to get exact code coverage for all output assemblies.
My question is: how to enable code coverage without using Visual Studio?
Note: I'm using TFS2010 (MSBuild) to build merged solution. VS 2010 Premium is installed on the build server. MSTest is used for test execution.
You can use JetBrain's TeamCity Professional. It is a CI server that supports executing unit tests and calculating code coverage. It is free for small installations.
I think you need to consider deploying a code coverage tool, see here for a comparison (provided you implement .net).We use NCover, which integrated in our TFS-Build in it's console variant and, although it's not trivial to set it up, we 're very satisfied with it.In this post I had briefly described how we inserted NCoverin our build, this might we useful to you even if you go with another tool.
If you create a Vsmdi file in your large solution (ms test will usually do this for you) you can use this to tell the build which assemblies you want to instrument.
This will only provide code coverage for assemblies that have tests run against them. If you're using testrun.config files to decide which tests you want to run, this should be all you need. The code coverage results should then be published to the build drop location
Edit:
This blog post looks like it covers setting up code coverage