Is there a reliable way of measuring code coverage for .Net Core projects (in particular .Net Core 2) in Linux hosts?
I have looked at coverlet and minicover but they still have some issues that don't allow me to use them.
VSTest command is working on that issue:
https://github.com/Microsoft/vstest/issues/981
OpenCover issue: https://github.com/OpenCover/opencover/issues/601
Coverlet is the most active project.
Read excellent article of Hanselman: https://www.hanselman.com/blog/AutomaticUnitTestingInNETCorePlusCodeCoverageInVisualStudioCode.aspx
AltCover is also very interesting project that works in .net core.
Related
This might be a very specific question. But it might be interesting for some other people, too.
The setting might be quite common: we develop an embedded device using C++ and Linux and have a build system for this. With our device we ship a C# desktop app using a DLL based plugin mechanism. The plugin contains the shared logic used both on the embedded device and the app. To keep processes easy it would be best to create the DLL on our Linux build system (hence I would like to use .Net Core and not "normal" .Net).
As far as my trial and error showed it's not possible - even with .Net Core 3.1. Can you confirm this? The software is for windows only (although it's compiled on Linux) - so theoretically it could possible since .Net Core supports CLI/C++ in Version 3.1 (https://devblogs.microsoft.com/dotnet/announcing-net-core-3-1/).
What I tried:
I compiled pure C# code on Linux to an exe running successfully on windows using the console app dotnet
I successfully created a simple C++/CLI test app on Windows using sample code following this guide: https://devblogs.microsoft.com/cppblog/an-update-on-cpp-cli-and-dotnet-core/
Finally I took the (probably for too naive) approach to replace the test app's .vcxproj with the .csproj of step 1 (.csproj seems to detect files automatically). I got the following error message:
quote CSC : error CS5001: Program does not contain a static 'Main' method suitable for an entry point [/home/vagrant/Projects/cpp_dotnet_on_linux/cs_on_linux.csproj]
Is there something I can improve or is the task just impossible (adding the cpp-files manually to .csproj, ...)? I'm afraid it's just technically impossible because I guess .Net Core is missing a C++ compiler on Linux but I'm not sure.
C++/CLI is not support on Linux.
Some sources:
https://devblogs.microsoft.com/cppblog/the-future-of-cpp-cli-and-dotnet-core-3/ :
We don’t currently have plans for C++/CLI for targeting macOS or Linux. Additionally, compiling with “/clr:pure” and “/clr:safe” won’t be supported for .NET Core.
https://github.com/dotnet/coreclr/issues/659#issuecomment-539742740
C++/CLI on Linux will not be supported and would be very challenging. It would require Microsoft VC++ to support Linux or require Clang or GCC to support C++/CLI. Those are both huge projects with uncertain payoff. It is also unclear if Clang or GCC would ever allow us to upstream our changes. Maintaining an up-to-date fork of a fast moving compiler project forever is very expensive. We have no plans to take on either of those projects.
I'm planning to use cpputest as a testing framework in my project which I need to cross compile as it will be used on ARM platform. The compiler I'm using for ARM development is arm-gcc which is built with pthreads disabled. Due to this, I need to build cpputest without pthreads. Currently l am following the autotool approach for building cpputest. Any help would be really appreciated.
Are you trying to compile, download, and run CppUTest on a target device?
Typically, CppUTest isn't compiled and downloaded to your target device but instead it's built as a Windows (or Linux) console application. This means that CppUTest gets compiled using a native compiler (Visual Studio or GCC) but your firmware will also need to compile using that same native compiler (Visual Studio or GCC).
Unit testing is meant to verify that your code's logic is correct, which means it doesn't matter if the code is executing on an ARM processor or in an Intel processor. For the record, when I first started getting into unit testing this blew my mind.
But what about hardware registers and stuff like that? Well, there's ways around testing embedded devices hardware in a Windows environment. At the end of the day, you'll still need to download your firmware onto the embedded device and verify that your port mappings are correct (i.e. That you're properly settings up a GPIO port to turn on an LED for instance).
There's several advantages to running your unit testing on your development PC instead of on target. Your development PC is much faster, has more resources (i.e. storage and RAM) and it also easily allows for running in a continuous integration environment (like Jenkins or TeamCity) whenever you check-in code to your version control system.
I highly recommend the book Test-Driven Development for Embedded C by James W. Grenning for more information. It will answer all your questions on unit testing on an embedded device. CppUTest can be used for C as well as C++ projects.
EDIT 2012.03.02: updated build tool section with some other requirements. Addding premake.
EDIT 2012.03.05: updated build tool section. Focused on CMake. Added Ant + CMakeAnt.
I have a lot of small Visual C++ projects and I want to start using continuous integration for that. A lot of tools are involved in the process. I start reading articles here and there and it is hard for me to see which to tool could integrate nicely with the others. So I am looking for an already tested continuous integration workflow that could respect my needs:
Current Context
C++/Qt applications developed with Visual Studio;
Less than 1'000 source files;
Small dev team;
Small budget;
Preferably open source/free software;
Looking for easy to use and simple tools. No need of something heavy and "powerful".
Finally packages software (installers ready for download) should work for windows and OSX. Linux (most popular distributions) is a plus.
Tools needed
Continuous Integration Server:
Needs to be Jenkins. I know it and I like it. I could change for another one if it is really needed.
Source Code Management:
GIT.
Documentation Generator:
Doxygen.
Build Tool:
We will need to generate solutions and configurations for all the platforms we need to support. I have been strongly advised to use CMake for that. Ant + CMakeAnt (http://code.google.com/p/cmakeant/) seems to be a nice addition too.
Unit Testing Framework:
CPPunit, C++Test, Googletest?
Installers Builder
CPack (part of CMake seems great too).
Hardware:
Preferably, one single machine with VirtualBox or VMWare. is it only possible? Would it be better to use Linux for such things? Or would a MAC mini be sufficient?
So, has anyone here already built something similar? With which tools exactly?
Thanks for any comments and suggestions.
I have successfully used CMake/CTest/CPack with Jenkins, there is a CMake builder plugin available.
I found the ctest --output-on-failure option to be useful as the output from the test is shown in the Jenkins log on failure.
I use the Boost.Test framework for testing C++, it works fine with ctest, I run it with the -l all option to get all the logging output when something goes wrong.
You should be able to bootstrap the entire process using a bunch of scripts.
Write them in Perl or Python. There are a million tutorials on how to bootstrap these sorts of processes.
If you're stuck on implementing a specific party of this build-process, ask a specific question about it.
What code coverage tools have you used with Symbian C++ and Maemo? What are the pros and cons of the tool you are using?
On Symbian I've used BullseyeCoverage and Testwell CTC++. Cannot really describe the pros/cons of them in detail. Both got the job done, eventually. Both needed some effort with setup and integration with an automated test suite. Both contained bugs that e.g. crashed the downstream compiler with slightly broken instrumented source code.
On Maemo, since the toolchain is GCC based, I'd guess gcov would be a good starting point. Though I haven't been working on Maemo much yet and haven't done any coverage measurement there.
See SD C++ Test Coverage for a tool that has extremely low overhead and works very well in embedded environments.
I have used Bullseye Coverage on Symbian and it is very good. The only problem is that it only runs on emulator and not hardware. Therefore you would not be able to get coverage metrics from a device or devboard. If your app runs on both hardware and emulator, this won't be a big deal (apart from the standard differences between running on emulator vs hardware). Also, as there are plans to replace the emulator with a proper hardware simulator, perhaps bullseye wouldn't be the best choice.
gcov can be used (and is used) in maemo platform and tool called lcov can then be used to generate "pretty" reports.
However, in order use gcov in maemo sdk, you need to disable compiler cache during the build time when you are creating binaries for coverage execution..
I have an Open Source app and I have it working on Windows, Linux and Macintosh ( it's in C++ and built with gcc ). I've only tested it on a few different flavors of Linux so I don't know if it compiles and runs on all different Linux versions. Is there a place where I can upload my code and have it tested across a bunch of different systems like other Linux flavors and things like, Solaris, FreeBSD and other operating systems?
What would be great is if I can have it directly connect to my svn repository and grab the latest code and then email me back any compile errors generated and what the OS was that it had a problem with.
I would be happy to just know it compiles as it is a GUI based app so I wouldn't expect it to actually be ran and tested.
There are a few options but there don't appear to be many (any?) free services like this, which isn't surprising considering the amount of effort and resources it requires. Sourceforge used to operate a compile farm like what you describe but it shut down a year or so ago. You might look into some of the following. If you're inclined to pay for a service or roll your own, then some of these links may be useful. If you're just looking for a free open source compile/build farm that covers multiple platforms it looks like you're pretty much out of luck.
OpenSuse Build Service
Mentioned by Ted first, worth repeating - only for Linux currently but does support a number of distros.
GCC Compile Farm
Mainly focused on testing builds for GCC but does also host a few other projects such as coLinux, BTG BitTorrent client, ClamAV, and others. May be something you can take advantage of, though I don't see what OSes are in the compile farm (contains at least Linux and Solaris based on the page notes).
BuildLocker
BuildLocker is a Web-based continuous integration solution for Java and .NET projects. BuildLocker is a virtual dedicated build machine that helps teams find bugs earlier in the development cycle, saving time and money. BuildLocker manages scheduled automated builds of source code in your ProjectLocker Source Control repository. Just check in the source code, and scheduled builds validate the integrity of the code. BuildLocker can even run automated tests, and can alert you anytime a test fails.
CruiseControl
CruiseControl is a framework for a continuous build process. It includes, but is not limited to, plugins for email notification, Ant, and various source control tools. A web interface is provided to view the details of the current and previous builds.
Interesting side note, CruiseControl is actually used by StackOverflow's dev team for automated build testing as well, according to the podcast.
Hudson
Hudson monitors executions of repeated jobs, such as building a software project or jobs run by cron.
RunCodeRun
Mentioned in the other linked question, only supports Ruby projects and is in private beta currently. However, if your project is in Ruby, it might be worth keeping an eye on RunCodeRun.
CI Feature Matrix
There are many Continuous Integration systems available. This page is an attempt to keep an unbiased comparison of as many as possible of them.
Take a look at the OpenSuSE build service, it includes a fairly wide variety of Linux distros (not just SuSE/OpenSuSE).
From a software point of view, there's also buildbot (sourceforge project site), which can be used to set up your own build/continous integration server.
This was suggested and considered to be used for gcc development (as mentioned on the gcc compile farm wiki page posted above).
If you are planning to go commercial with your open source product you might consider our Parabuild. It allows you to run a set of builds on multiple platforms and machines in parallel. The build will success only if all platform-specific builds success.