How to do Coverage exports for vscode extensions - unit-testing

I have seen a lot of tutorials how to make vscode extensions. Like:
https://code.visualstudio.com/docs/extensions/testing-extensions
And there is many tutorials how to do coverage exports, there many ways how to do it, but I didn't seen good examples which would follow the examples from their docs and work with vscode extensions (they need the extensionHost instead of nodejs).
I have all the tests written in the mocha which is bundled in the vscode as proposed by their documents.
I tried to experiment and I'm stuck, are there any hints or directions where I could continue and get my process working again?

The answer which worked is in the comments to my original question. But to simplify and boil down what had to be done, and what steps I did personally. Not all steps are really essential and required, but these steps made it pretty convient:
installed istanbul, istanbul-coveralls, gulp, gulp-json-editor, coveralls to my package as the DEV dependencies
included the MS supplied istanbul test runner for vscode extension testing into my tests
modified the tests to run an istanbul runner instead of mocha runner directly (the tests itself could stay as they are).
attached reference to the istanbul runner now I had to read the coverconfig.json to know how to do the coverage itself (what parts to ignore, where is source, where to output results, what formats the results should be etc...).
added coverage to gitignore and to vscode project as so they will not annoy me in the UI and will not polute the repo.
added few visual studio launchers/tasks to make it easier for me to trigger them from the UI
installed vscode plugin to display the coverage in the gutters: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters and then added to my project settings the "lcov.path": [ "coverage/lcov.info" ] (or whatever location you will have the line coverage) so the plugin can read and show the coverage directly in the UI.
setup gulp to delete old coverage results and start new ones on each run
added npm scripts for regular tests and for the coverage variant
changed travis steps so the coverage variant is running now instead of the generic non-coverage one. Added script step so the coveralls module is run and broadcasts the results from the coverage/lcov.info to the https://coveralls.io/ website. (i think signing in and doing the first-time setup is required to make it work on their website)
Now on each commit travis will do the work by itself and I can use coveralls banner inside my readme to give quick glipse how well is my coverage. And you can use gulp to watch it for changes and build/test locally and have fast feedback in the UI what lines were covered by your tests.
The setup is similar to the listed links:
https://github.com/Microsoft/vscode-mssql
https://github.com/kenhowardpdx/vscode-gist/pull/10

Related

What Build Step should be used for running QUnit tests in TeamCity?

I've got a TeamCity build configuration with two steps currently:
MSBuild
Execute powershell sciprt
I want to add another one:
Run QUnit tests
Inside Visual Studio I use Chutzpah to run my QUnit tests. It seems that there is no "Runner type" appropriate though for QUnit tests, and because I do see one for NUnit I'm left wondering if I'm looking at this right.
Should I get a TeamCity extension to add a "Runner type" for QUnit tests? Should I create a powershell script that runs chutzpah-console? And if so how should I make sure the test output is picked up by TeamCity (do I need the /nunit2 or /junit flag?)?
I've done some research, allow me to share...
I've applied my Google-fu but that leads to a blog post with complicated QUnit tests with special TeamCity messages (or perhaps I misunderstood?) and a blog post on using a command-line build step to call PhantomJS that uses a rather contrived html file that seems like overkill to me if I'm already using Chutzpah in Visual Studio anyways (or perhaps I misunderstood)?
I've gone through the Chutzpah Full Documentation but as far as CI goes it only has a section on TFS Build, nothing on TeamCity or CI in general. I've carefully looked through the command line options documentation, and it does have one relevant option:
/teamcity :Forces TeamCity mode (normally auto-detected)
This suggests that you normally don't need to do anything to get things working, but that's certainly not the case: my sln (or better: csproj) contains several js files with QUnit tests, but TeamCity shows no tests.
So, how do I get TeamCity to run QUnit tests in the (c)leanest way possible / how do I use Chutzpah in a TeamCity build step?
No need for "service messges" or custom html files that go into a special PhantomJS call if you're already using Chutzpah. Here's one (other) way to do it in a (c)lean way.
There is no specific runner type for this type of build step. But no need for a custom Powershell script either: it'll be one line of script only. You can just use a Command Line step to invoke Chutzpah. To be more complete, assuming you've got TeamCity running already on a Windows machine, follow these steps:
Install chutzpah.console.exe. The easiest way is probably using chocolatey to install the Chutzpah package using this command in Powershell:
choco install chutzpah
This installs Chutzpah and makes sure the exe is in your path (you may need to reboot TeamCity parts to get this noticed!).
Note that alternatively you could also manually get chutzpah.console.exe on your CI server somehow (e.g. grab it from your dev machine) and reference it with a full path or place it in your path yourself.
Add a Command Line build step with a custom script like this:
chutzpah.console.exe "MyProject/UnitTestsSubDir"
E.g.:
And by some black magic this is all it takes: your tests will be run (see the build log) and TeamCity will pick up the test results.

JaCoCo test coverage of ant based Netbean Module project

We have inherited a NetBeans Modules project (a.k.a. NetBeans Platform project) which builds using the NetBeans Ant tasks. We want to generate a code coverage report of the project to get an idea of what components are not being unit tested.
Generating a report is not part of the build process, it is simply something we want to do to understand the existing unit test coverage in the project.
We've decided to use JaCoCo to try and generate this. However the native support in NetBeans is for maven based projects. Unfortunately the TikiOne JaCoCoverage plugin currently does not support NetBeans Modules projects.
We are happy to just drive this via CLI to get an initial report. Does anyone have a success means of doing so under the given constraints?
Someone recently posted a related message to one of NetBeans mailing lists - https://netbeans.org/projects/platform/lists/dev/archive/2014-04/message/124
Short version: Cobertura plugin that does what you're looking for is not maintained now but you can build it and run it.
We have been successful in generating the report by doing the following:
Edited the project.properties file to run the JaCoCo agent when unit tests run. This is done by adding the following:
test.run.args=-javaagent:[yourpath/]jacocoagent.jar=destfile=[yourpath/]jacoco.exec
Use Netbeans to run the test task. This produces the jacoco.exec file.
Create a simple ant buildfile to generate reports from an .exec file (see this post)
Run the ant buildfile to generate the report.

How to enable code coverage without using Visual Studio?

I have 80+ VS2010 solutions, each contains some unit test projects.
All solutions are merged into one big solution before build process.
All tests (or some test subset) are executed after the successful build.
I want to enable test code coverage for all tests to get exact code coverage for all output assemblies.
My question is: how to enable code coverage without using Visual Studio?
Note: I'm using TFS2010 (MSBuild) to build merged solution. VS 2010 Premium is installed on the build server. MSTest is used for test execution.
You can use JetBrain's TeamCity Professional. It is a CI server that supports executing unit tests and calculating code coverage. It is free for small installations.
I think you need to consider deploying a code coverage tool, see here for a comparison (provided you implement .net).We use NCover, which integrated in our TFS-Build in it's console variant and, although it's not trivial to set it up, we 're very satisfied with it.In this post I had briefly described how we inserted NCoverin our build, this might we useful to you even if you go with another tool.
If you create a Vsmdi file in your large solution (ms test will usually do this for you) you can use this to tell the build which assemblies you want to instrument.
This will only provide code coverage for assemblies that have tests run against them. If you're using testrun.config files to decide which tests you want to run, this should be all you need. The code coverage results should then be published to the build drop location
Edit:
This blog post looks like it covers setting up code coverage

Run xUnit tests from TFS2010

I'm trying to work out how to run xUnit tests from TFS 2010.
I found some articles on how to achieve that with the older version (http://jonnekats.wordpress.com/2009/05/07/integrate-xunit-tests-into-your-daily-team-build/, http://weblogs.asp.net/mehfuzh/archive/2009/08/25/configuring-team-build-using-xunit.aspx).
These approaches don't work anymore with TFS 2010 because builds are no longer MSBuild files but instead are Workflows with different tasks.
What I would like to achieve is something similar to what I've got on my dev machine:
- build everything
- run the .xunit project file
- check the results
All that I can see on the web are custom build steps which I can't/won't use because I'll have to configure them for each single unit test assembly and they'll probably get completely messed up in TFS.
Any input would be appreciated.
Why dont use "InvokeProcess" in TFS Build workflow and invoke xUnit command line interface with the necessary xUnit project files as parameters?
You can capture the output of the command line by setting InvokeProcess->stdout to write a build message.
Take a look at the xUnit work item Support VSTS Test Runner on codeplex. You can find there a VS2010 extension in Beta version.

Running VSTS tests without mstest.exe

From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.