All tests are executed although no code changes - unit-testing

In vNext (TFS 2017.3 on prem) I have a build definition which builds a .NET application and run around 3000 tests.
In my Visual Studio Test task I have enabled "Run only impacted tests" to decrease build time. I have done several builds at the same branch (Git) where I know there are no code changes. So my expectation was that the first build would take more time that if the "Run only impacted test" wasn't enabled because it should do extended analysis.
But then in the next build I would assume that no tests were executed as the code haven't changed.
But what is actually happen is that only 14 tests are "Not impacted".
Do you know why?

TIA is presently scoped to the following:
Supported
TFS 2017 Update 1 onwards and on VSTS
Version 2.* of the Visual Studio Test task in the build definition
Build vNext, with multiple VSTest Tasks
VS2015 Update 3 onwards on the build agent
Local and hosted build agents
CI and in PR workflows
Git, GitHub, External Git, TFVC repos
IIS interactions (over REST, SOAP APIs), using HTTP/HTTPS protocols
Automated Tests
Single box topology (tests and application must be running on the
same machine)
Managed code (any .NET Framework application, any .NET Service)
Not yet supported
Multi box topology (where the test is exercising an app deployed to a
different machine)
Data driven tests
Test Adapter-specific parallel test execution
.NET Core
UWP
Please check whether you have a supported scope. More details please refer to the blog below:
https://blogs.msdn.microsoft.com/devops/2017/03/02/accelerated-continuous-testing-with-test-impact-analysis-part-1/

Related

Wanted to create UI interface for Business developer team which contains Test Cases

I wanted to create UI interface for Business developer team which contains Test scripts which are written in my PC in selenium with c#.
Problem :-
Business developer team has no visual studio installed in their PC
because they don't have licence for it.Now if they wanted to show
demo of Application to client then if they have UI which contains
Test scripts which i wrote in selenium.so how could i display those
Test script on business developers laptop which don't have visual
studio?
Here for demo purpose to client ,Business developer team will
click on particular test case on UI and run that feature. For
example ,If Business developer wanted to show Login feature in UI at
Business developer laptop then that script should run.
how can i define connectivity between business developer's laptop
which don't have visual studio and source laptop which has visual
studio which contain Test script?
It's kind of R & D work,If anybody has idea about this let me know.
To do this, you are going to have to setup some kind of infrastructure to support this. You could achieve this through setting up a build server and making a build plan per collection of test scripts. This is going to let the BDs run the tests, but not look into the tests till the execution is done. You will require proper reporting in order to do so.
What you could also do is use a tool like browserstack. With (I am not affiliated with browserstack what so ever) browserstack you can make every executed test viewable through their recording function and every step of every test will be administrated. This way, you execute the test once, they login into browserstack and check every test, test step, expected results, actual results, screenshots and 30 day history recording of each and every test executed.
You don't need an IDE to run your tests. The person who wishes to run tests to show off some features of the site should be able to download your test codebase and run it locally from the command line. If they don't want to pass in a lot of flags at the command line, you could bundle the commands into batch or shell scripts.

How to use Jacoco with application that requires install and configuration to run

I am attempting to set up Java code coverage for a fairly complex app that
combines multiple large modules, only one of which I need to check coverage on
uses a combination of ant and Maven for builds
cannot be run except as an installed application on a server, with configuration
the automated tests to be analyzed for coverage are not part of the application build and make use of API calls to the application server from a remote client
The examples given in the jacoco documentation and in the online sources I have found assume the app under test is not previously installed and the tests are unit/integration tests run as part of the build. The documentation does not cover the details of how the jacoco instrumentation is done or when the call is recorded to a particular line of code. If I use ant or maven to instrument a particular module, use that module to build the full app, install it on a server, and configure it, will my remote tests then generate the .exec file?
Any advice on how to achieve the end goal (knowing how much of our code is covered by the tests) is greatly appreciated, including better search terms than "jacoco for installed app" which as you can imagine is ... not very useful. My google-fu is humbled.

How to trigger a TeamCity project to deploy to a server

I am using TeamCity (version 9.1.5 if that matters) and I am trying to figure out how to create a trigger that deploys the project to a server. Or maybe there is a way to deploy a project to a server without using a trigger on TeamCity.
It's a very broad question, but I will share the approaches I have used in a couple of scenarios:
1) To deploy when a code checkin is performed, I have setup a build configuration that does the deployment, added the build configuration that does the compiling & packaging as a snapshot and artefact dependency which is then triggered with a Finish Build Trigger https://confluence.jetbrains.com/display/TCD9/Configuring+Finish+Build+Trigger
2) To deploy at a given time of the day but only when new code has been checked in, I have setup a build configuration as above but triggered with a Schedule Trigger https://confluence.jetbrains.com/display/TCD9/Configuring+Schedule+Triggers ensuring to select the dependent build in the Build Changes section.
With regards to how to perform the deployment there are many options, I have used WebDeploy for ASP.Net applications and MSI packages executed by Remote Powershell scripts for Windows Services, but other options are also available depending on the technology you have.
JetBrains provide an end to end example for ASP.Net in their on-line documentation, search for "Continuous Delivery to Windows Azure Web Sites (or IIS)"

TFS 2012 Auto-Deploy Process

I am trying to improve our general automation process. We use VS2012 and TFS2012.
Here is what I want to happen upon checkin to our CI branch:
BUILD
Build the selected projects / solutions as configured in the build definition settings.
Generate a deployment package that can be used to deploy the websites (without having to rebuild the entire project again)
Generate a nuget package that can later be published (without having to rebuild the entire project again, i need the dlls to match the symbols created from indexing so we can debug them)
TEST - IF AND ONLY IF BUILD WAS SUCCESSFUL
Run all configured unit tests.
DEPLOY - IF AND ONLY IF ALL UNIT TESTS PASS This is to prevent breaking changes entering our development environment
Take deployment package from (1.2) and publish it to it's intended environment (hopefully configured using Publishing Profiles and transforms)
PUBLISH - IF AND ONLY IF ALL UNIT TESTS PASS
Take nuget package from (1.3) and publish it to our private nuget gallery
I don't need a full tutorial (although that would be awesome) for the entire process, but more how to go about integrating it.
For instance:
Should I use msbuild on a wrapper project?
How do I deal with creating the packages upon build on the TFS build server?
How can I enforce the "IF AND ONLY IF ALL UNIT TESTS PASS" constraints?
What is the best / easiest way to perform the deployment /publishing after as part of the build.
This is the process we want to use and any help is realising this is very much appreciated.
And I'm sure many other people are interested in how to set about integrating this style of process.
Also if it's relevant most solutions have a mix of shared dll projects, websites / apis, and unit tests. One of the reasons I want this process is to be able to split them up and modularise our large dlls into smaller isolated units, which would be to unmanageable ATM without this auto publish mechanism.
Thanks,
Gary.
BUILD Build the selected projects / solutions as configured in the build definition settings. Generate a deployment package that can be
used to deploy the websites (without having to rebuild the entire
project again)
This is out of the box, add deployment profile to your projects, call them 'Release'
Add the following to your MSBuild Arguments
/p:DeployOnBuild=true;PublishProfile=Release
you don't have to use Release, as long as your Publish Profiles match what you put in the MSBuild arguments
This will generate the deployment files as part of your build (MSDEPLOY)
Generate a nuget package that can later be published (without having to rebuild the entire project again, i need the dlls to match the symbols created from indexing so we can debug them)
See Nugetter on code plex http://nugetter.codeplex.com/
TEST - IF AND ONLY IF BUILD WAS SUCCESSFUL Run all configured unit
tests.
Should be out of the box, but you can change the build template to fail the build should compilation be unsucessful, if this suits your needs better.
DEPLOY - IF AND ONLY IF ALL UNIT TESTS PASS This is to prevent
breaking changes entering our development environment Take deployment
package from (1.2) and publish it to it's intended environment
(hopefully configured using Publishing Profiles and transforms)
PUBLISH - IF AND ONLY IF ALL UNIT TESTS PASS Take nuget package from
(1.3) and publish it to our private nuget gallery
See Nugetter on codeplex as listed above

How to add profiling to unit test cases in Teamcity?

Is it possible to add memory/performance profiling of unit tests as a team city build step?
I am specifically interested in doing it for our .Net apps (test cases are written in NUnit). Any elaboration on this will be highly appreciated.
Teamcity version is "TeamCity Enterprise 7.1.2 (build 24170)"
Thanks.
As far as I know, TeamCity does not support profiling .Net apps yet. At least not in a direct way.
There is an open feature request TW-20190 planning to integrate dotTrace (a .Net profiler made also by JetBrains) in Teamcity.
But dotTrace can be called from command line, so you can write MSBuild/NAnt script to execute dotTrace. The problem is about how to display the profiling results. In "General Settings" page of your build config, you can point artifacts path to the dotTrace result folder, so that teamcity will publish the profiling results as artifacts for you to download.