I'm using tfs 2017 to run CodedUI ordered unit tests. These are my build steps:
These are my "Run Functional Tests" configurations:
And these are the "Publish Test Results" (I'm not sure they are correct):
The TestAgent is deployed and the tests are running fine, The problem is that the test results appear as only one result and I can't see detailed result for each test. This is how my tests results looks like (The attachments are screen shots I take for each test):
I can reproduce your situation and TFS will treat the tests as one in the log:
DistributedTests: Total Tests : 1, Passed Tests : 0
It's a know issue, please refer below link:
How to display test results of individual test in the ordered test
suite in TFS Web
This is a limitation of the run functional test task. You can
publish the .trx file using the "publish test results" task and it
will show you all the tests but you wont know which ".orderedtest"
they were associated with etc.
You need either open *.trx file in Visual Studio or use Publish Test Results task(need to check continue on error).
Besides also change the outcome from failed to all in the test result page.
Related
I am working on a java pluging which is built from a Jenkins project, the build is done using gradle.
From times to times, we may have to skip a test method, or a test overall. I do that by using the #Disable annotation.
However, in this case, the test method, or test, simply disappear from the xml report. I expect the test to be reported as "Skipped" in the xml report so Jenkins can know it was skipped and display it in yellow.
Any solution to that ?
If you are using the junitPlatform() support in Gradle, Gradle will generate XML output similar to the following showing that a test has been skipped.
<testcase name="enabledAndDisabled()" classname="org.junit.jupiter.extensions.EnabledIfTests" time="0.0">
<skipped/>
</testcase>
I copied the above from the TEST-org.junit.jupiter.extensions.EnabledIfTests.xml generated in the platform-tests module of the JUnit 5 build.
I am using Nunit4TeamBuild to try and publish the results of some unit tests back to TFS 2010.
When I try to publish a new test results XML the build is successful and it says the test results have been published. However there are no test results anywhere to be found in the build summary/log.
It even says "No Test Results" in the build summary beneath the compilation step.
So even though it was successful, it doesn't look like the test results actually get published.
But if I run another build definition that just takes a test result XML and publish it, I get this error:
Test run already uploaded.
So it would seem that the test results were, in fact, published to TFS. But where? How can I see them? Why don't they show up in the build summary/log like it's meant to? I cannot find the results anywhere.
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
How do you run your code to run through all scheme test cases in xcode4?
"the scheme '/name/' is not configured for testing. Edit the scheme to configure testing,... "
But I cant seem to set it up to test all the scheme cases.
Best to use the schemes that Xcode installed when you created the project: they have the test phase already configured. To execute the test, choose menu Product -> Test.
I have a strange problem,
my MSBuild runs tests, code-coverage and publishing fine (part of the build.txt shown):
Results Top Level Tests
------- ---------------
Passed BuildTestProject.UnitTest1.TestMethod1
Passed BuildTestProject.UnitTest1.TestMethod2
2/2 test(s) Passed
...
Results file: W:\BuildWorkspace\XXX\Test Release\TestResults\XXX_XXX 2009-08-20 11_47_09_Any CPU_Release.trx
Run Configuration: Local Test Run
Waiting to publish...
Publishing results of test run XXX#XXX 2009-08-20 11:47:09_Any CPU_Release to http://XXX:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
....Publish completed successfully.
When I import these testresults on my local machine i get to see the code coverage-data as expected. But the code-coverage details are not shown in the details of the build that Visual Studio shows when you expand the 'results details'.
Any tips?
Finally found the solution today, found out that my TeamFoundationServer itself had problems with publishing the results to my buildAgent. I read somewhere (after searching for CoverAn.exe) that it is installed as a service, then i checked the credentials for the service 'Code Coverage Analysis Service'.
These were running under TFSMachine\NETWORK instead of our service-account for the TeamFoundationServer. Changed this, reran a build with tests in it and it now publishes the results.
W00t!
Check that the .testrunconfig used by the server has coverage enabled. It may be different than your local .testrunconfig.