Visual Studio Code does not recognize unit tests grouped in group(...). But it recognizes unit tests which are not grouped. I mean they are standalone test(...). I want to use group, because I need setUp().
There is Run | Debug link above group. When I click Debug or Run the Debug console shows No tests match regular expression "^LocalRepository$".
The tests are running correctly with flutter test command.
Dart SDK: >=2.1.0 <3.0.0
Flutter channel: master
Edit: I found workaround - I just don't use group callback. But I can't run all tests by clicking Run above group.
Make sure you name you test file ending with _test.dart and you will see the run | debug options
I have a Flutter project with two separate test files, both ending in "..._test.dart".
Using the following two test scenarios, one of the files passes, while the second doesn't load and that test fails.
Using launch.json configuration:
{
"name": "Dart: Run all Tests",
"request": "launch",
"type": "dart",
"program": "./test/"
}
Using the VS Code menu item Debug > Start Debugging (F5), which I suspicion just uses the above launch.json configuration, yields the same result.
However, using the VS Code menu item Debug > Start Without Debugging (Ctrl+F5), or "flutter test" from the command line, both tests pass.
I suspect this is an internal configuration issue with the VS Code Dart/Flutter extension, as opposed to a problem with Dart/Flutter code structure.
Are your tests/groups spread across multiple files? There are some limitations to the integration between the VS Code extension and the test package that you may be hitting.
There was a fix in v2.23 (released a few days before your question) that should have improved this a little (removing those Run/Debug links in some places they wont' work) but if you're already on that version then maybe it didn't cover your case.
If you can post a small repro on GitHub I'd definitely like to see if I can improve this. Thanks!
Ensure the test has a name different from ''
Change
test('', () {
...
});
to
test('xxx', () {
...
});
Related
I have a multi-module android project. I have a bunch of unit tests in each module and I have always been able to run them all at once using a run configuration like this one:
Many of my tests use a base class that runs with RobolectricTestRunner. This base class looks like this:
#RunWith(RobolectricTestRunner::class)
#Config(application = AndroidTest.ApplicationStub::class,
manifest = Config.NONE,
sdk = [21])
abstract class AndroidTest {
#Suppress("LeakingThis")
#Rule #JvmField val injectMocks = InjectMocksRule.create(this#AndroidTest)
fun application(): Application = ApplicationProvider.getApplicationContext()
internal class ApplicationStub : Application()
}
**When running these tests using the above config, I get the error **
[Robolectric] NOTICE: legacy resources mode is deprecated; see http://robolectric.org/migrating/#migrating-to-40
This makes many of my tests fail with ResourceNotFoundException
However, when I run tests only in a specific module, everything passes. This is because Robolectric now uses Binary resources:
[Robolectric] sdk=21; resources=BINARY
I have followed the migration instructions in build.gradle files for each module, having added the following in each android block:
testOptions {
unitTests {
includeAndroidResources = true
returnDefaultValues = true
}
}
One clue I have found but have been unable to fix is this when I run the ALL UNIT TEST task:
WARNING: No manifest file found at build/intermediates/merged_manifests/debug/../../library_manifest/debug/AndroidManifest.xml.
Falling back to the Android OS resources only.
No such manifest file: build/intermediates/merged_manifests/debug/../../library_manifest/debug/AndroidManifest.xml
To remove this warning, annotate your test class with #Config(manifest=Config.NONE).
I have tried, as you have seen, to add the manifest=Config.NONE, which had no effect (and is now deprecated anyway).
Edit: Also tried android.enableUnitTestBinaryResources = true in settings.gradle, but this prevents the app from building due to it being a deprecated flag in the current gradle tools.
Thanks for any help provided!
So with the default unit test run platform being changed to Gradle in Android Studio, I managed to figure out a way to run unit tests in multiple modules all at once, circumventing the Robolectric bug.
First, go into run configurations and create a new Gradle Config.
Then, as the gradle project, select the top level project.
For arguments, use --tests "*"
And now for the gradle tasks, this is a little bit more error-prone. Here is an example of how I have it setup for my project:
:androidrma:cleanTestGoogleDebugUnitTest :androidrma:testGoogleDebugUnitTest
:calendar:cleanTestDebugUnitTest :calendar:testDebugUnitTest
:gamification:cleanTest :gamification:test
:player:cleanTest :player:test
:playlists:cleanTest :playlists:test
:sleepjournal:cleanTest :sleepjournal:test
:sound-content-filters:cleanTest :sound-content-filters:test
Please note that I inserted new lines between each module for more clarity here, in the tasks, just separate each entry with a space.
For your app module, in my case named androidrma, you must use your build variants name in the cleanTestUnitTest and testUnitTest , in my case being GoogleDebug.
If we look at the calendar module, it is an android module, , so it still operates with the same logic as the appModule.
However, if you look at player, playlists, sleepjournal, etc. those are pure kotlin modules. The tasks thus differ in their syntax.
Once you have entered all this information and everything is functioning, I recommend checking "store as project file" checkbox at the top right of the run config setup screen.
This works in Android Studio 4.2 as well as Arctic Fox, haven't tested on other versions.
I'm using tfs 2017 to run CodedUI ordered unit tests. These are my build steps:
These are my "Run Functional Tests" configurations:
And these are the "Publish Test Results" (I'm not sure they are correct):
The TestAgent is deployed and the tests are running fine, The problem is that the test results appear as only one result and I can't see detailed result for each test. This is how my tests results looks like (The attachments are screen shots I take for each test):
I can reproduce your situation and TFS will treat the tests as one in the log:
DistributedTests: Total Tests : 1, Passed Tests : 0
It's a know issue, please refer below link:
How to display test results of individual test in the ordered test
suite in TFS Web
This is a limitation of the run functional test task. You can
publish the .trx file using the "publish test results" task and it
will show you all the tests but you wont know which ".orderedtest"
they were associated with etc.
You need either open *.trx file in Visual Studio or use Publish Test Results task(need to check continue on error).
Besides also change the outcome from failed to all in the test result page.
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
Currently I am building and testing my app in Jenkins. If a test fails in the job then the build is marked as Unstable. Is it possible to have the build be marked as Stable even though tests fail? The build is marked as Failed if code does not compile, and that is fine.
There is a plugin called Jenkins Text Finder:
https://wiki.jenkins-ci.org/display/JENKINS/Text-finder+Plugin
It allows to search console log for regular expression matches, and if found, it can set the build to various states, including setting it as "success" (i.e. not unstable).
Just echo a unique label/line into your console output after the build is successful, and look for that line with this plugin. Use "Succeed if found" option in the plugin to mark build successful.
I have a strange problem,
my MSBuild runs tests, code-coverage and publishing fine (part of the build.txt shown):
Results Top Level Tests
------- ---------------
Passed BuildTestProject.UnitTest1.TestMethod1
Passed BuildTestProject.UnitTest1.TestMethod2
2/2 test(s) Passed
...
Results file: W:\BuildWorkspace\XXX\Test Release\TestResults\XXX_XXX 2009-08-20 11_47_09_Any CPU_Release.trx
Run Configuration: Local Test Run
Waiting to publish...
Publishing results of test run XXX#XXX 2009-08-20 11:47:09_Any CPU_Release to http://XXX:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
....Publish completed successfully.
When I import these testresults on my local machine i get to see the code coverage-data as expected. But the code-coverage details are not shown in the details of the build that Visual Studio shows when you expand the 'results details'.
Any tips?
Finally found the solution today, found out that my TeamFoundationServer itself had problems with publishing the results to my buildAgent. I read somewhere (after searching for CoverAn.exe) that it is installed as a service, then i checked the credentials for the service 'Code Coverage Analysis Service'.
These were running under TFSMachine\NETWORK instead of our service-account for the TeamFoundationServer. Changed this, reran a build with tests in it and it now publishes the results.
W00t!
Check that the .testrunconfig used by the server has coverage enabled. It may be different than your local .testrunconfig.