I have a pure Rust CLI app that I'm unit testing, and if I run
cargo clean
cargo test --release
it fails to find the binary, but if I run
cargo clean
cargo build --release
cargo test --release
it works. Is there a way to tell cargo not to run the test until the binary app has been built?
Here's my test harness:
#[cfg(test)]
pub mod tests {
use std::process::Command;
use assert_cmd::prelude::{CommandCargoExt, OutputAssertExt};
use predicates::prelude::predicate;
#[test]
fn test_that_cli_app_produces_result() -> Result<(), Box<dyn std::error::Error>> {
let mut cmd = Command::cargo_bin("fastsim-cli")?;
let mut cyc_file = project_root::get_project_root().unwrap();
cyc_file.push("../fastsim/resources/cycles/udds.csv");
assert!(cyc_file.exists());
let mut veh_file = project_root::get_project_root().unwrap();
veh_file.push("../fastsim/resources/vehdb/2012_Ford_Fusion.yaml");
assert!(veh_file.exists());
cmd.args([
"--cyc-file",
cyc_file.to_str().unwrap(),
"--veh-file",
veh_file.to_str().unwrap(),
]);
cmd.assert()
.success()
.stdout(predicate::str::contains("32.4"));
Ok(())
}
}
From the #[cfg(test)] mod tests { I am guessing that this is code that exists inside of your library or binary source files. That is not the correct location for a test which needs to run a built binary. Instead, you should write a so-called “integration test” which are located in the tests/ directory next to src/, and are separate compilation targets (so that they can be built after everything else). Cargo specifically provides for testing binaries via integration tests:
Binary targets are automatically built if there is an integration test. This allows an integration test to execute the binary to exercise and test its behavior. The CARGO_BIN_EXE_<name> environment variable is set when the integration test is built so that it can use the env macro to locate the executable.
If you're already using an integration test, then something else has gone wrong that isn't obvious from your description, because Cargo should be building the binary, as per the documentation.
Related
I'm using VSCode and the Ionide suite of packages to create a console application in F#. I need to add unit tests to the application so that when I ctrl+shift+p FAKE: Build the project, the tests are run during the build process.
I've created a dummy project in Github as an example.
Initially, the test dir was not there. I created the test dir and into that folder created a second project TestProj.Test (in hindsight, I should have used more descriptive names) for testing purposes. I added the .fsproj file from TestProj to this project so that I could reference the SimpleFunctions.fs. NUnit.Framework and FsUnit are added to the TestProj.Test. Test.fs contains two simple tests.
I intentionally created the TestProj.Test as an F# library because I read on SO that the testing project needed to be a library rather than a console app.
I added lines 9, 31-37, and 47 to the default build.fsx file that comes from Ionide.. However, when I build the whole project (i.e., TestProj), the build fails and I get the following error:
1) System.Exception: NUnit: cannot run tests (the assembly list is empty).
at Fake.NUnitSequential.NUnit(FSharpFunc`2 setParams, IEnumerable`1 assemblies) in C:\code\fake\src\app\FakeLib\UnitTest\NUnit\Sequential.fs:line 22
at FSI_0005.Build.clo#31-3.Invoke(Unit _arg3)
at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 492
Line 22 of the Sequential.fs suggests that assemblies is empty.
What am I doing wrong? How should I set up the build.fsx file so that the tests in TestProj.test run successfully? Alternatively, is there something wrong with the Tests.fs file in TestProj.Test? This seems particularly difficult; is there an easier way to include tests that run automatically with VSCode, Iondide, and F#?
There are a few issues in your project:
trying to test before build "Clean" ==> "Test" ==> "Build" ==> "Deploy"
=> change target dependencies to "Clean" ==> "Build" ==> "Test" ==> "Deploy"
separate paket configuration for test (paket.dependencies, paket.lock in test subfolder) which leads to inconsistent versions of referenced dependencies
=> remove paket.dependencies and paket.lock from test
poisonous mix of NUnit versions
=> remove explicit references to NUnit.Framework from paket.dependencies and run paket.exe install
invalid type extension in test project
=> change to type Test() or delete useless file
building creates output of all projects (and not just src/app) in ./build but tests look for DLLs in ./test
=> change test file pattern to buildDir + "**/*.Test.dll"
if you want to use NUnit3
=> open Fake.Testing and use NUnit3 instead of NUnit
finally, you should commit paket.bootstrapper.exe
I recommend you either use a predefined template or start small and make sure you understand each step and check that it is working as expected. Once you've run over the point of a non-working solution it is extremely hard to get back on track.
I am maintaining an autoconf package and wanted to integrate automatic testing. I use the Boost Unit Test Framework for my unit tests and was able to sucessfully integrate it into the package.
That is it can be compiled via make check, but is is not run (although I read that make check both compiles and runs the tests). As result, I have to run it manually after building the tests which is cumbersome.
Makefile.am in the test folder looks like this:
check_PROGRAMS = prog_test
prog_test_SOURCES = test_main.cpp ../src/class1.cpp class1_test.cpp class2.cpp ../src/class2_test.cpp ../src/class3.cpp ../src/class4.cpp
prog_test_LDADD = $(BOOST_FILESYSTEM_LIB) $(BOOST_SYSTEM_LIB) $(BOOST_UNIT_TEST_FRAMEWORK_LIB)
Makefile.am in the root folder:
SUBDIRS = src test
dist_doc_DATA = README
ACLOCAL_AMFLAGS = ${ACLOCAL_FLAGS} -I m4
Running test/prog yields the output:
Running 4 test cases...
*** No errors detected
(I don't think you need the contents of my test cases in order to answer my question, so I omitted them for now)
So how can I make automake run my tests every time I run make check?
At least one way of doing this involves setting TESTS variable. Here's what documentation on automake says about it:
If the special variable TESTS is defined, its value is taken to be a list of programs or scripts to run in order to do the testing.
So adding the line
TESTS = $(check_PROGRAMS)
should instruct it to run the tests on make check.
Is there any way to provide some command-line argument in order to skip all tests but one on some module? So I will not need to change pom.xml every time I will need to run another test?
For example, I want to create build configuration on TeamCity, and provide command-line arguments to run only single test in some module. Next time I will need to change it and run another test, and so on.
Perhaps it is not how CI is intended to be used, but still.
I assume you've read the docs about running a single test under surefire? What they don't tell you is how to do that in a sub-module:
mvn test -Dtest=testname -pl subproject
Where subproject is the project containing that test. From the mvn man page:
-pl,--projects arg Comma-delimited list of specified reactor projects to build instead of all projects. A project can be specified by [groupId]:artifactId or by its relative path.
Other answers I see are not fully complete, for projects that depend on other sub-modules to be built. One option is to run mvn install to have the required jars to be installed into ~/.m2/..., but that option is not very "clean".
Following command will build the sub-modules, and run only the test class that is specified. This is to be run at parent module level. Also, no need to specify sub-module name.
mvn test -DfailIfNoTests=false -Dtest={test_class_name} -am
As an aside, this can also be mvn clean test -Dfa...... I have a habit of always running clean when running tests.
References..
-am will make all the other sub-modules.
-DfailIfNoTests=false does not fail the entire process since we are not intending to run tests in other modules.
-pl option is not needed since -am is already building everything
In case the module to be tested depends on other projects, solution works by changing commands as:
mvn test -DfailIfNoTests=false -Dtest=testname -pl subproject
FWIW, if you have a multi-module project, you can run all tests with this command at parent directory.
mvn test -pl subproject
And the subproject's name can be found by running the following command, usually in the form of group-id:artifact-id.
mvn help:active-profiles
How do I create a simple unit test for my application using SBT's test feature?
I'm hoping the answer is that I can write a single file in src/test/scala for my project that imports some special testing package from SBT which makes writing tests as easy as writing a single method.
The tutorial ExampleSbtTest seems to be doing something more complicated than what I need, and I can't find anything simpler on the SBT GoogleCode page.
Testing with SBT
No matter which version of SBT you want to use, basically you have to do the following steps:
Include your desired testing framework as test-dependency in your project configuration.
Create a dedicated testing folder within your source tree, usually src/test/scala, if it isn't present already.
As always: Write your tests, specs ...
Those basic steps are identical for the sbt 0.7 branch (that's the one from google-code) and the current sbt 0.10 branch (now developed and documented on github). However, there are minor differences how to define the testing dependencies since 0.10 provides a new quick configuration method not present in 0.7.
Defining the dependency for SBT 0.7
Here is how you create a basic test (based on scalacheck) with sbt 0.7. Create a new sbt 0.7 project by calling sbt in an empty folder. Change into the automatically created project folder and create a new build folder
# cd [your-project-root]/project
# mkdir build
change into the newly created build folder and create your first project build file Project.scala with the follwing content:
class Project(info: ProjectInfo) extends DefaultProject(info) {
val scalacheck = "org.scala-tools.testing" %% "scalacheck" % "1.9" % "test"
}
Since for 0.7 the testing folder is created automatically you can then start to write your first test right away. Step to the paragraph "Create a simple test".
Defining the dependency for SBT 0.10
For 0.10 one can use the sbt console to add the dependency. Just start sbt in your project directory and enter the following commands:
set libraryDependencies += "org.scala-tools.testing" %% "scalacheck" % "1.9" % "test"
session save
You can then close the sbt console and have a look at your projects build.sbt file. As you can easily spot the above libraryDependencies line was added to your projects quick configuration.
Since 0.10 doesn't create the source folders automatically. You have to create the testing folder on your own:
# cd [project-root]
# mkdir -p src/test/scala
That's all it takes to get started with 0.10. Moreover, the documentation about testing with 0.10 is far more detailed then the old one. See the testing wiki page for further details.
Create a simple scalacheck test
Create the following test file src/test/scala/StringSpecification.scala (taken from the scalacheck homepage):
import org.scalacheck._
object StringSpecification extends Properties("String") {
property("startsWith") = Prop.forAll((a: String, b: String) => (a+b).startsWith(a))
property("endsWith") = Prop.forAll((a: String, b: String) => (a+b).endsWith(b))
// Is this really always true?
property("concat") = Prop.forAll((a: String, b: String) =>
(a+b).length > a.length && (a+b).length > b.length
)
property("substring") = Prop.forAll((a: String, b: String) =>
(a+b).substring(a.length) == b
)
property("substring") = Prop.forAll((a: String, b: String, c: String) =>
(a+b+c).substring(a.length, a.length+b.length) == b
)
}
As already indicated this basic check will fail for the "concat" specification, but that are the basic testing steps needed to get started with testing and sbt. Just adapt the included dependency if you want to use another testing framework.
Run your tests
To run your test, open the sbt console and type
> test
That will run all tests present in your src/test tree no matter if those are java or scala based tests. So you can easily reuse your existing java unit tests and convert them step by step to scala.
I'm using GNU autotools for the build system on a particular project. I want to start writing automated tests for verifcation. I would like to just type "make check" to have it automatically run these. My project is in C++, although I am still curious about writing automated tests for other languages as well.
Is this compatible with pretty much every unit testing framework out there (I was thinking of using cppunit)? How do I hook these unit testing frameworks into make check? Can I make sure that I don't require the unit test software to be installed to be able to configure and build the rest of the project?
To make test run when you issue make check, you need to add them to the TESTS variable
Assuming you've already built the executable that runs the unit tests, you just add the name of the executable to the TESTS variable like this:
TESTS=my-test-executable
It should then be automatically run when you make check, and if the executable returns a non-zero value, it will report that as a test failure. If you have multiple unit test executables, just list them all in the TESTS variable:
TESTS=my-first-test my-second-test my-third-test
and they will all get run.
I'm using Check 0.9.10
configure.ac
Makefile.am
src/Makefile.am
src/foo.c
tests/check_foo.c
tests/Makefile.am
./configure.ac
PKG_CHECK_MODULES([CHECK], [check >= 0.9.10])
./tests/Makefile.am for test codes
TESTS = check_foo
check_PROGRAMS = check_foo
check_foo_SOURCES = check_foo.c $(top_builddir)/src/foo.h
check_foo_CFLAGS = #CHECK_CFLAGS#
and write test code, ./tests/check_foo.c
START_TEST (test_foo)
{
ck_assert( foo() == 0 );
ck_assert_int_eq( foo(), 0);
}
END_TEST
/// And there are some tcase_xxx codes to run this test
Using check you can use timeout and raise signal. it is very helpful.
You seem to be asking 2 questions in the first paragraph.
The first is about adding tests to the GNU autotools toolchain - but those tests, if I'm understanding you correctly, are for both validating that the environment necessary to build your application exists (dependent libraries and tools) as well as adapt the build to the environment (platform specific differences).
The second is about unit testing your C++ application and where to invoke those tests, you've proposed doing so from the autotools tool chain, presumably from the configure script. Doing that isn't conventional though - putting a 'test' target in your Makefile is a more conventional way of executing your test suite. The typical steps for building and installing an application with autotools (at least from a user's perspective, not from your, the developer, perspective) is to run the configure script, then run make, then optionally run make test and finally make install.
For the second issue, not wanting cppunit to be a dependency, why not just distribute it with your c++ application? Can you just put it right in what ever archive format you're using (be it tar.gz, tar.bz2 or .zip) along with your source code. I've used cppunit in the past and was happy with it, having used JUnit and other xUnit style frameworks.
Here is a method without dependencies:
#src/Makefile.am
check_PROGRAMS = test1 test2
test1_SOURCES = test/test1.c code_needed_to_test1.h code_needed_to_test1.c
test2_SOURCES = test/test2.c code_needed_to_test2.h code_needed_to_test2.c
TESTS = $(check_PROGRAMS)
The make check will naturally work and show formatted and summarized output:
$ make check
...
PASS: test1
PASS: test2
============================================================================
Testsuite summary for foo 1.0
============================================================================
# TOTAL: 2
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
When you do a make dist nothing from src/test/* will be
in the tarball. Test code is not in the dist, only source will be.
When you do a make distcheck it will run make check and run your tests.
You can use Automake's TESTS to run programs generated with check_PROGRAMS but this will assume that you are using a log driver and a compiler for the output. It is probably easier to still use check_PROGRAMS but to invoke the test suite using a local rule in the Makefile:
check_PROGRAMS=testsuite
testsuite_SOURCES=...
testsuite_CFLAGS=...
testsuite_LDADD=...
check-local:
./testsuite