Running Some Tests Sequentially While Others in Parallel [duplicate] - unit-testing

Question

I have a collection of tests. There are a few tests that need to access a shared resource (external library/API/hardware device). If any of these tests run in parallel, they fail.
I know I could run everything using --test-threads=1 but I find that inconvenient just for a couple of special tests.
Is there any way to keep running all tests in parallel and have an exception for a few? Ideally, I would like to say do not run X, Y, Z at the same time.

Answer 1

Use the serial_test crate. With this crate added, you put in your code:
#[serial]
in front of any test you want run in sequentially.

Answer 2

As mcarton mentions in the comments, you can use a Mutex to prevent multiple pieces of code from running at the same time:
use once_cell::sync::Lazy; // 1.4.0
use std::{sync::Mutex, thread::sleep, time::Duration};
static THE_RESOURCE: Lazy<Mutex<()>> = Lazy::new(Mutex::default);
type TestResult<T = (), E = Box<dyn std::error::Error>> = std::result::Result<T, E>;
#[test]
fn one() -> TestResult {
let _shared = THE_RESOURCE.lock()?;
eprintln!("Starting test one");
sleep(Duration::from_secs(1));
eprintln!("Finishing test one");
Ok(())
}
#[test]
fn two() -> TestResult {
let _shared = THE_RESOURCE.lock()?;
eprintln!("Starting test two");
sleep(Duration::from_secs(1));
eprintln!("Finishing test two");
Ok(())
}
If you run with cargo test -- --nocapture, you can see the difference in behavior:
No lock
running 2 tests
Starting test one
Starting test two
Finishing test two
Finishing test one
test one ... ok
test two ... ok
With lock
running 2 tests
Starting test one
Finishing test one
Starting test two
test one ... ok
Finishing test two
test two ... ok
Ideally, you'd put the external resource itself in the Mutex to make the code represent the fact that it's a singleton and remove the need to remember to lock the otherwise-unused Mutex.
This does have the massive downside that a panic in a test (a.k.a an assert! failure) will cause the Mutex to become poisoned. This will then cause subsequent tests to fail to acquire the lock. If you need to avoid that and you know the locked resource is in a good state (and () should be fine...) you can handle the poisoning:
let _shared = THE_RESOURCE.lock().unwrap_or_else(|e| e.into_inner());
If you need the ability to run a limited set of threads in parallel, you can use a semaphore. Here, I've built a poor one using Condvar with a Mutex:
use std::{
sync::{Condvar, Mutex},
thread::sleep,
time::Duration,
};
#[derive(Debug)]
struct Semaphore {
mutex: Mutex<usize>,
condvar: Condvar,
}
impl Semaphore {
fn new(count: usize) -> Self {
Semaphore {
mutex: Mutex::new(count),
condvar: Condvar::new(),
}
}
fn wait(&self) -> TestResult {
let mut count = self.mutex.lock().map_err(|_| "unable to lock")?;
while *count == 0 {
count = self.condvar.wait(count).map_err(|_| "unable to lock")?;
}
*count -= 1;
Ok(())
}
fn signal(&self) -> TestResult {
let mut count = self.mutex.lock().map_err(|_| "unable to lock")?;
*count += 1;
self.condvar.notify_one();
Ok(())
}
fn guarded(&self, f: impl FnOnce() -> TestResult) -> TestResult {
// Not panic-safe!
self.wait()?;
let x = f();
self.signal()?;
x
}
}
lazy_static! {
static ref THE_COUNT: Semaphore = Semaphore::new(4);
}
THE_COUNT.guarded(|| {
eprintln!("Starting test {}", id);
sleep(Duration::from_secs(1));
eprintln!("Finishing test {}", id);
Ok(())
})
See also:
How to limit the number of test threads in Cargo.toml?

Answer 3

You can always provide your own test harness. You can do that by adding a [[test]] entry to Cargo.toml:
[[test]]
name = "my_test"
# If your test file is not `tests/my_test.rs`, add this key:
#path = "path/to/my_test.rs"
harness = false
In that case, cargo test will compile my_test.rs as a normal executable file. That means you have to provide a main function and add all the "run tests" logic yourself. Yes, this is some work, but at least you can decide everything about running tests yourself.
You can also create two test files:
tests/
- sequential.rs
- parallel.rs
You then would need to run cargo test --test sequential -- --test-threads=1 and cargo test --test parallel. So it doesn't work with a single cargo test, but you don't need to write your own test harness logic.