Google Tests ValuesIn with a global vector - c++

Question

I'm trying to run a bunch of google tests based off configuration files in a directory. This way I can just add a new file and run the tests without having to add it to my parametrized test and recompile. Here is my code:
typedef std::pair<QString, int> TestParam;
QString generatedLogic;
std::vector<TestParam> badExpressionTests;
class LogicBuilderTest : public ::testing::TestWithParam<TestParam> {};
GTEST_API_ int main(int argc, char **argv) {
testing::InitGoogleTest(&argc, argv);
::generatedLogic = "/home/mitydsp/trunk/System/logicExecutionEngine/engine/include/GeneratedLogic.h";
QString dir = "/home/mitydsp/trunk/System/logicExecutionEngine/unittest/expressions/bad/";
QDirIterator it(dir, QStringList() << "*.txt", QDir::Files, QDirIterator::Subdirectories);
while (it.hasNext()) {
QString path = it.next();
QStringList nameParts = it.fileName().split("_");
int exitCode = nameParts[0].toInt();
::badExpressionTests.push_back(TestParam(path, exitCode));
}
std::cout << "Size of vector: " << badExpressionTests.size() << std::endl;
return RUN_ALL_TESTS();
}
/**
* Run parameterized test
*/
TEST_P(LogicBuilderTest, TestExitWithCode)
{
::testing::FLAGS_gtest_death_test_style = "threadsafe";
// Simulate fake main()
char arg0[] = "logicTest";
char* argv[] = { &arg0[0], NULL };
int argc = (int)(sizeof(argv) / sizeof(argv[0])) - 1;
EXPECT_EXIT({
// Need to run Qt Application because logic builder uses async QTimer::singleShot()
QCoreApplication app(argc, argv);
// Create a logic builder instance
LogicBuilder logicBuilder(
(int) ParameterStoreInterface::DEFAULT_PARAM_STORE_VALUES_KEY,
(int) ParameterStoreInterface::DEFAULT_PARAM_STORE_NAMES_KEY,
(int) ParameterStoreInterface::DEFAULT_PARAM_STORE_LOCK_KEY,
QString(GetParam().first),
QStringList(""),
QStringList(generatedLogic),
QString(LogicBuilder::DEFAULT_INTERMEDIATE_DIR),
QString(LogicBuilder::DEFAULT_OUTPUT_SRC),
QString(LogicBuilder::DEFAULT_OUTPUT_LIB),
true );
app.exec();
}, ::testing::ExitedWithCode(GetParam().second), "");
}
INSTANTIATE_TEST_CASE_P(TestBadExpressions, LogicBuilderTest,
::testing::ValuesIn(::badExpressionTests));
When I run this, it shows that 0 tests are being ran even though the vector says its size is two. How come these parametrized tests are not being ran?
Size of vector: 2
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[ PASSED ] 0 tests.
Originally I was running the tests by manually defining the configuration files but I don't like this:
INSTANTIATE_TEST_CASE_P(TestBadExpressions, LogicBuilderTest,
::testing::Values(
TestParam("unittest/expressions/bad/17_no_lhs.txt", LogicBuilder::LB_VARIABLE_NO_LHS),
TestParam("unittest/expressions/bad/25_incomplete_rhs.txt", LogicBuilder::LB_PARSE_ERROR)
));

Answer 1

I've spent the last hour trying to figure this out and then as soon as I post I come up with the solution. I'll leave this here because it may help someone in the future:
Instead of trying to create the list of files in main, pass the ValuesIn a function which loads the files and returns a vector.
std::vector<TestParam> GetFilesInDir()
{
std::vector<TestParam> values;
QString dir = "/home/mitydsp/trunk/System/logicExecutionEngine/unittest/expressions/bad/";
QDirIterator it(dir, QStringList() << "*.txt", QDir::Files, QDirIterator::Subdirectories);
while (it.hasNext()) {
QString path = it.next();
QStringList nameParts = it.fileName().split("_");
int exitCode = nameParts[0].toInt();
values.push_back(TestParam(path, exitCode));
}
return values;
}
INSTANTIATE_TEST_CASE_P(TestBadExpressions, LogicBuilderTest,
::testing::ValuesIn(GetFilesInDir()));
Thanks to Maksim Solovjov for reminding me that INSTANTIATE_TEST_CASE_P macro was being executed before main.

Answer 2

we spent some hours on this issue and thought this snippet will be helpful.
#include <iostream>
#include <string>
#include <vector>
using namespace std;
#include "gtest/gtest.h"
GTEST_API_ main( int argc, char* argv[] ) {
testing::InitGoogleTest( &argc, argv );
return RUN_ALL_TESTS();
}
struct test_parms_s {
string name; // Simple name of this test
string description; // Description of this test
void* function_parms; // Pointer to test function specific data
};
std::vector<test_parms_s*> BuildTestData() {
test_parms_s* pTestDataA = new test_parms_s();
test_parms_s* pTestDataB = new test_parms_s();
test_parms_s* pTestDataC = new test_parms_s();
pTestDataA->name.assign("testA");
pTestDataA->description.assign("testA_desc");
pTestDataA->function_parms = NULL;
pTestDataB->name.assign("testB");
pTestDataB->description.assign("testB_desc");
pTestDataB->function_parms = NULL;
pTestDataC->name.assign("testC");
pTestDataC->description.assign("testC_desc");
pTestDataC->function_parms = NULL;
std::vector<test_parms_s*> values;
values.push_back(pTestDataA);
values.push_back(pTestDataB);
values.push_back(pTestDataC);
cout << "BuildTestData" << endl;
return values;
}
//------------------------------------------------------------------------
class Testy {
private:
string testname_;
public:
Testy( string testname );
~Testy();
string GetTestName();
void SetTestName( string testname );
};
Testy::Testy( string testname ) {
testname_.assign(testname);
}
Testy::~Testy() {}
string Testy::GetTestName() { return testname_; }
void Testy::SetTestName( string testname ) { testname_ = testname; }
//------------------------------------------------
class TestFixture : public ::testing::TestWithParam<test_parms_s*> {
protected:
Testy* testy;
virtual void SetUp() {
testy = new Testy( "Test");
}
virtual void TearDown() {
delete( testy );
}
};
//------------------------------------------------
TEST_P( TestFixture, ParamTest ) {
test_parms_s* pTestParms = GetParam();
cout << pTestParms->name << endl;
}
//------------------------------------------------
INSTANTIATE_TEST_CASE_P( TestFixture_instance, TestFixture,
::testing::ValuesIn(BuildTestData()));
//========================================================================
the output will be
BuildTestData
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from TestFixture_instance/TestFixture
[ RUN ] TestFixture_instance/TestFixture.ParamTest/0
testA
[ OK ] TestFixture_instance/TestFixture.ParamTest/0 (0 ms)
[ RUN ] TestFixture_instance/TestFixture.ParamTest/1
testB
[ OK ] TestFixture_instance/TestFixture.ParamTest/1 (0 ms)
[ RUN ] TestFixture_instance/TestFixture.ParamTest/2
testC
[ OK ] TestFixture_instance/TestFixture.ParamTest/2 (0 ms)
[----------] 3 tests from TestFixture_instance/TestFixture (0 ms total)
[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (0 ms total)
[ PASSED ] 3 tests.

Answer 3

I'm not an expert in manually instantiating Google Tests, but the way you did it can't possibly work: you call INSTANTIATE_TEST_CASE_P in the static context, so it is being evaluated before main is ever called.
You could try moving instantiation just before RUN_ALL_TESTS; however, I don't know what that macro does and this might be illegal. In that case, I think that you can't create your tests with INSTANTIATE_TEST_CASE_P in the dynamic way that you want.