I run into a rather strange situation when using std::future and ThreadPool, though I do not think it's ThreadPool (I'm using https://github.com/bandi13/ThreadPool/blob/master/example.cpp) since (I've tried multiple forks of it and after some debugging I do not see how it would be related to the issue).
The issue is that under certain situation my doProcess method just goes nirvana - it does not return. It just disappears midst of a long running loop.
Therefore I think I must be doing something wrong, but can't figure out what.
Here's the code:
ThreadPool pool(numThreads);
std::vector< std::future<bool> > futures;
int count = 0;
string orgOut = outFile;
for (auto fileToProcess : filesToProcess) {
count++;
outFile = orgOut + std::to_string(count);
// enque processing in the thread pool
futures.emplace_back(
pool.enqueue([count, fileToProcess, outFile, filteredKeys, sql] {
return doProcess(fileToProcess, outFile, filteredKeys, sql);
})
);
}
Then I wait for all processings to be done (I think this could be done in a more elegant way also):
bool done = false;
while (!done) {
done = true;
for (auto && futr : futures) {
auto status = futr.wait_for(std::chrono::milliseconds(1));
if (status != std::future_status::ready) {
done = false;
break;
}
}
}
Edit: At first I also tried the obvius wait(), with the same result however:
bool done = false;
while (!done) {
done = true;
for (auto && futr : futures) {
futr.wait();
}
}
Edit: The doProcess() method. The behavior is this: The loopcnt variable is just a counter to debug how often the method was entered and the loop started. As you can see, there is no return from this loop, but the thread just vanishes when inside this loop with no error whatsoever and wasHereCnt is reached only occasionally (like 1 of 100 times the method is run). I'm really puzzled.
bool doProcess([...]) {
// ....
vector<vector<KVO*>*>& features = filter.result();
vector<vector<KVO*>*> filteredFeatures;
static int loopcnt = 0;
std::cout << "loops " << loopcnt << endl;
loopcnt++;
for (vector<KVO*>* feature : features) {
for (KVO *kv : *feature) {
switch (kv->value.type()) {
case Variant::JNULL:
sqlFilter.setNullValue(kv->key);
break;
case Variant::INT:
sqlFilter.setValue(static_cast<int64_t>(kv->value), kv->key);
break;
case Variant::UINT:
sqlFilter.setValue(static_cast<int64_t>(kv->value), kv->key);
break;
case Variant::DOUBLE:
sqlFilter.setValue(static_cast<double>(kv->value), kv->key);
break;
case Variant::STRING:
sqlFilter.setValue(static_cast<string>(kv->value), kv->key);
break;
default:
assert(false);
break;
}
}
int filterResult = sqlFilter.exec();
if (filterResult > 0) {
filteredFeatures.push_back(feature);
}
sqlFilter.reset();
}
static int wasHereCnt = 0;
std::cout << "was here: " << wasHereCnt << endl;
wasHereCnt++;
JsonWriter<Writer<FileWriteStream>> geojsonWriter(writer, filteredFeatures);
bool res = geojsonWriter.write();
os.Flush();
fclose(fp);
return res;
}
The doProcess method does work when it's taking less time. It breaks and disappears when it takes somewhat more time. The difference being just the complexity of an SQL query I run in the method. So I don't post the code for doProcess().
What causes the thread of the thread pool to be interrupted, and how to fix it?
UPDATE
Well, I found it out. After several hours I decided to remove the future tasks and just ran the task on the main thread. The issue was that an exception was thrown via:
throw std::runtime_error("bad cast");
... some time down the code flow after this:
case Variant::UINT:
sqlFilter.setValue(static_cast<int64_t>(kv->value), kv->key);
break;
This error was thrown as expected when running on the main thread. But it's never raised when run as future task. This is really odd and seems like a compiler or debugger issue.
Related
I am writing unit tests for one of the classes I want to test. Function infFooLoop() what I want to test is executed in endless loop (request to stop it comes externally). My problem is I want to change the private variable state_ via setter in setState when function executes in separate thread asynchronously. Some simplified code example is here:
enum class State : int
{
Success = 1,
Failure = 2
};
class Foo
{
private:
State state_{State::Success};
bool stop_flag_{false};
public:
void setState(State state) { state_ = state; }
void infFooLoop(bool &start)
{
while (start)
{
std::cout << "While loop executes \n";
if (state_ == State::Failure)
{
stop_flag_ = true;
}
else if (stop_flag_)
{
std::cout << "Program stopped\n";
// Report error but for this example break
break;
}
std::this_thread::sleep_for(std::chrono::milliseconds(200));
}
}
};
int main()
{
Foo obj;
bool flag{true};
std::future<void> future = std::async(std::launch::async, [&](){ obj.infFooLoop(flag);});
// Try to change the data:
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
// I want to change data for `Failure` so stop_flag condition will be used
obj.setState(State::Failure);
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
// terminate loop
flag = false;
return 0;
}
state_ will be taken from external component and stop_flag_ is used to handle some corner-case here (in real it wont break the loop, just report error to different component).
I assume infFooLoop executes on a separate thread so I can't just call another function in a different thread to change this data. stop_flag_ is some internal variable used only in this one function so I want to leave it as simple as possible (do not introduce mutexes/atomic in Foo class only for tests). Can you give me some suggestions on what should I do here?
I try to handle a deadlock in my code but I can't fugure out how to prevent it. I have a thread which accesses data and an update method which update the data. The code looks like this:
thread {
forever {
if (Running) {
LOCK
access data
UNLOCK
}
Running = false;
}
}
update {
Running = false;
LOCK
access data
UNLOCK
Running = true;
}
I tried to fix it with a second access variable but it doesn't change anything.
thread {
forever {
if (!Updating) {
if (Running) {
LOCK
access data
UNLOCK
}
}
Running = false;
}
}
update {
Updating = true;
Running = false;
LOCK
access data
UNLOCK
Updating = false;
Running = true;
}
Thanks for your help.
UPDATE
This is a better description of the problem:
thread {
forever {
if (Running) {
LOCK
if (!Running) leave
access data
UNLOCK
}
Running = false;
}
}
update {
Running = false;
LOCK
access data
UNLOCK
Running = true;
}
My update function is a bit more complex, so that I can't see a way to use one of the standard algorithm for this.
UPDATE 2
Here is the simplified c++ source code. maybe it's better to read as the pseudocode:
void run() {
forever {
if (mRunning) {
QMutexLocker locker(&mMutex);
for (int i; i < 10; i++) {
qDebug("run %d", i);
sleep(1);
if (!mRunning) break;
}
mRunning = false;
}
}
}
void update() {
mRunning = false;
QMutexLocker locker(&mMutex);
qDebug("update");
mRunning = true;
}
UPDATE 3
Ok. The problem is a bit more complex. I forgot that my accesss data part in the thread starts also some child threads to fill the data structure
datathread {
access data
}
thread {
forever {
if (Running) {
LOCK
if (!Running) leave
forloop
start datathread to fill data to accessdata list
UNLOCK
}
Running = false;
}
}
update {
Running = false;
LOCK
access data
UNLOCK
Running = true;
}
Standard way for read method being restarted when during write is to use seqlock. With single writer and reader seqlock is just atomic integer variable, which is incremented every time when writer is started and when it is ended. Such a way reader method can periodically check whether variable is unchanged since read is started:
atomic<int> seq = 0;
updater() // *writer*
{
seq = seq + 1;
<update data>
seq = seq + 1;
}
thread() // *reader*
{
retry: // Start point of unmodified data processing.
{
int seq_old = seq;
if(seq_old & 1)
{
// odd value of the counter means that updater is in progress
goto retry;
}
for(int i = 0; i < 10; i++)
{
<process data[i]>
if(seq_old != seq)
{
// updater has been started. Restart processing.
goto retry;
}
}
// Data processing is done.
}
}
If several updater() can be executed concurrently, whole update code should be executed with mutex taken:
updater() // *writer*
{
QMutexLocker locker(&updater_Mutex);
seq = seq + 1;
<update data>
seq = seq + 1;
}
If even single element of data cannot be accessed concurrently with updating, both <update data> and <process data[i]> should be executed with mutex taken.
Hy all :)
I am using 1.5.4-all (2014-10-22) in my VC++ project (Microsoft Visual C++ Compiler 18.00.21005.1 for x86 platform).
My problem is that I get the following error message after some time. The time, after which the error occurs differ quiet a lot - sometimes it happens after 30 secs and sometimes after 5 minutes.
I could locate the source for the error in the LinearHashTable.h file at line 214:
I have the following method where a Shot (struct) is added to the table:
void ShotSimulationService::SimulateShot(Shot shot) {
MutexThreadLock.lock();
shots.insert(ShotsSetType::ValueType(SimulationShot(shot)));
errorCount = 0;
MutexThreadLock.unlock();
}
The call of SimulateShot is from another thread than the handling of the following code:
void ShotSimulationService::Update(WebcamService* observable) {
if (shots.empty()) {
return;
}
try {
Mat frame = observable->GetLastImage().clone();
ShotsSetType::Iterator iter = shots.begin();
vector<Shot> deleteShots;
errorCount++;
while (iter != shots.end()){
if (iter->SimulateStartExplosion()) {
//simulate gun explosion
OverlayImage(frame, gunShotImg, iter->startPoint);
}
//simulate explosion
SimulationShot::SimulationHitStatus status = iter->status;
if (status == SimulationShot::SimulationHitStatus::UNKNOWN) {
if (detectionService.HasShotHitPlayer(frame, *iter)) {
iter->status = SimulationShot::HIT_PLAYER;
iter->SetCurrentPointAsEndoint();
//Notify that player was hit
playerHitQueue.enqueueNotification(new PlayerHitNotification(iter->hitPlayer));
}
}
if (iter->SimulateEndExplosion()) {
if (status == SimulationShot::HIT_PLAYER) {
int explosionx = iter->endPoint.x - robotExplosionHalfXSize > 0 ? iter->endPoint.x - robotExplosionHalfXSize : 0;
int explosionY = iter->endPoint.y - robotExplosionHalfYSize > 0 ? iter->endPoint.y - robotExplosionHalfYSize : 0;
OverlayImage(frame, robotExplosionImg, Point2i(explosionx, explosionY));
}
else {
// status == SimulationShot::HIT_WALL or UNKNOWN
int explosionx = iter->endPoint.x - wallExplosionHalfXSize > 0 ? iter->endPoint.x - wallExplosionHalfXSize : 0;
int explosionY = iter->endPoint.y - wallExplosionHalfYSize > 0 ? iter->endPoint.y - wallExplosionHalfYSize : 0;
OverlayImage(frame, robotExplosionImg, Point2i(explosionx, explosionY));
if (status != SimulationShot::HIT_WALL) {
iter->status = SimulationShot::HIT_WALL;
}
}
if (iter->IsSimulationFinished()) {
deleteShots.push_back(*iter);
}
}
else {
//simulate bullet
OverlayImage(frame, cheeseImg, iter->GetNextShotPoint());
}
++iter;
}
//delete finished simulations
MutexThreadLock.lock();
for each (Shot shot in deleteShots)
{
shots.erase(shot);
}
MutexThreadLock.unlock();
}
catch (cv::Exception& e) {
Logger& logger = Logger::get("Test");
logger.error(e.what());
}
}
The Update method is called quiet often - always when a new webcam frame is available.
The callstack of the error starts in the following line:
if (iter->SimulateEndExplosion()) {
In the method SimulateEndExplosion only members of the struct were used:
bool SimulateEndExplosion() {
if (status == HIT_PLAYER) {
currPercentage = 1.0;
return true;
}
if (currPercentage < 1.0) {
return false;
}
++endExplosionCtr;
return endExplosionCtr <= maxEndExplosions;
}
Does anybody have an idea why this problem occurs?
Any help and any feedback is welcome!! I have absolutly no idea what is going wrong here :(
Thanks!
Iterating in one thread and inserting in another without protecting the operations with mutex in both threads will cause this problem; when you insert, iterator will be invalidated and you will get the assertion failure. You should protect both insertion and iteration with mutex.
Also, the way you are using mutex is not safe because mutex will not be unlocked if an exception is thrown between lock() and unlock(). Use ScopedLock instead and RAII will do the job automatically and safely in all cases:
void ShotSimulationService::SimulateShot(Shot shot) {
Mutex::ScopedLock lock(MutexThreadLock);
shots.insert(ShotsSetType::ValueType(SimulationShot(shot)));
errorCount = 0;
// unlock will be called by ScopedLock destructor
}
I've come across a situation where I have a bunch of "systems" that need to be initialized in sequence, with the next system only being initialized if all of the proceeding systems initialized successfully.
This has led me to a whole slew of nested if - else statements. Here's some pseudo-code for visualization.
bool mainInit () {
if (!system1Init ()) {
reportError (); // some error reporting function
}
else {
if (!system2Init ()) {
reportError ();
}
else {
if (!system3Init ()) {
// ... and so on
I find that this starts to look like a mess when you get even a handful of levels to it.
Now I thought of using a switch statement instead, starting at the first case and falling through to the other cases on success, only breaking if there's an error.
bool mainInit () {
switch (1) {
case 1:
if (!system1Init ()) {
reportError ();
break;
}
case 2:
if (!system2Init ())
reportError ();
break;
}
// ....
}
Now, I like this a lot better. I find it much easier to read, especially with some decent comments, but I'm fairly new to programming.
So, my question is: Seeing how this is not how switch statements are traditionally used(at least from what I've seen), is something like this acceptable, or would this be considered bad form?
Being new to programming, I'm trying not to develop too many bad habits that might frustrate and make things more difficult for other programmers down the road.
I did a search, but most of what I found had to do with replacing chains of if - else if statements, not replacing nested ones.
Reference all of the systems in an array, for example an std::vector<mySystem*>, and loop over them sequentially, breaking off on the first fail. This way your entire code is reduced to less than 5 lines of code, even for 500+ systems.
The suggested switch hack is an evil example of XY problem solving: your real problem is that you don't have the array of systems, and are using named variables, thus eliminating all options to more flexibly use all systems, like in a loop.
Assuming that all your system#Init() calls are known at compile time, you can very easily put them in a table and then iterate over that table.
typedef (*system_init)(void);
system_init initialization_functions[] =
{
system1Init,
system2Init,
system3Init,
...
systemNInit
};
bool mainInit()
{
for(size_t idx(0); idx < sizeof(initialization_functions) / sizeof(initialization_functions[0]); ++idx)
{
if(!initialization_functions[idx]())
{
ReportError();
return false;
}
}
return true;
}
However, your existing code looks incorrect since the first mainInit() only calls system1Init() and then exits. Probably not what you wanted in the first place.
if(!system1Init())
{
ReportError();
return false;
}
// if you add an else, the system2Init() does not get called
// even if system1Init() succeeds
if(!system2Init())
{
ReportError();
return false;
}
[...]
return true;
Would the switch answer your problem? Not as it was written. That is, if you wanted to call the mainInit() function with a counter, it could be useful. Drupal uses that mechanism:
bool mainInit(int idx)
{
bool r(true);
switch(idx)
{
case 1:
r = system1Init();
break;
case 2:
r = system2Init();
break;
[...]
}
if(!r)
{
ReportError();
}
return r
}
Note that the table mechanism works the same way as the switch. As long as all the code is found in the systemNInit() functions (and it should be), the switch does not add anything, so you could do something like this too:
bool mainInit(int idx)
{
if(idx < 0 || idx >= sizeof(initialization_functions) / sizeof(initialization_functions[0]))
{
throw std::range_error("index out of bounds");
}
if(!initialization_functions[idx]())
{
ReportError();
return false;
}
return true;
}
Calling the mainInit() with an index can be helpful in case you want to "de-initialize" properly:
int main()
{
for(size_t idx(0); idx < ...; ++idx)
{
if(!mainInit(idx))
{
while(idx > 0)
{
--idx;
mainDeinit(idx);
}
exit(1);
}
}
...app do something here...
}
Use custom exceptions with clear error messages and add a try-catch-report-die around the code in main(). Exceptions are there to specifically make your case look good by making "bad path" implicit.
void initX() { ...; throw std::invalid_argument_exception("..."); }
int main() {
try {
init1(); init2(); ... run();
return 0;
} catch (std::exception const& e) {
log(e.what()); exit 42;
}
}
I'd do it this way:
bool mainInit () {
if (!system1Init ()) {
return(false);
}
if (!system2Init ()) {
return(false);
}
if (!system3Init ()) {
return(false);
}
//...
return(true);
}
//...
if(!mainInit()) {
reportError();
}
If I desire to run a piece of code in a function, only from the second invocation of the function onwards,
Questions:
Is there something wrong to do that?
How can I possibly achieve this ? Is using a static variable to do this a good idea ?
There's two answers to this question, depending on whether you have to deal with multi-threaded serialization or not.
No threading:
void doSomething() {
static bool firstTime = true;
if (firstTime) {
// do code specific to first pass
firstTime = false;
} else {
// do code specific to 2nd+ pass
}
// do any code that is common
}
With threading:
I'll write the generic boilerplate, but this code is system specific (requiring some variant of an atomic compareAndSet).
void doSomethingThreadSafe() {
static volatile atomic<int> passState = 0;
do {
if ( passState == 2 ) {
//perform pass 2+ code
break;
} else
if ( passState.compareAndSet(0,1) ) { // if passState==0 set passState=1 return true else return false
//perform pass 1 initialization code
passState = 2;
break;
} else {
//loser in setup collision, delay (wait for init code to finish) then retry
sleep(1);
}
} while(1);
//perform code common to all passes
}
Multi-threading will be a problem. To prevent this, if required, you'll probably need something like a mutex.
Like this:
void someFunction()
{
static bool firstRun = true;
if (!firstRun)
{
// code to execute from the second time onwards
}
else
{
firstRun = false;
}
// other code
}
Add a global counter.
eg:-
static int counter = 0;
public void testFunc(){
if(counter==1){
........
<Execute the functionality>
........
}
counter++;
}