My scenario is as follows:
I have 5 linux machines and I have 10(could be more than this) files in HDFS. My requirement is that a single machine should get lock on one of the file and process it while other machine should not process this file but get lock on another file and process those.
For ex: machine1 - gets lock on file2 and process it
machine2 - gets lock on file3 and process it
machine3 - gets lock on file1 and process it
I have written a dummy multithreaded java program to simulate it .But it is not working:
public class DistributedLock {
private final ZooKeeper zk;
private final String lockBasePath;
private String lockPath;
public DistributedLock(ZooKeeper zk, String lockBasePath) {
this.zk = zk;
this.lockBasePath = lockBasePath;
}
public boolean lock(String lockName) throws IOException {
try {
boolean locked = false;
if(zk.exists(lockBasePath + "/" + lockName, false) == null){
lockPath = zk.create(lockBasePath + "/" + lockName, null,
ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL_SEQUENTIAL);
if(lockPath != null ){
locked =true;
}
}
final Object lock = new Object();
return locked;
} catch (KeeperException e) {
throw new IOException(e);
} catch (InterruptedException e) {
throw new IOException(e);
}
}
public void unlock() throws IOException {
try {
zk.delete(lockPath, -1);
lockPath = null;
} catch (KeeperException e) {
throw new IOException(e);
} catch (InterruptedException e) {
throw new IOException(e);
}
}
}
public class DistributedLockTest {
public static void main(String[] args) throws Exception {
new DistributedLockTest().run();
}
public void run() throws Exception {
Thread t1 = new Thread(new Process(1));
Thread t2 = new Thread(new Process(2));
Thread t3 = new Thread(new Process(3));
Thread t4 = new Thread(new Process(4));
t1.start();
t2.start();
t3.start();
t4.start();
}
class Process implements Runnable {
int id;
List<String> fileNames = new ArrayList<String>();
public Process(int id) {
this.id = id;
for (int i = 1; i < 11; i++) {
fileNames.add("file" + i);
}
}
// #Override
public void run() {
try {
System.out.println("machine " + id + " started");
String resource = "resource";
String path = "/LockDir";
ZooKeeper zooKeeper = new ZooKeeper("127.0.0.1", 2181, null);
if (zooKeeper.exists(path, false) == null) {
zooKeeper.create(path, null, ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
}
DistributedLock lock = new DistributedLock(zooKeeper, path);
String lockedFile;
for (String fileName : fileNames) {
System.out.println("machine " + id + " Acquiring Lock on "+ fileName);
boolean locked = lock.lock(fileName);
if(locked){
System.out.println("machine " + id + "got Lock on "+ fileName);
lockedFile = fileName;
}
else continue;
Thread.sleep(500);
}
System.out.println("machine " + id + " Releasing Lock");
lock.unlock();
System.out.println("machine " + id + " Released Lock");
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
The output I am getting for this is :
machine 1 started
machine 2 started
machine 3 started
machine 4 started
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
machine 2 Acquiring Lock on file1
machine 1 Acquiring Lock on file1
machine 4 Acquiring Lock on file1
machine 3 Acquiring Lock on file1
machine 1got Lock on file1
machine 3got Lock on file1
machine 2got Lock on file1
machine 4got Lock on file1
machine 1 Acquiring Lock on file2
machine 3 Acquiring Lock on file2
machine 4 Acquiring Lock on file2
machine 2 Acquiring Lock on file2
machine 1got Lock on file2
machine 3got Lock on file2
machine 2got Lock on file2
machine 4got Lock on file2
machine 3 Acquiring Lock on file3
machine 1 Acquiring Lock on file3
machine 2 Acquiring Lock on file3
machine 4 Acquiring Lock on file3
machine 1got Lock on file3
machine 4got Lock on file3
machine 3got Lock on file3
machine 2got Lock on file3
machine 2 Acquiring Lock on file4
machine 4 Acquiring Lock on file4
machine 3 Acquiring Lock on file4
machine 1 Acquiring Lock on file4
machine 4got Lock on file4
machine 2got Lock on file4
machine 3got Lock on file4
machine 1got Lock on file4
machine 4 Acquiring Lock on file5
machine 3 Acquiring Lock on file5
machine 2 Acquiring Lock on file5
machine 1 Acquiring Lock on file5
machine 3got Lock on file5
machine 2got Lock on file5
machine 4got Lock on file5
machine 1got Lock on file5
machine 2 Acquiring Lock on file6
machine 4 Acquiring Lock on file6
machine 3 Acquiring Lock on file6
machine 1 Acquiring Lock on file6
machine 2got Lock on file6
machine 1got Lock on file6
machine 4got Lock on file6
machine 3got Lock on file6
machine 2 Acquiring Lock on file7
machine 4 Acquiring Lock on file7
machine 1 Acquiring Lock on file7
machine 3 Acquiring Lock on file7
machine 4got Lock on file7
machine 2got Lock on file7
machine 1got Lock on file7
machine 3got Lock on file7
machine 4 Acquiring Lock on file8
machine 3 Acquiring Lock on file8
machine 1 Acquiring Lock on file8
machine 2 Acquiring Lock on file8
machine 1got Lock on file8
machine 4got Lock on file8
machine 3got Lock on file8
machine 2got Lock on file8
machine 2 Acquiring Lock on file9
machine 4 Acquiring Lock on file9
machine 3 Acquiring Lock on file9
machine 1 Acquiring Lock on file9
machine 4got Lock on file9
machine 3got Lock on file9
machine 1got Lock on file9
machine 2got Lock on file9
machine 4 Acquiring Lock on file10
machine 3 Acquiring Lock on file10
machine 1 Acquiring Lock on file10
machine 2 Acquiring Lock on file10
machine 2got Lock on file10
machine 4got Lock on file10
machine 1got Lock on file10
machine 3got Lock on file10
machine 4 Releasing Lock
machine 1 Releasing Lock
machine 2 Releasing Lock
machine 3 Releasing Lock
machine 2 Released Lock
machine 1 Released Lock
machine 4 Released Lock
machine 3 Released Lock
This shows each thread/machine is trying for lock on each file and gets it. But what i wanted was if a machine doesnt get lock on particular machine it should try for lock on another file and process that.
Any suggestion on this?
I found two mistakes in your code, the first one is that you use CreateMode.EPHEMERAL_SEQUENTIAL for your lock-nodes. When you probebly want to use CreateMode.EPHEMERAL. Sequential is mainly used for queues not for locks, and it will create nodes with names that look something like: file10000000000123 file10000000000124 etc etc. Therfore you will never create the node you use to check if the lock is taken.
If you fix that issue you will most likely get a race condition between the threads since they first check if the node exists and then creates it. Making it possible for multiple threads to try and create the same node, because of this my solution looks like this:
public class DistributedLock {
public static final String _LOCK = "lock";
...
public boolean lock(String lockName) throws IOException {
try {
boolean locked = false;
synchronized(_LOCK){
if(zk.exists(lockBasePath + "/" + lockName, false) == null){
lockPath = zk.create(lockBasePath + "/" + lockName, null,
ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL);
if(lockPath != null ){
locked =true;
}
}
}
final Object lock = new Object();
return locked;
...
}
...
and the output looks like this, which is what I assume you want:
machine 1 Acquiring Lock on file1
machine 4 Acquiring Lock on file1
machine 2 Acquiring Lock on file1
machine 3 Acquiring Lock on file1
machine 1 got Lock on file1
machine 3 Acquiring Lock on file2
machine 3 got Lock on file2
machine 2 Acquiring Lock on file2
machine 4 Acquiring Lock on file2
machine 2 Acquiring Lock on file3
machine 2 got Lock on file3
...
PS: As a side note I would suggest that you use Apache Curator instead of writing your own looks for Zookeeper, it's much easier and they've covered most of the edge-cases.
Related
This question already has answers here:
How pthread_mutex_lock is implemented
(3 answers)
Closed 4 years ago.
If Thread1 tried to lock resource locked by Thread2.
Does it go to sleep for finite time ?
Now if the Thread2 unlock the mutex then how would Thread1 will come to know that resource is available ? Is the operating system wakes it up or it checks for resource periodically ?
your second assumption is correct. When a mutex is locked by a thread already, all the remaining threads that are trying to lock it again will be placed on hold and will be in the sleep state. Once the mutex lock is unlocked the O/S wakes them all up and who can unlock first can access the lock. This is not in FIFO basis, actually there is no rule which thread should get first preference to lock the mutex once wakes up. You can consider my below example where I have use condition variable to control the threads:-
pthread_cond_t cond1 = PTHREAD_COND_INITIALIZER;
pthread_cond_t cond2 = PTHREAD_COND_INITIALIZER;
pthread_cond_t cond3 = PTHREAD_COND_INITIALIZER;
pthread_mutex_t lock1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t lock2 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t lock3 = PTHREAD_MUTEX_INITIALIZER;
int TRUE = 1;
void print(char *p)
{
printf("%s",p);
}
void * threadMethod1(void *arg)
{
printf("In thread1\n");
do{
pthread_mutex_lock(&lock1);
pthread_cond_wait(&cond1, &lock1);
print("I am thread 1st\n");
pthread_cond_signal(&cond3);/* Now allow 3rd thread to process */
pthread_mutex_unlock(&lock1);
}while(TRUE);
pthread_exit(NULL);
}
void * threadMethod2(void *arg)
{
printf("In thread2\n");
do
{
pthread_mutex_lock(&lock2);
pthread_cond_wait(&cond2, &lock2);
print("I am thread 2nd\n");
pthread_cond_signal(&cond1);
pthread_mutex_unlock(&lock2);
}while(TRUE);
pthread_exit(NULL);
}
void * threadMethod3(void *arg)
{
printf("In thread3\n");
do
{
pthread_mutex_lock(&lock3);
pthread_cond_wait(&cond3, &lock3);
print("I am thread 3rd\n");
pthread_cond_signal(&cond2);
pthread_mutex_unlock(&lock3);
}while(TRUE);
pthread_exit(NULL);
}
int main(void)
{
pthread_t tid1, tid2, tid3;
int i = 0;
printf("Before creating the threads\n");
if( pthread_create(&tid1, NULL, threadMethod1, NULL) != 0 )
printf("Failed to create thread1\n");
if( pthread_create(&tid2, NULL, threadMethod2, NULL) != 0 )
printf("Failed to create thread2\n");
if( pthread_create(&tid3, NULL, threadMethod3, NULL) != 0 )
printf("Failed to create thread3\n");
pthread_cond_signal(&cond1);/* Now allow first thread to process first */
sleep(1);
TRUE = 0;/* Stop all the thread */
sleep(3);
/* this is how we join thread before exit from a system */
/*
pthread_join(tid1,NULL);
pthread_join(tid2,NULL);
pthread_join(tid3,NULL);*/
exit(0);
}
Here I am using 3 mutexs and 3 conditions. With the above example you can schedule/control or prioritize any number of threads in C. If you see the first thread here it locked mutex lock1 and waiting on cond1, likewise second thread locked mutex lock2 and waits on condition cond2 and 3rd thread locked mutex lock3 and waits on condition cond3. This is the current situation of all the threads after they are being created and now all the threads are waiting for a signal to execute further on its condition variable. In the main thread (i.e. main function, every program has one main thread, in C/C++ this main thread created automatically by operating system once control pass to the main method by kernal) we are calling pthread_cond_signal(&cond1); once this system call done thread1 who was waiting on cond1 will be release and it will start executing. Once it finished with its task it will call pthread_cond_signal(&cond3); now thread who was waiting on condition cond3 i.e. thread3 will be release and it will start execute and will call pthread_cond_signal(&cond2); which will release the thread who is waiting on condition cond2 i.e. in this case thread2.
Fundamental information about the mutex (MUtual Exclusion locks)
A mutex is a special lock that only one thread may lock at a time. If a thread locks a mutex and then a second thread also tries to lock the same mutex, the second thread is blocked, or put on hold. Only when the first thread unlocks the mutex is the second thread unblocked—allowed to resume execution.
Linux guarantees that race conditions do not occur among threads attempting to lock a mutex; only one thread will ever get the lock, and all other threads will be blocked.
A thread may attempt to lock a mutex by calling pthread_mutex_lock on it. If the mutex was unlocked, it becomes locked and the function returns immediately.
What happens trying to lock the when its locked by another thread?
If the mutex was locked by another thread, pthread_mutex_lock blocks execution and returns only eventually when the mutex is unlocked by the other thread.
I'm trying to understand condition_variables.
I guess my code should work like:
1. main lock mx
2. main wait() notify <= here lock released
3. threads lock mx
4. threads send notify
5. threads unlock mx
6. main wait() finished and lock mx
So why threads can lock mx faster than wait() call after notify?
Example
#include <iostream>
#include <future>
#include <condition_variable>
#include <vector>
using namespace std::chrono_literals;
std::shared_future<void> ready;
std::mutex finish_mx;
std::condition_variable finish_cv;
int execute(int val, const std::shared_future<void> &ready){
ready.wait();
std::lock_guard<std::mutex> lock(finish_mx);
std::cout<<"Locked: "<<val<<std::endl;
finish_cv.notify_one();
return val;
}
int main()
{
std::promise<void> promise;
auto shared = promise.get_future().share();
std::vector<std::future<int>> pool;
for (int i=0; i<10; ++i){
auto fut = std::async(std::launch::async, execute, i, std::cref(shared));
pool.push_back(std::move(fut));
}
std::this_thread::sleep_for(100ms);
std::unique_lock<std::mutex> finish_lock(finish_mx);
promise.set_value();
for (int i=0; pool.size() > 0; ++i)
{
finish_cv.wait(finish_lock);
std::cout<<"Notifies: "<<i<<std::endl;
for (auto it = pool.begin(); it != pool.end(); ++it) {
auto state = it->wait_for(0ms);
if (state == std::future_status::ready) {
pool.erase(it);
break;
}
}
}
}
example output:
Locked: 6
Locked: 7
Locked: 8
Locked: 9
Locked: 5
Locked: 4
Locked: 3
Locked: 2
Locked: 1
Notifies: 0
Locked: 0
Notifies: 1
Edit
for (int i=0; pool.size() > 0; ++i)
{
finish_cv.wait(finish_lock);
std::cout<<"Notifies: "<<i<<std::endl;
auto it = pool.begin();
while (it != pool.end()) {
auto state = it->wait_for(0ms);
if (state == std::future_status::ready) {
/* process result */
it = pool.erase(it);
} else {
++it;
}
}
}
This depends on how your OS schedules threads that are waiting to acquire a mutex lock. All the execute threads are already waiting to acquire the mutex lock before the first notify_one, so if there's a simple FIFO queue of threads waiting to lock the mutex then they are all ahead of the main thread in the queue. As each mutex unlocks the mutex, the next one in the queue locks it.
This has nothing to do with mutexes being "faster" than condition variables, the condition variable has to lock the same mutex to return from the wait.
As soon as the future becomes ready all the execute threads return from the wait and all try to lock the mutex, joining the queue of waiters. When the condition variable starts to wait the mutex is unlocked, and one of the other threads (the one at the front of the queue) gets the lock. It calls notify_one which causes the condition variable to try to relock the mutex, joining the back of the queue. The notifying thread unlocks the mutex, and the next thread in the queue gets the lock, and calls notify_one (which does nothing because the condition variable is already notified and waiting to lock the mutex). Then the next thread in the queue gets the mutex, and so on.
It seems that one of the execute threads didn't run quickly enough to get in the queue before the first notify_one call, so it ended up in the queue behind the condition variable.
I am running the following chunk of the code. This code is going to create 5 slave threads and 1 main thread. All slave threads are waited for the main thread to make the data ready and when the data gets ready, all slaves will notify to start processing.
My question is, it is possible that before the slave threads start waiting for the conditional_variable, the main thread make the data ready and notify the waited threads. In this case, some threads which were waited will get the notification and start processing but the ones which were not waited, will starting waiting for a notification which will NEVER come.
If you run this example, this case won't happen but I am looking for a way to make sure that all the slave threads are waiting for the notification, then notifying them. Do you know how can I do that?
/*
Condition Variables - Many waiting threads
Shows how one condition variable can be used to notify multiple threads
that a condition has occured.
* Part of "Threading with Boost - Part IV: Condition Variables", published at:
http://antonym.org/boost
Copyright (c) 2015 Gavin Baker <gavinb#antonym.org>
Published under the MIT license, see LICENSE for details
*/
#include <cstdio>
#include <boost/thread.hpp>
boost::condition_variable data_ready_cond;
boost::mutex data_ready_mutex;
bool data_ready = false;
void master_thread()
{
printf("+++ master thread\n");
// Pretend to work
printf(" master sleeping...\n");
boost::chrono::milliseconds sleepDuration(750);
boost::this_thread::sleep_for(sleepDuration);
// Let other threads know we're done
printf(" master notifying...\n");
data_ready = true;
data_ready_cond.notify_all();
printf("--- master thread\n");
}
void slave_thread(int id)
{
printf("+++ slave thread: %d\n", id);
boost::unique_lock<boost::mutex> lock(data_ready_mutex);
while (!data_ready)
{
data_ready_cond.wait(lock);
}
printf("--- slave thread: %d\n", id);
}
int main()
{
printf("Spawning threads...\n");
boost::thread slave_1(slave_thread, 1);
boost::thread slave_2(slave_thread, 2);
boost::thread slave_3(slave_thread, 3);
boost::thread slave_4(slave_thread, 4);
boost::thread master(master_thread);
printf("Waiting for threads to complete...\n");
slave_1.join();
slave_2.join();
slave_3.join();
slave_4.join();
master.join();
printf("Done\n");
return 0;
}
You have race condition - setting flag and notifying slave threads is not atomic. So you just have to lock data_ready_mutex before you are modifying data_ready flag in main thread. This will eliminate race condition, slave thread either will see data_ready false and go to wait on condition variable and will be notified, or it will acquire mutex lock only after data_ready is set to true and so it will not wait at all.
we are working on a project to generate a general thread class that permit us to process a set of interconnected data.
The basic idea is to evaluate in different threads only the datasets that are not connected and that can be simultaneously processed.
We developed a ThreadClass based on boost::thread and a OF_bmutex class based based on boost::mutex in order to perform logging operation.
The scheme of the code is in the linked pdf (http://cdm.unimore.it/dep/test.pd) while the skeleton of the main classes are below...
// encapsulate boost::mutex to log...
class OF_bmutex{
public:
std::string mutex_type;
int m_id;
boost::mutex m;
void lock(){
std::cout << "Mutex " << mutex_type << m_id << " locking from " << boost::this_thread::get_id() << std::endl;
m.lock();
std::cout << "Mutex " << mutex_type << m_id << " locked from " << boost::this_thread::get_id() << std::endl;
}
void unlock(){
std::cout << "Mutex " << mutex_type << m_id << " unlocking from " << boost::this_thread::get_id() << std::endl;
m.unlock();
std::cout << "Mutex " << mutex_type << m_id << " unlocked from " << boost::this_thread::get_id() << std::endl;
}
bool try_lock(){
std::cout << "Mutex " << mutex_type << m_id << " try locking from " << boost::this_thread::get_id() << std::endl;
bool ret = m.try_lock();
if( ret ){
std::cout << "Mutex " << mutex_type << m_id << " try locked from " << boost::this_thread::get_id() << std::endl;
}
return(ret);
}
};
// My thread class
class OF_ThreadClass {
private:
//! running variable
bool running;
//! The thread executing this process...
boost::thread *m_thread;
//! The data to process...
LinkedDataSet *my_data;
//! The id of this thread
int thread_id;
//! Process the data...
virtual int processData();
public:
//! The boost thread id
boost::thread::id boost_id;
//! Thread function
void operator () ();
//! Default constructor
OF_ThreadClass();
//! Destructor
~OF_ThreadClass();
//! Connect this thread with the process data to evaluate
void setProcessData( DataToProcess *pd );
//! return the thread id
int getId() const { return this->thread_id; }
//! post process the thread...
void post_process();
};
// The core function with the execution point of the Thread class...
void OF_ThreadClass::operator () (){
while( this->running ){
OF_AVAILABLE_THREADS_MUTEX[ this->thread_id ]->unlock();
OF_RUNNING_THREADS_MUTEX[ this->thread_id ]->lock();
// PUT HERE OUR CODE...
if( running == true ){
if( my_data != NULL ){
this->processData();
}
this->my_data->done = true;
}
std::cout << ">>>>>> Thread " << thread_id << " notified that evaluation terminated\n";
OF_RUNNING_THREADS_MUTEX[ this->thread_id ]->unlock();
OF_AVAILABLE_THREADS_MUTEX[ this->thread_id ]->lock();
}
OF_AVAILABLE_THREADS_MUTEX[ this->thread_id ]->unlock();
}
// A class to perform multithread calculation...
class OF_SmartThreads{
private:
//! The number of threads to activate
int _n_threads;
//! The polling time
int _polling_time;
//! The thread pool...
std::vector< OF_ThreadClass *> threadPool;
//! The stack of the available threads
std::set< OF_ThreadClass *> *OF_AVAILABLE_THREADS;
//! The set of the running threads
std::set< OF_ThreadClass *> OF_RUNNING_THREADS;
//! The set of the locked datasets
std::set< LinkedDataSet* > locked_data;
//! The set of the available datasets
std::set< LinkedDataSet* > unlocked_data;
//! The set of the datasets under processing
std::set< LinkedDataSet* > processing_data;
//! The size of the progress bar
int progBarDim;
public:
//! Constructor
OF_SmartThreads();
//! Destructor
~OF_SmartThreads();
//! Initialize the SmartThreads
int init_function( std::list< LinkedDataSet * > *dList, int n_max_threads);
//! Initialize the SmartThreads
int init_function( std::set< LinkedDataSet * > *dSet, int n_max_threads);
//! Process all the cuncurrent threads..
int process_data();
//! Process all the cuncurrent threads..
int process_data_polling( int polling_time );
//! stop the process..
int post_process();
};
// Initialization function...
int OF_SmartThreads::init_function( ... ){
// in the main thread...
// Fill the pool of thread mutex...
for(int i = 0; i< _n_threads; i++ ){
_tm = new OF_BMUTEX;
_tm->mutex_type.assign( "A" );
_tm->m_id = i;
OF_AVAILABLE_THREADS_MUTEX.push_back( _tm );
_tm = new OF_BMUTEX;
_tm->mutex_type.assign( "R" );
_tm->m_id = i;
OF_RUNNING_THREADS_MUTEX.push_back( _tm );
}
// Create the threads...
threadPool.resize( _n_threads );
for(int i = 0; i< _n_threads; i++ ){
// ...preventivally lock the resources...
OF_RUNNING_THREADS_MUTEX[i]->lock();
OF_AVAILABLE_THREADS_MUTEX[i]->unlock();
// ..create the new thread...
pc = new OF_ThreadClass;
// insert the new thread in the list...
threadPool.at( pc->getId() ) = pc;
// set it as available...
OF_AVAILABLE_THREADS->insert( pc );
}
}
// Execution function...
void process_data_polling( int _polling_time ){
while ( running ){
if ( something_changed ){
//Print the status on the screen...
...
}
something_changed = false;
// Poll the status of the processing data periodically
boost::this_thread::sleep(boost::posix_time::millisec( _polling_time ));
// Are there some data ready to process?
if( OF_UNLOCKED_DATASETS->size() > 0 ){
// Take the first
pd = *OF_UNLOCKED_DATASETS->begin();
// are there some threads available?
if( OF_AVAILABLE_THREADS->size() != 0 ){
//...lock and move the datasets linked to pd...
ret = lock_data( pd, LOCK );
std::cout << "\tNumber of available threads: " << OF_AVAILABLE_THREADS->size() << std::endl;
// Take the available thread...
pc = *OF_AVAILABLE_THREADS->begin();
// ...link it the dataset to process...
pc->setProcess( pd );
OF_AVAILABLE_THREADS_MUTEX[ pc->getId() ]->lock();
OF_RUNNING_THREADS_MUTEX[ pc->getId() ]->unlock();
something_changed = true;
} // available threads
} // unlock datasets
// Find, unlock and remove finished datasets...
pIter2 = OF_RUNNING_THREADS->begin();
pEnd2 = OF_RUNNING_THREADS->end();
while( pIter2 != pEnd2 ){
pc = *pIter2++;
pd = pc->getDataSet();
if( pd->isDone() ){
//...unlock and move the datasets linked to the current dataset...
ret_move = lock_data( pd, RELEASE_LOCK );
//...remove the data from the active set
ret_remove = OF_ACTIVE_DATASETS->erase( pd );
// make the threads available
moveThreads( pc, _RUNNING_, _AVAILABLE_ );
something_changed = true;
}
}
pIter2 = OF_AVAILABLE_THREADS->begin();
pEnd2 = OF_AVAILABLE_THREADS->end();
while( pIter2 != pEnd2 ){
pc = *pIter2++;
bool obtained = OF_RUNNING_THREADS_MUTEX[ pc->getId() ]->try_lock();
if( obtained ){
std::cout << "\t\t\tOF_SMART_THREADS: Thread " << pc->getId() << " obtained running mutex..." << std::endl;
}
else{
std::cout << "\t\t\tOF_SMART_THREADS: Thread " << pc->getId() << " failed to obtain running mutex..." << std::endl;
}
OF_AVAILABLE_THREADS_MUTEX[ pc->getId() ]->unlock();
std::cout << "\t\t\tOF_SMART_THREADS: Thread " << pc->getId() << " released available mutex..." << std::endl;
}
if( ( OF_LOCKED_DATASETS->size() + OF_UNLOCKED_DATASETS->size() + OF_ACTIVE_DATASETS->size() ) > 0 ){
running = true;
}
else{
running = false;
}
} // end running...
}
// The main function...
int main( int argc, char* argv[]) {
init_function( &data, INT_MAX );
process_data_polling( 100 );
lc.post_process();
return 0;
}
all the system function perfectly when compiled on Linux and OSX with boost 1.53. The number of threads used is 2. An extract of the log is presented below.
Note the mutex logs emitted from the proper threads...
---> LOG FROM OSX ...
---------------------------------
Number of data: 2
Data: 0, links:
Data: 1, links:
---> OF_SmartThreads::init_function --
------------------------------------
--> 8 processors/cores detected.
--> n_max_threads = 2
------------------------------------
Mutex R0 locking from thread master
Mutex R0 locked from thread master
Mutex R0 try locking from thread master
OF_SMART_THREADS: Thread 0 failed to obtain running mutex...
Mutex A0 unlocking from thread master
Mutex A0 unlocked from thread master
New thread 0 created
Mutex R1 locking from thread master
Mutex R1 locked from thread master
Mutex R1 try locking from thread master
OF_SMART_THREADS: Thread 1 failed to obtain running mutex...
Mutex A1 unlocking from thread master
Mutex A1 unlocked from thread master
New thread 1 created
---------------------------------
Available threads: 2
Unlocked datasets: 2
---> OF_SmartThreads::process_data_function
Mutex A1 unlocking from thread1
Mutex A1 unlocked from thread1
>>>>>> Thread 1 released available mutex...
Mutex R1 locking from thread1
Mutex A0 unlocking from thread0
Mutex A0 unlocked from thread0
>>>>>> Thread 0 released available mutex...
Mutex R0 locking from thread0
UNLOCKED DATASETS : 0 1
LOCKED DATASETS :
ACTIVE DATASETS :
RUNNING THREADS :
OF_SMART_THREADS: THREADS AVAILABLE
Number of available threads: 2
OF_SMART_THREADS: take the thread 0
OF_SMART_THREADS: Thread master try to lock available mutex... 0
Mutex A0 locking from thread master
Mutex A0 locked from thread master
OF_SMART_THREADS: Thread obtained available mutex... 0
OF_SMART_THREADS: Thread try to unlock running mutex... 0
Mutex R0 unlocking from thread master
Mutex R0 unlocked from thread master
OF_SMART_THREADS: Thread released running mutex... 0
OF_SMART_THREADS: PREPARE AVAILABLE THREADS
Mutex R1 try locking from thread master
OF_SMART_THREADS: Thread 1 failed to obtain running mutex...
Mutex A1 unlocking from thread master
Mutex A1 unlocked from thread master
OF_SMART_THREADS: Thread 1 released available mutex...
UNLOCKED DATASETS : 1
LOCKED DATASETS :
ACTIVE DATASETS : 0
RUNNING THREADS : 0->0
Mutex R0 locked from thread0
>>>>>> Thread 0 obtained running mutex...
>>>>>> Thread 0 is going to process the dataset 0
>>>>>> Thread 0 terminated to process the dataset 0
>>>>>> Thread 0 notified that evaluation terminated
Mutex R0 unlocking from thread0
Mutex R0 unlocked from thread0
>>>>>> Thread 0 released running mutex...
Mutex A0 locking from thread0
OF_SMART_THREADS: THREADS AVAILABLE
Number of available threads: 1
OF_SMART_THREADS: take the thread 1
OF_SMART_THREADS: Thread master try to lock available mutex... 1
Mutex A1 locking from thread master
Mutex A1 locked from thread master
OF_SMART_THREADS: Thread obtained available mutex... 1
OF_SMART_THREADS: Thread try to unlock running mutex... 1
Mutex R1 unlocking from thread master
Mutex R1 unlocked from thread master
OF_SMART_THREADS: Thread released running mutex... 1
OF_SMART_THREADS: CHECK THREADS DONE
------------> DATASETS 0 done...
------------> DATASETS 0 removed from the active set.
OF_SMART_THREADS: PREPARE AVAILABLE THREADS
Mutex R0 try locking from thread master
Mutex R0 try locked from thread master
OF_SMART_THREADS: Thread 0 obtained running mutex...
Mutex R1 locked from thread1
Mutex A0 unlocking from thread master
>>>>>> Thread 1 obtained running mutex...
Mutex A0 unlocked from thread master
>>>>>> Thread 1 is going to process the dataset 1
Mutex A0 locked from thread0
OF_SMART_THREADS: Thread 0 released available mutex...
>>>>>> Thread 0 obtained available mutex...
UNLOCKED DATASETS :
LOCKED DATASETS :
ACTIVE DATASETS : 1
RUNNING THREADS : 1->1
>>>>>> Thread 1 terminated to process the dataset 1
Mutex A0 unlocking from thread0
>>>>>> Thread 1 notified that evaluation terminated
Mutex A0 unlocked from thread0
Mutex R1 unlocking from thread1
>>>>>> Thread 0 released available mutex...
Mutex R1 unlocked from thread1
Mutex R0 locking from thread0
>>>>>> Thread 1 released running mutex...
Mutex A1 locking from thread1
OF_SMART_THREADS: CHECK THREADS DONE
------------> DATASETS 1 done...
------------> DATASETS 1 removed from the active set.
OF_SMART_THREADS: PREPARE AVAILABLE THREADS
Mutex R0 try locking from thread master
OF_SMART_THREADS: Thread 0 failed to obtain running mutex...
Mutex A0 unlocking from thread master
Mutex A0 unlocked from thread master
OF_SMART_THREADS: Thread 0 released available mutex...
Mutex R1 try locking from thread master
Mutex R1 try locked from thread master
OF_SMART_THREADS: Thread 1 obtained running mutex...
Mutex A1 unlocking from thread master
Mutex A1 unlocked from thread master
OF_SMART_THREADS: Thread 1 released available mutex...
OF_SMART_THREADS: ALL THE DATASETS HAS BEEN SUCCESFULLY PROCESSED...
Mutex A1 locked from thread1
Mutex R0 unlocking from thread master
>>>>>> Thread 1 obtained available mutex...
Mutex R0 unlocked from thread master
Mutex R0 locked from thread0
Mutex A1 unlocking from thread1
>>>>>> Thread 0 obtained running mutex...
Mutex A1 unlocked from thread1
>>>>>> Thread 0 notified that evaluation terminated
>>>>>> Thread 1 released available mutex...
Mutex R0 unlocking from thread0
Mutex R1 locking from thread1
Mutex R0 unlocked from thread0
>>>>>> Thread 0 released running mutex...
Mutex A0 locking from thread0
Mutex A0 locked from thread0
>>>>>> Thread 0 obtained available mutex...
Mutex A0 unlocking from thread0
Mutex A0 unlocked from thread0
>>>>>> Thread 0 is terminating...
Mutex R1 unlocking from thread master
Mutex R1 unlocked from thread master
Mutex R1 locked from thread1
>>>>>> Thread 1 obtained running mutex...
>>>>>> Thread 1 notified that evaluation terminated
Mutex R1 unlocking from thread1
Mutex R1 unlocked from thread1
>>>>>> Thread 1 released running mutex...
Mutex A1 locking from thread1
Mutex A1 locked from thread1
>>>>>> Thread 1 obtained available mutex...
Mutex A1 unlocking from thread1
Mutex A1 unlocked from thread1
>>>>>> Thread 1 is terminating...
The problem arise when compiling system on Windows 7, both with Visual Studio 64 bits and with Mingw 32 bit. As it is possible to see from the log before
there is a deadlock at the beginning. This appears to us very strange and cannot be explained by the mutex logs coming from the different threads.
Some suggestion on how to debug this problem?
---> LOG FROM WINDOWS 7...
---------------------------------
Number of data: 2
Data: 0, links:
Data: 1, links:
-———> OF_SmartThreads::init_function --
------------------------------------
--> 4 processors/cores detected.
--> n_max_threads = 2
------------------------------------
Mutex R0 locking from thread master
Mutex R0 locked from thread master
Mutex R0 try locking from thread master
OF_SMART_THREADS: Thread 0 failed to obtain running mutex...
Mutex A0 unlocking from thread master
Mutex A0 unlocked from thread master
New thread 0 created
Mutex A0 unlocking from thread0
Mutex A0 unlocked from thread0
Mutex R1 locking from thread master
Mutex R1 locked from thread master
>>>>>> Thread 0 released available mutex...
Mutex R0 locking from thread0
Mutex R1 try locking from thread master
OF_SMART_THREADS: Thread 1 failed to obtain running mutex...
Mutex A1 unlocking from thread master
Mutex A1 unlocked from thread master
New thread 1 created
Mutex A1 unlocking from thread1
Mutex A1 unlocked from thread1
---------------------------------
Available threads: 2
>>>>>> Thread 1 released available mutex...
Mutex R1 locking from thread1
Unlocked datasets: 2
---> OF_SmartThreads::process_data_function
UNLOCKED DATASETS : 0 1
LOCKED DATASETS :
ACTIVE DATASETS :
RUNNING THREADS :
OF_SMART_THREADS: THREADS AVAILABLE
Number of available threads: 2
OF_SMART_THREADS: take the thread 0
OF_SMART_THREADS: Thread master try to lock available mutex... 0
Mutex A0 locking from thread master
Mutex A0 locked from thread master
OF_SMART_THREADS: Thread obtained available mutex... 0
OF_SMART_THREADS: Thread try to unlock running mutex... 0
Mutex R0 unlocking from thread master
Mutex R0 unlocked from thread master
Mutex R0 locked from thread0
OF_SMART_THREADS: Thread released running mutex... 0
>>>>>> Thread 0 obtained running mutex...
>>>>>> Thread 0 is going to process the dataset 0
Process Data: delay 41
OF_SMART_THREADS: PREPARE AVAILABLE THREADS
>>>>>> Thread 0 terminated to process the dataset 0
>>>>>> Thread 0 notified that evaluation terminated
Mutex R1 try locking from thread master
OF_SMART_THREADS: Thread 1 failed to obtain running mutex...
Mutex A1 unlocking from thread master
Mutex A1 unlocked from thread master
Mutex R0 unlocking from thread0
Mutex R0 unlocked from thread0
OF_SMART_THREADS: Thread 1 released available mutex...
UNLOCKED DATASETS : 1
LOCKED DATASETS :
ACTIVE DATASETS : 0*
RUNNING THREADS : 0->0
>>>>>> Thread 0 released running mutex...
Mutex A0 locking from thread0
OF_SMART_THREADS: THREADS AVAILABLE
Number of available threads: 1
OF_SMART_THREADS: take the thread 1
OF_SMART_THREADS: Thread master try to lock available mutex... 1
Mutex A1 locking from thread master
There is a deadlock, the thread master cannot lock mutex A1 but, as you can see from the log, no other threads locked that mutex before. Some suggestions on how to debug this problem?
Regards
Add lock monitoring to your OF_bmutex, like bool locked. You should not unclock not locked mutex, or lock locked mutex - so place assert. It seems like your init_function does OF_AVAILABLE_THREADS_MUTEX[i]->unlock(); without prior lock.
Boost BasicLockable Concept:
m.unlock();
Requires: The current thread owns m
So looks like you are violating unlock() preconditions. That can be seen in your log:
Mutex A0 unlocking from thread master
Mutex A0 unlocked from thread master
I'm trying to solve a locking problem with Boost 1.46.1 - I tried a few things but I am not happy - and therefore would love to hear inputs from the clean cut.
Thread A:
Must always wait and obtain a lock for a critical data section
Updates some critical data
Manual unlock (or scoped)
Thread B
- Must never block (try_lock?)
- Reads data from the mentioned critical section if lock obtained
I'm unsure if I need a shared_lock or if I can solve this otherwise.
EDIT, my code looks like:
Thread A:
{
// Critical section
boost::mutex::scoped_lock lock( _mutex) ;
}
Thread B:
boost::mutex::scoped_lock lock(_mutex, boost::try_to_lock);
if( lock ) {
}
But I haven't been able to log a failed try_lock - so I wonder if it really works.
Regards,
Paul
To me it seems like it's working, except that scoped locks are not FIFO, i.e., first thread to request the lock, is not the first thread to get the lock. I used the following code for testing:
#include <boost/thread.hpp>
#include <cstdio>
boost::mutex mutex;
void threadA()
{
for (int i = 10; i > 0; )
{
boost::mutex::scoped_lock lock(mutex);
{
fprintf(stderr, "Locked by A\n");
usleep(100000);
fprintf(stderr, "Unlocked by A\n");
i--;
}
}
}
void threadB()
{
for (int i = 10; i > 0; )
{
boost::mutex::scoped_lock lock(mutex, boost::try_to_lock);
if (lock)
{
fprintf(stderr, "Locked by B\n");
usleep(100000);
fprintf(stderr, "Unlocked by B\n");
i--;
}
else
{
fprintf(stderr, "Lock failed in B\n");
usleep(100000);
}
}
}
int main(int argc, char **argv)
{
boost::thread a(threadA);
boost::thread b(threadB);
a.join();
b.join();
}
And I get the output
Locked by A
Lock failed in B
Lock failed in B
Unlocked by A
Locked by A
Unlocked by A
Locked by A
Lock failed in B
Unlocked by A
Locked by A
Lock failed in B
Unlocked by A
Locked by A
Lock failed in B
Unlocked by A
Locked by A
Lock failed in B
Unlocked by A
Locked by A
Lock failed in B
Unlocked by A
Locked by A
Lock failed in B
Unlocked by A
Locked by A
Lock failed in B
Lock failed in B
Unlocked by A
Locked by A
Unlocked by A
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B
Locked by B
Unlocked by B