First off, it's been a while since I've used any sort of mutex or semaphore, so go easy on me.
I have implemented a generic logging class that right now only receives a message from other classes and prepends that message with date/time and the level of debug, and then prints the message to stdout.
I would like to implement some sort of queue or buffer that will hold many messages that are sent to the logging class and then write them to a file.
The problem that I'm running into is I can't decide how/where to protect the queue.
Below is some pseudo-code of what I've come up with so far:
logMessage(char *msg, int debugLevel){
formattedMsg = formatMsg(msg, debugLevel) //formats the msg to include date/time & debugLevel
lockMutext()
queue.add(formattedMsg)
unlockMutex()
}
wrtieToFile(){
if (isMessageAvailable()) { //would check to see if there is a message in the queue
lockMutext()
file << queue.getFirst() //would append file with the first available msg from the queue
unlockMutex()
}
}
My questions are:
Do I really need to use the mutex in both places?
Is a mutex really what I'm looking for?
I'm thinking I may need a thread for the writing to the file part - does that sound like a good idea?
FYI I looking for a way to do this without using Boost or any 3rd party library.
EDIT The intended platform is Linux.
EDIT 2 Moved formatMsg to before the mutex lock (thank you #Paul Rubel)
With respect to do you really need the mutex. Think what could happen if you didn't lock things. Unless your queue is thread-safe you probably need to protect both insertion and removal.
Imagine execution contexts changing as you are removing the first element. The add could find the queue in a inconsistent state, and then who knows what could happen.
Regarding creating the message, unless formatMsg makes use of shared resources you can probably more it out of the locked section, which can increase your parallelism.
Extracting the writing to file into its own thread sounds like a reasonable choice, that way the logging threads will not have to make the calls themselves.
correct me if i'm wrong. Multiple callers from multiple threads all trying to access the same resource concurrently.
Maybe you could just have one mutex wrapping the entirety of your logging functionality.
watch out for race conditions.
Edit
Readers take a look at the comments to this answer for some valuable discussion
You can define a global variable which contains the number of element present in the queue or buffer. That means you need to increment or decrement this variable while adding data or removing data from buffer or queue. So you keep this variable inside a mutex for your above logging framework.
Related
I am designing a tcp server which takes information from a request and puts everything in a queue to be processed. I am using a asio web server to handle all web interaction. I am looking for an effective way to queue everything up to be processes. I am using boost signals and global vector to do this right now similar to this.
void request_handler::handle_request(request &req, reply &rep)
{
std::string parsedInfo = parse_request(req);
shared_queue.push_back(parsedInfo);
new_entry();
}
new_entry is a boost signal
boost::signal<void ()> new_entry;
Right now I have a signal handler class to catch the signal.
void sig_handler::process_next()
{
boost::try_mutex::scoped_try_lock lock(guard);
if(!lock)
return;
while(!shared_queue.empty())
{
... //Do Stuff
std::string cur_entry = shared_queue.at(0);
shared_queue.erase(shared_queue.begin());
... //Do more stuff
}
}
My goal is to clear out the vector queue when there is information in it, and every time something is pushed on the vector. I would like to avoid polling as much as possible. I believe this part is working how I expect it too. However I am getting an occasional crash from, what I believe based on my backtrace, pushing information on the shared queue. This is only happening when I trying to do 1000's of transactions a second, which makes it hard to debug in a multi-threaded environment. My back trace is here:
Error: signal 11:
./UpdateServer/build/UpdateServer(_Z7handleri+0x18)[0x469f68]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7fbbdd67a4a0]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSsC1ERKSs+0xb)[0x7fbbddfb4f2b]
./UpdateServer/build/UpdateServer[0x4749d0]
./UpdateServer/build/UpdateServer(_ZNSt6vectorISsSaISsEE13_M_insert_auxEN9__gnu_cxx17__normal_iteratorIPSsS1_EERKSs+0x111)[0x476521]
./UpdateServer/build/UpdateServer(_ZN15request_handler14handle_requestERK7requestR5reply+0x3d3)[0x475873]
./UpdateServer/build/UpdateServer(_ZN10connection11handle_readERKN5boost6system10error_codeEm+0x234)[0x46c774]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail14strand_service8dispatchINS1_7binder2INS_3_bi6bind_tIvNS_4_mfi3mf2Iv10connectionRKNS_6system10error_codeEmEENS5_5list3INS5_5valueINS_10shared_ptrIS9_EEEEPFNS_3argILi1EEEvEPFNSK_ILi2EEEvEEEEESB_mEEEEvRPNS2_11strand_implET_+0xcd)[0x47216d]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail15wrapped_handlerINS0_10io_service6strandENS_3_bi6bind_tIvNS_4_mfi3mf2Iv10connectionRKNS_6system10error_codeEmEENS5_5list3INS5_5valueINS_10shared_ptrIS9_EEEEPFNS_3argILi1EEEvEPFNSK_ILi2EEEvEEEEEEclISB_mEEvRKT_RKT0_+0xd9)[0x472439]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail18completion_handlerINS1_17rewrapped_handlerINS1_7binder2INS1_15wrapped_handlerINS0_10io_service6strandENS_3_bi6bind_tIvNS_4_mfi3mf2Iv10connectionRKNS_6system10error_codeEmEENS8_5list3INS8_5valueINS_10shared_ptrISC_EEEEPFNS_3argILi1EEEvEPFNSN_ILi2EEEvEEEEEEESE_mEESV_EEE11do_completeEPNS1_15task_io_serviceEPNS1_25task_io_service_operationESG_m+0x1e5)[0x4726f5]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail14strand_service8dispatchINS1_17rewrapped_handlerINS1_7binder2INS1_15wrapped_handlerINS0_10io_service6strandENS_3_bi6bind_tIvNS_4_mfi3mf2Iv10connectionRKNS_6system10error_codeEmEENS9_5list3INS9_5valueINS_10shared_ptrISD_EEEEPFNS_3argILi1EEEvEPFNSO_ILi2EEEvEEEEEEESF_mEESW_EEEEvRPNS2_11strand_implET_+0x2ad)[0x472aed]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail19asio_handler_invokeINS1_7binder2INS1_15wrapped_handlerINS0_10io_service6strandENS_3_bi6bind_tIvNS_4_mfi3mf2Iv10connectionRKNS_6system10error_codeEmEENS7_5list3INS7_5valueINS_10shared_ptrISB_EEEEPFNS_3argILi1EEEvEPFNSM_ILi2EEEvEEEEEEESD_mEES6_SU_EEvRT_PNS4_IT0_T1_EE+0x15f)[0x472d3f]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail23reactive_socket_recv_opINS0_17mutable_buffers_1ENS1_15wrapped_handlerINS0_10io_service6strandENS_3_bi6bind_tIvNS_4_mfi3mf2Iv10connectionRKNS_6system10error_codeEmEENS7_5list3INS7_5valueINS_10shared_ptrISB_EEEEPFNS_3argILi1EEEvEPFNSM_ILi2EEEvEEEEEEEE11do_completeEPNS1_15task_io_serviceEPNS1_25task_io_service_operationESF_m+0xce)[0x472ede]
./UpdateServer/build/UpdateServer(_ZN5boost4asio6detail15task_io_service3runERNS_6system10error_codeE+0x79a)[0x47ceea]
./UpdateServer/build/UpdateServer(_ZN5boost4asio10io_service3runEv+0x25)[0x47d1d5]
/usr/lib/libboost_thread.so.1.48.0(+0xdda9)[0x7fbbdecd4da9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fbbdd42ee9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fbbdd737cbd]
The line ./UpdateServer/build/UpdateServer(_ZNSt6vectorISsSaISsEE13_M_insert_auxEN9__gnu_cxx17__normal_iteratorIPSsS1_EERKSs+0x111)[0x476521]
seems to demangle to std::vector insert iterator, which is why I believe my program is crashing on the shared vector insert(I do not believe I have any other vectors of strings in my program), however I am fairly positive I am using my vector in a safe way, on both the insert and the read.
So I guess my question is when I am pushing information onto a shared vector, are there race condition problems I have to worry about that would cause my crash? And is the approach I am taking a feasible approach, or should I rethink my design in someway? Please let me know if you need any more information, I will be glad to provide anything I can.
Thank you
std data structures are not thread safe (for the most part) and therefore require additional synchronization if accessed by multiple threads simultaneously. In your case, one thread could be calling push_back while another thread is calling erase. This will produce undefined behavior. To fix this, both the push_back and the erase need to be protected by the same lock. I recommend you google for thread safety in the c++ standard library and read more about it.
Also, using a vector here is probably not the best choice. You should look into std::queue instead. When you erase the first element of the vector is has to copy all of the strings later in the vector down one, which can be very expensive. queue does not suffer from this problem.
I have a situation where 2 different processes(mine C++, other done by other people in JAVA) are a writer and a reader from some shared data file. So I was trying to avoid race condition by writing a class like this(EDIT:this code is broken, it was just an example)
class ReadStatus
{
bool canRead;
public:
ReadStatus()
{
if (filesystem::exists(noReadFileName))
{
canRead = false;
return;
}
ofstream noWriteFile;
noWriteFile.open (noWriteFileName.c_str());
if ( ! noWriteFile.is_open())
{
canRead = false;
return;
}
boost::this_thread::sleep(boost::posix_time::seconds(1));
if (filesystem::exists(noReadFileName))
{
filesystem::remove(noWriteFileName);
canRead= false;
return;
}
canRead= true;
}
~ReadStatus()
{
if (filesystem::exists(noWriteFileName))
filesystem::remove(noWriteFileName);
}
inline bool OKToRead()
{
return canRead;
}
};
usage:
ReadStatus readStatus; //RAII FTW
if ( ! readStatus.OKToRead())
return;
This is for one program ofc, other will have analogous class.
Idea is:
1. check if other program created his "I'm owner file", if it has break else go to 2.
2. create my "I'm the owner" file, check again if other program created his own, if it has delete my file and break else go to 3.
3. do my reading, then delete mine "I'm the owner file".
Please note that rare occurences when they both dont read or write are OK, but the problem is that I still see a small chance of race conditions because theoretically other program can check for the existence of my lock file, see that there isnt one, then I create mine, other program creates his own, but before FS creates his file I check again, and it isnt there, then disaster occurs. This is why I added the one sec delay, but as a CS nerd I find it unnerving to have code like that running.
Ofc I don't expect anybody here to write me a solution, but I would be happy if someone does know a link to a reliable code that I can use.
P.S. It has to be files, cuz I'm not writing entire project and that is how it is arranged to be done.
P.P.S.: access to data file isn't reader,writer,reader,writer.... it can be reader,reader,writer,writer,writer,reader,writer....
P.P.S: other process is not written in C++ :(, so boost is out of the question.
On Unices the traditional way of doing pure filesystem based locking is to use dedicated lockfiles with mkdir() and rmdir(), which can be created and removed atomically via single system calls. You avoid races by never explicitly testing for the existence of the lock --- instead you always try to take the lock. So:
lock:
while mkdir(lockfile) fails
sleep
unlock:
rmdir(lockfile)
I believe this even works over NFS (which usually sucks for this sort of thing).
However, you probably also want to look into proper file locking, which is loads better; I use F_SETLK/F_UNLCK fcntl locks for this on Linux (note that these are different from flock locks, despite the name of the structure). This allows you to properly block until the lock is released. These locks also get automatically released if the app dies, which is usually a good thing. Plus, these will let you lock your shared file directly without having to have a separate lockfile. This, too, work on NFS.
Windows has very similar file locking functions, and it also has easy to use global named semaphores that are very convenient for synchronisation between processes.
As far as I've seen it, you can't reliably use files as locks for multiple processes. The problem is, while you create the file in one thread, you might get an interrupt and the OS switches to another process because I/O is taking so long. The same holds true for deletion of the lock file.
If you can, take a look at Boost.Interprocess, under the synchronization mechanisms part.
While I'm generally against making API calls which can throw from a constructor/destructor (see docs on boost::filesystem::remove) or making throwing calls without a catch block in general that's not really what you were asking about.
You could check out the Overlapped IO library if this is for windows. Otherwise have you considered using shared memory between the processes instead?
Edit: Just saw the other process was Java. You may still be able to create a named mutex that can be shared between processes and used that to create locks around the file IO bits so they have to take turns writing. Sorry I don't know Java so no I idea if that's more feasible than shared memory.
I'm doing a program to handle many blocking I/O operations at a time by spawning an Agent/MailboxProcessor per operation.
I've got a bunch of files I've cached in memory in a Map which I want to share among these agents. However, I've also got a FileSystemWatcher to callback whenever changes are made to the files, so that I can update the cache.
How do I make this happen without risking the cache being corrupted by multi-threaded read and write ?
It seems to me that the Map is already based on pointers to objects, so would that automatically solve my problem as I'm simply changing the pointers to the new objects as they are loaded, or is this a broken understanding of it?
Thanks
It seems to me that the Map is already based on pointers to objects, so would that automatically solve my problem as I'm simply changing the pointers to the new objects as they are loaded, or is this a broken understanding of it?
I think your understanding is correct. You can just have a single mutable reference to an immutable Map. Writing a new map to the reference is atomic so there is no need to synchronize that.
When I've seen similar Erlang programs, the systems are set up like this:
You can wrap up the FileSystemWatcher with a MailboxProcessor, that way you're handling incoming updates as messages and not windows events. Your FileSystemWatcherProcess can hold a list of children who are listening, and push out updates as needed. This is basically the same thing event-based programming, only with messages and actors instead.
Your FileSystemWatcherProcess should not need to maintain a your cache of files, it just blindly pushes out messages.
OR You have a master process which holds the state of the map. File SystemWatcher sends updates to the master. Each child thread holds a reference to the master, so that each time they finish processing an item or batch of items, they send a message to the master process requesting the latest Map.
Neither system requires any locking.
Following up to Jon's answer. if you end up having multiple writers, then instead of locks you can always do CAS:
let updateMap value =
let mutable success = false
while not success do
let v = !x
let result = Interlocked.CompareExchange(x, v.Add(value), v)
success <- Object.ReferenceEqual(v, result)
And, if targeting only .Net 4.0 is an option for you, you shouldn't be inventing all that stuff yourself: there is a System.Collections.Concurrent.ConcurrentDictionary class which implements concurrently readable and writable dictionary already.
I have a thread pool with some threads (e.g. as many as number of cores) that work on many objects, say thousands of objects. Normally I would give each object a mutex to protect access to its internals, lock it when I'm doing work, then release it. When two threads would try to access the same object, one of the threads has to wait.
Now I want to save some resources and be scalable, as there may be thousands of objects, and still only a hand full of threads. I'm thinking about a class design where the thread has some sort of mutex or lock object, and assigns the lock to the object when the object should be accessed. This would save resources, as I only have as much lock objects as I have threads.
Now comes the programming part, where I want to transfer this design into code, but don't know quite where to start. I'm programming in C++ and want to use Boost classes where possible, but self written classes that handle these special requirements are ok. How would I implement this?
My first idea was to have a boost::mutex object per thread, and each object has a boost::shared_ptr that initially is unset (or NULL). Now when I want to access the object, I lock it by creating a scoped_lock object and assign it to the shared_ptr. When the shared_ptr is already set, I wait on the present lock. This idea sounds like a heap full of race conditions, so I sort of abandoned it. Is there another way to accomplish this design? A completely different way?
Edit:
The above description is a bit abstract, so let me add a specific example. Imagine a virtual world with many objects (think > 100.000). Users moving in the world could move through the world and modify objects (e.g. shoot arrows at monsters). When only using one thread, I'm good with a work queue where modifications to objects are queued. I want a more scalable design, though. If 128 core processors are available, I want to use all 128, so use that number of threads, each with work queues. One solution would be to use spatial separation, e.g. use a lock for an area. This could reduce number of locks used, but I'm more interested if there's a design which saves as much locks as possible.
You could use a mutex pool instead of allocating one mutex per resource or one mutex per thread. As mutexes are requested, first check the object in question. If it already has a mutex tagged to it, block on that mutex. If not, assign a mutex to that object and signal it, taking the mutex out of the pool. Once the mutex is unsignaled, clear the slot and return the mutex to the pool.
Without knowing it, what you were looking for is Software Transactional Memory (STM).
STM systems manage with the needed locks internally to ensure the ACI properties (Atomic,Consistent,Isolated). This is a research activity. You can find a lot of STM libraries; in particular I'm working on Boost.STM (The library is not yet for beta test, and the documentation is not really up to date, but you can play with). There are also some compilers that are introducing TM in (as Intel, IBM, and SUN compilers). You can get the draft specification from here
The idea is to identify the critical regions as follows
transaction {
// transactional block
}
and let the STM system to manage with the needed locks as far as it ensures the ACI properties.
The Boost.STM approach let you write things like
int inc_and_ret(stm::object<int>& i) {
BOOST_STM_TRANSACTION {
return ++i;
} BOOST_STM_END_TRANSACTION
}
You can see the couple BOOST_STM_TRANSACTION/BOOST_STM_END_TRANSACTION as a way to determine a scoped implicit lock.
The cost of this pseudo transparency is of 4 meta-data bytes for each stm::object.
Even if this is far from your initial design I really think is what was behind your goal and initial design.
I doubt there's any clean way to accomplish your design. The problem that assigning the mutex to the object looks like it'll modify the contents of the object -- so you need a mutex to protect the object from several threads trying to assign mutexes to it at once, so to keep your first mutex assignment safe, you'd need another mutex to protect the first one.
Personally, I think what you're trying to cure probably isn't a problem in the first place. Before I spent much time on trying to fix it, I'd do a bit of testing to see what (if anything) you lose by simply including a Mutex in each object and being done with it. I doubt you'll need to go any further than that.
If you need to do more than that I'd think of having a thread-safe pool of objects, and anytime a thread wants to operate on an object, it has to obtain ownership from that pool. The call to obtain ownership would release any object currently owned by the requesting thread (to avoid deadlocks), and then give it ownership of the requested object (blocking if the object is currently owned by another thread). The object pool manager would probably operate in a thread by itself, automatically serializing all access to the pool management, so the pool management code could avoid having to lock access to the variables telling it who currently owns what object and such.
Personally, here's what I would do. You have a number of objects, all probably have a key of some sort, say names. So take the following list of people's names:
Bill Clinton
Bill Cosby
John Doe
Abraham Lincoln
Jon Stewart
So now you would create a number of lists: one per letter of the alphabet, say. Bill and Bill would go in one list, John, Jon Abraham all by themselves.
Each list would be assigned to a specific thread - access would have to go through that thread (you would have to marshall operations to an object onto that thread - a great use of functors). Then you only have two places to lock:
thread() {
loop {
scoped_lock lock(list.mutex);
list.objectAccess();
}
}
list_add() {
scoped_lock lock(list.mutex);
list.add(..);
}
Keep the locks to a minimum, and if you're still doing a lot of locking, you can optimise the number of iterations you perform on the objects in your lists from 1 to 5, to minimize the amount of time spent acquiring locks. If your data set grows or is keyed by number, you can do any amount of segregating data to keep the locking to a minimum.
It sounds to me like you need a work queue. If the lock on the work queue became a bottle neck you could switch it around so that each thread had its own work queue then some sort of scheduler would give the incoming object to the thread with the least amount of work to do. The next level up from that is work stealing where threads that have run out of work look at the work queues of other threads.(See Intel's thread building blocks library.)
If I follow you correctly ....
struct table_entry {
void * pObject; // substitute with your object
sem_t sem; // init to empty
int nPenders; // init to zero
};
struct table_entry * table;
object_lock (void * pObject) {
goto label; // yes it is an evil goto
do {
pEntry->nPenders++;
unlock (mutex);
sem_wait (sem);
label:
lock (mutex);
found = search (table, pObject, &pEntry);
} while (found);
add_object_to_table (table, pObject);
unlock (mutex);
}
object_unlock (void * pObject) {
lock (mutex);
pEntry = remove (table, pObject); // assuming it is in the table
if (nPenders != 0) {
nPenders--;
sem_post (pEntry->sem);
}
unlock (mutex);
}
The above should work, but it does have some potential drawbacks such as ...
A possible bottleneck in the search.
Thread starvation. There is no guarantee that any given thread will get out of the do-while loop in object_lock().
However, depending upon your setup, these potential draw-backs might not matter.
Hope this helps.
We here have an interest in a similar model. A solution we have considered is to have a global (or shared) lock but used in the following manner:
A flag that can be atomically set on the object. If you set the flag you then own the object.
You perform your action then reset the variable and signal (broadcast) a condition variable.
If the acquire failed you wait on the condition variable. When it is broadcast you check its state to see if it is available.
It does appear though that we need to lock the mutex each time we change the value of this variable. So there is a lot of locking and unlocking but you do not need to keep the lock for any long period.
With a "shared" lock you have one lock applying to multiple items. You would use some kind of "hash" function to determine which mutex/condition variable applies to this particular entry.
Answer the following question under the #JohnDibling's post.
did you implement this solution ? I've a similar problem and I would like to know how you solved to release the mutex back to the pool. I mean, how do you know, when you release the mutex, that it can be safely put back in queue if you do not know if another thread is holding it ?
by #LeonardoBernardini
I'm currently trying to solve the same kind of problem. My approach is create your own mutex struct (call it counterMutex) with a counter field and the real resource mutex field. So every time you try to lock the counterMutex, first you increment the counter then lock the underlying mutex. When you're done with it, you decrement the coutner and unlock the mutex, after that check the counter to see if it's zero which means no other thread is trying to acquire the lock . If so put the counterMutex back to the pool. Is there a race condition when manipulating the counter? you may ask. The answer is NO. Remember you have a global mutex to ensure that only one thread can access the coutnerMutex at one time.
I have a multithreaded C++ application which holds a complex data structure in memory (cached data).
Everything is great while I just read the data. I can have as many threads as I want access the data.
However the cached structure is not static.
If the requested data item is not available it will be read from database and is then inserted into the data tree. This is probably also not problematic and even if I use a mutex while I add the new data item to the tree that will only take few cycles (it's just adding a pointer).
There is a Garbage Collection process that's executed every now and then. It removes all old items from the tree. To do so I need to lock the whole thing down to make sure that no other process is currently accessing any data that's going to be removed from memory. I also have to lock the tree while I read from the cache so that I don't remove items while they are processed (kind of "the same thing the other way around").
"Pseudocode":
function getItem(key)
lockMutex()
foundItem = walkTreeToFindItem(key)
copyItem(foundItem, safeCopy)
unlockMutex()
return safeCopy
end function
function garbageCollection()
while item = nextItemInTree
if (tooOld) then
lockMutex()
deleteItem(item)
unlockMutex()
end if
end while
end function
What's bothering me: This means, that I have to lock the tree while I'm reading (to avoid the garbage collection to start while I read). However - as a side-effect - I also can't have two reading processes at the same time anymore.
Any suggestions?
Is there some kind of "this is a readonly action that only collides with writes" Mutex?
Look into read-write-lock.
You didn't specify which framework can you use but both pThread and boost have implemented that pattern.
The concept is a "shared reader, single writer" lock as others have stated. In Linux environments you should be able to use pthread_rwlock_t without any framework. I would suggest looking into boost::shared_lock as well.
I suggest a reader-writer lock. The idea is you can acquire a lock for "reading" or for "writing", and the lock will allow multiple readers, but only one writer. Very handy.
In C++17 this type of access (multiple read, single write) is supported directly with std::shared_mutex. I think this was adopted from boost::shared_lock. The topic is also described with an example in Anthony Williams's C++ Concurrency in Action.
https://livebook.manning.com/book/c-plus-plus-concurrency-in-action-second-edition/chapter-3/185.
The essential bits are:
use std::shared_lock on reads where shared access is allowed
existing shared_locks don't prevent a new shared_lock from locking
use std::unique_lock on update/write functions, where exclusive access is needed
will wait until all existing shared reads complete, as well as other writes
precludes any other reads or writes while locked