This question already has answers here:
On linux, how to make sure to unlock a mutex which was locked in a thread that dies/terminates?
(2 answers)
Closed 6 years ago.
Currently I'm porting a software from Windows to Mac OS X in C++.
In Windows, there's an abandoned state in global named mutex which means that current owner process of the mutex is gone without releasing the mutex. (It will likely be caused by application crash)
Because of abandoned state exists, trying to lock for abandoned mutex will not cause deadlock.
If there's no abandoned state, it will wait forever for a mutex which is not owned by anyone.
There's another approach by using timeout to assume the mutex is abandoned if unable to obtain the mutex for certain time, but it is not a perfect solution compared against abandoned mutex way. In the worst case, accidentally two processes can access to the object locked by the mutex.
Is there any mutex support abandoned state in Mac OS X/Linux?
I researched for the boost library, the boost library has a named mutex, but that one is based on a shared file so it does not have abandoned state.
Please give me some advise.
Well maybe a little late but you can use the pthread_mutexattr_t to set your mutex attribute to be shared between pthread_mutexattr_setpshared(&mutexAttr, PTHREAD_PROCESS_SHARED); API. This mutex value needs to be shared between processes by storing it into a named shared memory.
Here is a code snippet:
int key = ftok(NAMED_MEMORY, ID_TAG);
if (-1 == key)
{
printf("Unable to name shared memory\n");
exit(1);
}
// Create the segment exclusively (if the segment already exists then a combination of IPC_CREAT | IPC_EXCL returns an error EEXIST)
int m_iShmid = shmget(key, TOTAL_SIZE, READ_WRITE_PERMISSIONS | IPC_CREAT | IPC_EXCL);
if (m_iShmid < 0)
{
if (EEXIST == errno)
{
// if the shared memory already exists we only fetch the id to that memory
m_iShmid = shmget(key, TOTAL_SIZE, READ_WRITE_PERMISSIONS);
}
if (m_iShmid < 0)
{
printf("Unable to create shared memory - %s\n",strerror(errno));
exit(1);
}
else
printf("Attached to the existing shared memory\n");
}
else
printf("Created new shared memory\n");
// Now we attach the segment to our data space.
mutex = reinterpret_cast<pthread_mutex_t*>(shmat(m_iShmid, NULL, 0));
if (reinterpret_cast<pthread_mutex_t*>(-1) == mutex)
{
printf("Unable to attach shared memory to the process - %s\n",strerror(errno));
exit(1);
}
// Now we can set this mutex to be shared between processes
pthread_mutex_t* mutex;
pthread_mutexattr_t mutexAttr;
ret = pthread_mutexattr_init(&mutexAttr);
if(ret != 0)
{
printf("pthread_mutexattr_init failed - err=%d\n",ret);
exit(1);
}
ret = pthread_mutexattr_setpshared(&mutexAttr, PTHREAD_PROCESS_SHARED);
if(ret != 0)
{
printf("pthread_mutexattr_setpshared failed - err=%d\n",ret);
exit(1);
}
ret = pthread_mutexattr_setrobust_np(&mutexAttr, PTHREAD_MUTEX_ROBUST_NP);
if(ret != 0)
{
printf("pthread_mutexattr_setrobust_np failed - err=%d\n",ret);
exit(1);
}
ret = pthread_mutex_init(mutex, &mutexAttr);
if(ret != 0)
{
printf("pthread_mutex_init failed - err=%d\n",ret);
exit(1);
}
// ------ Use the mutex from here on between processes
Given that the only answer here uses the ancient ftok()/shmget() approach to acquiring shared memory, I'm going to point you at a library I develop and maintain:
https://github.com/cubiclesoft/cross-platform-cpp
Specifically, you want the 'sync/sync_mutex.*' and 'sync/sync_util.*' files. You get Windows, Mac, Linux, and probably a few variants of those with a single Sync::Mutex class using modern POSIX pthreads and POSIX shared memory for named objects on *NIX-style OSes. The code also handles scenarios such as only letting one thread create and initialize an object and other threads wait until the object is fully initialized before continuing on.
The Sync portion of the library used to use POSIX semaphores, but I discovered that those are quite broken on some OSes while shared memory and pthreads share wider support on those same OSes.
As far as abandoned states go, the OS itself has to take ownership of synchronization objects to handle that particular scenario. That is, a process exits and all acquired synchronization objects get marked as abandoned. The Windows kernel generally handles cleanup of synchronization objects when a process exits. Other OS kernels won't/can't do that. One possible option is to write a system service or driver for other OSes whose sole purpose is to handle abandoned states and clean up and reinitialize objects. Of course, if you figure out a nifty way to handle abandoned objects from within an application itself, let me know/submit a patch/etc.
Related
I have an ancient C++ application originally built in Visual C++ 6.0, that uses a very complex shared-memory DLL to share data among about 8 EXEs and DLLs that all have a pool of values that could be replaced by one or two dictionaries with Strings for the keys, and records for the values. The application is multi-threaded and multi-process. There are three primary executables reading and writing into the shared memory area, and several of the executables have 3 or more threads that read/write or "queue" information into this pooled memory area. About a few hundred places, Structured Exception Handling (SEH) of the __try and __except is used to filter exceptions, and to try to handle Access Violations by resizing the shared memory, which are in segments managed by a class called CGMMF which means growable memory mapped file.
The most salient details are shown here because I cannot find any cohesive source of documentation on the technique in use, or it's safety and suitability. Experimentally I have found that this library didn't work very well on a single core system in 1998, it works somewhat on a single-core virtual machine running windows XP, and that it doesn't work at all on modern 2+ ghz multi-core Windows 7 64-bit systems in 2013. I'm trying to repair it or replace it.
#define ResAddrSpace(pvAddress, dwSize) \
(m_hFileMapRes = CreateFileMapping(HFILE_PAGEFILE, &m_SecAttr, \
PAGE_READWRITE| SEC_RESERVE, 0, dwSize, m_szRegionName), \
(m_hFileMapRes == NULL) ? NULL : \
MapViewOfFileEx(m_hFileMapRes, FILE_MAP_ALL_ACCESS, 0, 0, dwSize, 0))
void CGmmf::Create(void)
{
DWORD dwMaxRgnSize;
if (Gsinf.dwAllocationGranularity == 0)
{
GetSystemInfo(&Gsinf);
}
m_dwFileSizeMax = RoundUp(m_dwFileSizeMax, Gsinf.dwAllocationGranularity);
m_dwFileGrowInc = RoundUp(m_dwFileGrowInc, Gsinf.dwAllocationGranularity);
dwMaxRgnSize = m_dwFileSizeMax + m_dwOverrunBuf;
m_pbFile = (PBYTE)ResAddrSpace(NULL, dwMaxRgnSize);
Adjust(m_dwFileSizeNow);
}
void CGmmf::Adjust(IN DWORD dwDiskFileNow)
{
int nThreadPriority;
__try
{
//
// Boost our thread's priority so that another thread is
// less likely to use the same address space while
// we're changing it.
//
nThreadPriority = GetThreadPriority(GetCurrentThread());
SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL);
//
// Restore the contents with the properly adjusted lengths
//
Construct(dwDiskFileNow);
}
__finally
{
//
// Make sure that we always restore our priority class and thread
// priority so that we do not continue to adversely affect other
// threads in the system.
//
SetThreadPriority(GetCurrentThread(), nThreadPriority);
}
}
void CGmmf::Construct(IN DWORD dwDiskFileNow)
{
DWORD dwDiskFileNew = RoundUp(dwDiskFileNow, m_dwFileGrowInc),
dwStatus = ERROR_SUCCESS;
PBYTE pbTemp;
if (dwDiskFileNew > 0)
{
//
// Grow the MMF by creating a new file-mapping object.
//
// use VirtualAlloc() here to commit
// the requested memory: VirtualAlloc will not fail
// even if the memory block is already committed:
pbTemp = (PBYTE)VirtualAlloc(m_pbFile,dwDiskFileNew,MEM_COMMIT,PAGE_READWRITE);
if(NULL == pbTemp)
{
LogError(GetLastError(), MEM_CREATE_MMF, m_szRegionName);
//
// File-mapping could not be created, the disk is
// probably full.
//
RaiseException(EXCEPTION_GMMF_DISKFULL,
EXCEPTION_NONCONTINUABLE,
0,
NULL);
}
//
// Check to see if our region has been corrupted
// by another thread.
//
if (pbTemp != m_pbFile)
{
RaiseException(EXCEPTION_GMMF_CORRUPTEDRGN,
EXCEPTION_NONCONTINUABLE,
0,
NULL);
}
}
}
So far my options for replacing it include attempting to replace all the shared memory with DCOM (out of process COM) and COM (in process COM) as appropriate to the places where the memory mapped files, and to guard against concurrency issues by hand, using synchronization/mutex/criticalsection or other threadsafe constructs as appropriate.
I want to know if there is already some thread-safe memory-dictionary type I could replace all of this with. Even in the above snippet which is less than 1% of the code of this ancient shared-memory-library-for-visual-C++-6, there are things that make me shudder. For example, raising thread priority as a strategy for avoiding deadlocks, race conditions and general corruption. Maybe that used to make this code stop crashing quite so much on an 80486 CPU at 33 mhz. Shudder.
I have the code building and running in Visual C++ 6.0 and also a branch of it runs in Visual C++ 2008, and I could probably get it going in Visual C++ 2010. What could I use that would give me dictionary semantics, shared memory across processes, and is stable and reliable?
Update By "dictionary" I mean the dictionary datatype as known in Python, which is also called a "key/value store" in some places, and in others (like in the C++ standard library), it's known as std::map. Boost documentation that discusses this is here.
It sounds like you should take a look at Boost Interprocess. You can use it to have std::map-like objects in a shared memory and a lot more. Its been years since I used it last time, so cannot go into much details, but the library documentation is good and has tons of examples, it should get you going in 30 minutes.
I'm not clear about this, can someone confirm this for me?
I have the following synchronization issue. I have the following objects:
A. Process 1, thread 1: Read & write access to the resource.
B. Process 1, thread 2: Read access to the resource.
C. Process 2, thread 3: Read access to the resource.
And here's the access conditions:
A must be blocked while B or C are on.
B must be blocked only while A is on.
C must be blocked only while A is on.
So I thought to use 2 named mutexes for that:
hMutex2 = used to satisfy condition 2 above.
hMutex3 = used to satisfy condition 3 above.
hStopEvent = a stop event (needs to stop the thread if the app is closing).
So for A:
HANDLE hHandles[3] = {hMutex2, hMutex3, hStopEvent};
DWORD dwRes = WaitForMultipleObjects(3, hHandles, FALSE, INFINITE);
if(dwRes == WAIT_OBJECT_0 + 2)
{
//Quit now
return;
}
else if(dwRes == WAIT_OBJECT_0 + 0 ||
dwRes == WAIT_OBJECT_0 + 1)
{
//Do reading & writing here
...
//Release ownership
ReleaseMutex(hMutex2);
ReleaseMutex(hMutex3);
}
else
{
//Error
}
For B:
DWORD dwRes = WaitForSingleObject(hMutex2, INFINITE);
if(dwRes == WAIT_OBJECT_0)
{
//Do reading here
...
//Release ownership
ReleaseMutex(hMutex2);
}
else
{
//Error
}
For C:
DWORD dwRes = WaitForSingleObject(hMutex3, INFINITE);
if(dwRes == WAIT_OBJECT_0)
{
//Do reading here
...
//Release ownership
ReleaseMutex(hMutex3);
}
else
{
//Error
}
Can someone confirm this:
When calling WaitForMultipleObjects on both mutexes, do they both become signaled (or blocked)?
Also do I needs to release both mutexes?
The WaitForMultipleObjects call as written (FALSE for the 3rd parameter) will return when any one of the mutexes is signaled. This means that both the writer and one of the readers could obtain simultaneous access to the resource. One reader could be accessing the resource while the other reader releases its mutex. At that point, the writer would be released.
So to use both mutexes like that, you would need to wait on both of them. However, you cannot just set that third parameter to TRUE since it would mean that it would require hStopEvent to also be signaled in order to release that thread (which is obviously not desired).
One possibility might be to check which mutex was released and then have the writer wait for the other one as well before continuing. Then it would need to release both of them after finishing its task. A problem with this type of solution is that it can start getting complex in a hurry and if you add more processes that need the mutexes, you can end up with deadlock if you are not careful. Using a reader-writer type of lock would simplify the processing quite a bit.
Edit This is not really part of the answer to the question, but depending on the processes involved and how often they will access the resource and how long they will hold the mutex while accessing it, you could really simplify it by using one mutex and just treating it as a critical section ... each process gets it when it needs access to the resource. It of course would not allow both reader threads/processes to have concurrent access, though, so that may or may not be acceptable. But it is a lot easier to verify in the long run.
What you are looking for is reader-writer lock. In your algorithm there is one serious problem - starvation of process A: if B and C keeps working and taking their mutexes, A might be unable to enter.
As a matter of fact, I can contradict it. WaitForMultipleObjects with the waitAll parameter set to FALSE will return if any of the objects are signaled. Here's the documentation :) Set it to TRUE and you'll have it waiting for all objects.
Your solution doesn't scale well, though: add another reading thread, and you're stuck with a third mutex...
The Writer/Readers problem has been solved many times before, however; why not take a look into existing implementations? Will save you a lot of debug time, especially if you're not yet familiar with the windows synchronization API. (Teaser: posix threads have a readwritelock, boost has a shared_mutex.)
I read a few documents about Mutex and still the only Idea I have got is that it helps preventing threads from accessing a resource that is already being used by another resource.
I got from Code snippet and executed which works fine:
#include <windows.h>
#include <process.h>
#include <iostream>
using namespace std;
BOOL FunctionToWriteToDatabase(HANDLE hMutex)
{
DWORD dwWaitResult;
// Request ownership of mutex.
dwWaitResult = WaitForSingleObject(
hMutex, // handle to mutex
5000L); // five-second time-out interval
switch (dwWaitResult)
{
// The thread got mutex ownership.
case WAIT_OBJECT_0:
__try
{
// Write to the database.
}
__finally {
// Release ownership of the mutex object.
if (! ReleaseMutex(hMutex)) {
// Deal with error.
}
break;
}
// Cannot get mutex ownership due to time-out.
case WAIT_TIMEOUT:
return FALSE;
// Got ownership of the abandoned mutex object.
case WAIT_ABANDONED:
return FALSE;
}
return TRUE;
}
void main()
{
HANDLE hMutex;
hMutex=CreateMutex(NULL,FALSE,"MutexExample");
if (hMutex == NULL)
{
printf("CreateMutex error: %d\n", GetLastError() );
}
else if ( GetLastError() == ERROR_ALREADY_EXISTS )
printf("CreateMutex opened existing mutex\n");
else
printf("CreateMutex created new mutex\n");
}
But What I don't understand is where is the thread and where is the shared resource? Can anyone please explain or provide a better article or document?
A mutex provides mutually exclusive access to a resource; in your case, a database. There aren't multiple threads in your program, but you can have multiple instances of your program running, which is what your mutex is protecting against. Effectively, it is still protecting against access from more than one thread, it's just that those threads can be in separate processes.
Your code is creating a named mutex that can be shared across multiple instances of your application. This is a form of interprocess communication. MSDN documentation on CreateMutex has additional helpful information about named mutexes:
Two or more processes can call
CreateMutex to create the same named
mutex. The first process actually
creates the mutex, and subsequent
processes with sufficient access
rights simply open a handle to the
existing mutex...
Multiple processes can have handles of
the same mutex object, enabling use of
the object for interprocess
synchronization.
A mutex is only necessary here if the database you're working against doesn't inherently support multithreaded access.
Maybe It will be the best source to you
http://en.wikipedia.org/wiki/Mutual_exclusion
You can refer this SO post for comparison of various thread synchronization mechanisms
Difference between Locks, Mutex and Critical Sections
If you want specific information Mutex then wikipedia will give you enough details.
This link in msdn provides a similar example as yours with threads made in the main() function. But again the shared resource, which is supposed to be a database is not included.
Anyway, a shared resource is whatever that needs to be accessed from multiple threads: settingsfiles, drivers, database,...
Mind you that the counter in the example is written while protected by the mutex, while it is been read while not being protected. While in this case, there is probably no problem, it is a bit sloppy.
I'm using multithreading in my application with _beginthread and right now to wait until all threads are done I have global bools that get set to true as each thread completes so I'm in a while loop until then. There must be a cleaner way of doing this?
Thanks
You can use WaitForMultipleObjects to wait for the threads to finish in primary thread.
What you want to have a look at is thread synchronization techniques - luckily there is quite a bit of information on MSDN which can probably help you out. It's likely you'll want to use Events and WaitHandles here's the main stuff on MSDN: http://msdn.microsoft.com/en-us/library/ms681924%28v=VS.85%29.aspx there are a number of examples.
There's also some info on synchronization in MFC (which may or may not prove helpful, added for reference purposes): http://msdn.microsoft.com/en-us/library/975t8ks0%28VS.71%29.aspx
I've done a bit of searching, but I've had a hard time trying to track down some helpful info for you which doesn't use the MFC implementation. There's a good tutorial here ( http://www.informit.com/library/content.aspx?b=Visual_C_PlusPlus&seqNum=149 ) but, again, using MFC. You could take a look at the MFC implementation of mutexes though as a start.
So, you'd need to get familiar with synchronization functions and structures - all covered here on MSDN: http://msdn.microsoft.com/en-us/library/ms686679%28v=VS.85%29.aspx
Use _beginthreadex instead. Both _beginthread and _beginthreadex return a thread handle, but the thread started with _beginthread automatically closes its handle when it finishes, so using it for synchronization is not reliable.
Thread handle can be used with one of the synchronization functions of Win32, such as WaitForSingleObject or WaitForMultipleObjects.
When done, handles returned by _beginthreadex must be closed with CloseHandle().
The usual method is to keep all of the thread handles and then wait on each handle. When the handle is signaled, the thread has finished so it is removed from the set of threads. I use std::set<HANDLE> to keep track of the thread handles. There are two different methods for waiting on multiple objects in Windows:
Iterate over the set and call WaitForSingleObject with a timeout on each one
Convert the set into an array or vector and call WaitForMultipleObjects
The first sounds inefficient, but it is actually the most direct and least error prone of the two. If you need to wait for all of the threads, then use the following loop:
std::set<HANDLE> thread_handles; // contains the handle of each worker thread
while (!thread_handles.empty()) {
std::set<HANDLE> threads_left;
for (std::set<HANDLE>::iterator cur_thread=thread_handles.begin(),
last=thread_handles.end();
cur_thread != last; ++cur_thread)
{
DWORD rc = ::WaitForSingleObject(*cur_thread, some_timeout);
if (rc == WAIT_OBJECT_0) {
::CloseHandle(*cur_thread); // necessary with _beginthreadex
} else if (rc == WAIT_TIMEOUT) {
threads_left.add(cur_thread); // wait again
} else {
// this shouldn't happen... try to close the handle and hope
// for the best!
::CloseHandle(*cur_thread); // necessary with _beginthreadex
}
}
std::swap(threads_left, thread_handles);
}
Using WaitForMultipleObjects to wait for the threads to finish is a bit more difficult than it sounds. The following will wait for all of the threads; however, it only waits for WAIT_MAXIMUM_OBJECTS threads at a time. Another options is to loop over each page of threads. I'll leave that exercise to the reader ;)
DWORD large_timeout = (5 * 60 * 1000); // five minutes
std::set<HANDLE> thread_handles; // contains the handle of each worker thread
std::vector<HANDLE> ary; // WaitForMultipleObjects wants an array...
while (!thread_handles.empty()) {
ary.assign(thread_handles.begin(), thread_handles.end());
DWORD rc = ::WaitForMultipleObjects(std::min(ary.size(), WAIT_MAXIMUM_OBJECTS),
&ary[0], FALSE, large_timeout);
if (rc == WAIT_FAILED) {
// handle a failure case... this is usually something pretty bad
break;
} else if (rc == WAIT_TIMEOUT) {
// no thread exited in five minutes... this can be tricky since one of
// the threads beyond the first WAIT_MAXIMUM_OBJECTS may have terminated
} else {
long idx = (rc - WAIT_OBJECT_0);
if (idx > 0 && idx < ary.size()) {
// the object at `idx` was signaled, this means that the
// thread has terminated.
thread_handles.erase(ary[idx]);
::CloseHandle(ary[idx]); // necessary with _beginthreadex
}
}
}
This isn't exactly pretty but it should work. If you trust that all of your threads will exit and don't mind waiting for them, then you can use WaitForMultipleObjects(ary.size(), &ary[0], TRUE, INFINITE). This usually isn't very safe though since a runaway thread will cause your application to block indefinitely and it will only work if ary.size() is less than MAXIMUM_WAIT_OBJECTS.
Of course the other option is to find a thread pool implementation and use it instead. Writing threading code is not really a lot of fun especially once you have to support it in the wild. Consider using something like boost::thread_group instead.
You can use boost::thread objects. Call join on the object and it will wait for the thread to finish.
Windows provides events for one thread to notify another. Out of the box Visual C++ provides support for events only inside MFC. For a portable, non-MFC version, check the thread management classes of the Boost library. They make launching and waiting for threads a lot easier, although they don't provide direct access to all of Windows API's functionality.
What sync mechanism should I use to give exclusive access to the text file in boost?
The file will likely be accessed by threads from only one process.
The file locking APIs are generally for inter process locking. If you are in a single process everything in Boost.Thread package that suits your needs will do. Outside processes the Boost.Interprocess should be used. You might want to read the following warning from Boost.Interprocess:
Caution: Synchronization limitations
If you plan to use file locks just like named mutexes, be careful, because portable file locks have synchronization limitations, mainly because different implementations (POSIX, Windows) offer different guarantees. Interprocess file locks have the following limitations:
It's unspecified if a file_lock synchronizes two threads from the same process.
It's unspecified if a process can use two file_lock objects pointing to the same file.
The first limitation comes mainly from POSIX, since a file handle is a per-process attribute and not a per-thread attribute. This means that if a thread uses a file_lock object to lock a file, other threads will see the file as locked. Windows file locking mechanism, on the other hand, offer thread-synchronization guarantees so a thread trying to lock the already locked file, would block.
The second limitation comes from the fact that file locking synchronization state is tied with a single file descriptor in Windows. This means that if two file_lock objects are created pointing to the same file, no synchronization is guaranteed. In POSIX, when two file descriptors are used to lock a file if a descriptor is closed, all file locks set by the calling process are cleared.
To sum up, if you plan to use file locking in your processes, use the following restrictions:
For each file, use a single file_lock object per process.
Use the same thread to lock and unlock a file.
If you are using a std::fstream/native file handle to write to the file while using file locks on that file, don't close the file before releasing all the locks of the file.
I suppose it is acquire_file_lock
inline bool acquire_file_lock(file_handle_t hnd)
{
struct ::flock lock;
lock.l_type = F_WRLCK;
lock.l_whence = SEEK_SET;
lock.l_start = 0;
lock.l_len = 0;
return -1 != ::fcntl(hnd, F_SETLKW, &lock);
}
It is consistent with a non-boost implementation of a lock.
struct flock fl = {F_WRLCK, SEEK_SET, 0, 0, 0 };
int fd;
fl.l_pid = getpid();
if (argc > 1)
fl.l_type = F_RDLCK;
if ((fd = open("lockdemo.c", O_RDWR)) == -1) {
perror("open");
exit(1);
}
printf("Press <RETURN> to try to get lock: ");
getchar();
printf("Trying to get lock...");
if (fcntl(fd, F_SETLKW, &fl) == -1) {
perror("fcntl");
exit(1);
}
printf("got lock\n");
printf("Press <RETURN> to
If you are sure it will only be accessed from one process, a read-write lock with file handles in thread local storage could be a solution. That would simulate the above with only one writer but several readers.