What sync mechanism should I use to give exclusive access to the text file in boost?
The file will likely be accessed by threads from only one process.
The file locking APIs are generally for inter process locking. If you are in a single process everything in Boost.Thread package that suits your needs will do. Outside processes the Boost.Interprocess should be used. You might want to read the following warning from Boost.Interprocess:
Caution: Synchronization limitations
If you plan to use file locks just like named mutexes, be careful, because portable file locks have synchronization limitations, mainly because different implementations (POSIX, Windows) offer different guarantees. Interprocess file locks have the following limitations:
It's unspecified if a file_lock synchronizes two threads from the same process.
It's unspecified if a process can use two file_lock objects pointing to the same file.
The first limitation comes mainly from POSIX, since a file handle is a per-process attribute and not a per-thread attribute. This means that if a thread uses a file_lock object to lock a file, other threads will see the file as locked. Windows file locking mechanism, on the other hand, offer thread-synchronization guarantees so a thread trying to lock the already locked file, would block.
The second limitation comes from the fact that file locking synchronization state is tied with a single file descriptor in Windows. This means that if two file_lock objects are created pointing to the same file, no synchronization is guaranteed. In POSIX, when two file descriptors are used to lock a file if a descriptor is closed, all file locks set by the calling process are cleared.
To sum up, if you plan to use file locking in your processes, use the following restrictions:
For each file, use a single file_lock object per process.
Use the same thread to lock and unlock a file.
If you are using a std::fstream/native file handle to write to the file while using file locks on that file, don't close the file before releasing all the locks of the file.
I suppose it is acquire_file_lock
inline bool acquire_file_lock(file_handle_t hnd)
{
struct ::flock lock;
lock.l_type = F_WRLCK;
lock.l_whence = SEEK_SET;
lock.l_start = 0;
lock.l_len = 0;
return -1 != ::fcntl(hnd, F_SETLKW, &lock);
}
It is consistent with a non-boost implementation of a lock.
struct flock fl = {F_WRLCK, SEEK_SET, 0, 0, 0 };
int fd;
fl.l_pid = getpid();
if (argc > 1)
fl.l_type = F_RDLCK;
if ((fd = open("lockdemo.c", O_RDWR)) == -1) {
perror("open");
exit(1);
}
printf("Press <RETURN> to try to get lock: ");
getchar();
printf("Trying to get lock...");
if (fcntl(fd, F_SETLKW, &fl) == -1) {
perror("fcntl");
exit(1);
}
printf("got lock\n");
printf("Press <RETURN> to
If you are sure it will only be accessed from one process, a read-write lock with file handles in thread local storage could be a solution. That would simulate the above with only one writer but several readers.
Related
I don't know if this is good practice or not but I am doing work on a real time stream of input data and using pthreads in lockstep order to allow one thread at a time to do different operations at the same. This is my program flow for each thread:
void * my_thread() {
pthread_mutex_lock(&read_mutex);
/*
read data from a stream such as stdin into global buffer
*/
pthread_mutex_lock(&operation_mutex);
pthread_mutex_unlock(&read_mutex);
/*
perform some work on the data you read
*/
pthread_mutex_lock(&output_mutex);
pthread_mutex_unlock(&operation_mutex);
/*
Write the data to output such as stdout
*/
pthread_mutex_unlock(&output_mutex);
}
I know there is pthread conditional lock, but is my approach a good idea or a bad idea? I tested this on various size streams and I am trying to think of corner cases to make this deadlock, produce race condition, or both. I know mutexes don't guarantee thread order execution but I need help to think of scenarios that will break this.
UPDATE:
I stepped away from this, but had sometime recently to rethink about this. I rewrote the code using C++ threads and mutexes. I am trying to use condition variables but have no such luck. This is my approach to the problem:
void my_thread_v2() {
//Let only 1 thread read in at a time
std::unique_lock<std::mutex> stdin_lock(stdin_mutex);
stdin_cond.wait(stdin_lock);
/*
Read from stdin stream
*/
//Unlock the stdin mutex
stdin_lock.unlock();
stdin_cond.notify_one();
//Lock step
std::unique_lock<std::mutex> operation_lock(operation_mutex);
operation_cond.wait(operation_lock);
/*
Perform work on the data that you read in
*/
operation_lock.unlock();
operation_cond.notify_one();
std::unique_lock<std::mutex> stdout_lock(stdout_mutex);
stdout_cond.wait(stdout_lock);
/*
Write the data out to stdout
*/
//Unlock the stdout mutex
stdout_lock.unlock();
stdout_cond.notify_one();
}
I know the issue with this code is that there is no way to signal the first condition. I definitely am not understanding the proper use of the condition variable. I looked at various examples on cpp references, but can't seem to get away from the thought that the initial approach maybe the only way of doing what I want to do which is to lock step the threads. Can someone shed some light on this?
UPDATE 2:
So I implemented a simple Monitor class that utilizes C++ condition_variable and unique_lock:
class ThreadMonitor{
public:
ThreadMonitor() : is_occupied(false) {}
void Wait() {
std::unique_lock<std::mutex> lock(mx);
while(is_occupied) {
cond.wait(lock);
}
is_occupied = true;
}
void Notify() {
std::unique_lock<std::mutex> lock(mx);
is_occupied = false;
cond.notify_one();
}
private:
bool is_occupied;
std::condition_variable cond;
std::mutex mx;
};
This is my initial approach assuming i have three ThreadMonitors called stdin_mon, operation_mon, and stdout_mon:
void my_thread_v3() {
//Let only 1 thread read in at a time
stdin_mon.Wait();
/*
Read from stdin stream
*/
stdin_mon.Notify();
operation_mon.Wait();
/*
Perform work on the data that you read in
*/
operation_mon.Notify();
stdout_mon.Wait();
/*
Write the data out to stdout
*/
//Unlock the stdout
stdout_mon.notify();
}
The issue with this was that the data was still being corrupted so I had to change back to the original logic of lock stepping the threads:
void my_thread_v4() {
//Let only 1 thread read in at a time
stdin_mon.Wait();
/*
Read from stdin stream
*/
operation_mon.Wait();
stdin_mon.Notify();
/*
Perform work on the data that you read in
*/
stdout_mon.Wait();
operation_mon.Notify();
/*
Write the data out to stdout
*/
//Unlock the stdout
stdout_mon.notify();
}
I am beginning to suspect that if thread order matters that this is the only way to handle it. I am also questioning what the benefit is of using a Monitor that utilizes condition_variable over just using a mutex.
The problem with your approach is that you still can modify the data while another thread is reading it:
Thread A acquired read, then operation and released read again, and starts writing some data, but is interrupted.
Now thread B operates, acquires read and can read the partially modified, possibly inconsistent data!
I assume you want to allow multiple threads reading the same data without blocking, but as soon as writing, the data shall be protected. Finally, while outputting data, we are just reading the modified data again and thus can do this concurrently again, but need to prevent simultaneous write.
Instead of having multiple mutex instances, you can do this better with a read/write mutex:
Any function only reading the data acquires the read lock.
Any function intending to write acquires write lock right from the start (be aware that first acquiring read, then write lock without releasing the read lock in between can result in dead-lock; if you release read lock in between, though, your data handling needs to be robust against data being modified by another thread in between as well!).
Reducing write lock to shared without releasing in between is safe, so we can do so now before outputting. If data must not be modified in between writing data and outputting it, we even need to do this without entirely releasing the lock.
Last point is problematic as not supported neither by C++ standard's thread support library nor by pthreads library.
For C++ boost provides a solution; if you don't want to or cannot (C!) use boost a simple, but possibly not most efficient approach would be protecting acquiring write lock via another mutex:
acquire standard (non-rw) mutex protecting the read write mutex
acquire RW mutex for writing
release protecting mutex
read data, write modified data
acquire protecting mutex
release RW mutex
re-acquire RW mutex for reading; it does not matter if another thread acquired for reading as well, we only need to protect against locking for write here
release protecting mutex
output
release RW mutex (no need to protect)...
Non-modifying functions can just acquire the read lock without any further protection, there aren't any conflicts with...
In C++, you'd prefer using the thread support library and additionally gain platform independent code for free, in C, you would use a standard pthread mutex for protecting acquiring the write lock just as you did before and use the RW variants from pthread for the read write lock.
This question already has answers here:
On linux, how to make sure to unlock a mutex which was locked in a thread that dies/terminates?
(2 answers)
Closed 6 years ago.
Currently I'm porting a software from Windows to Mac OS X in C++.
In Windows, there's an abandoned state in global named mutex which means that current owner process of the mutex is gone without releasing the mutex. (It will likely be caused by application crash)
Because of abandoned state exists, trying to lock for abandoned mutex will not cause deadlock.
If there's no abandoned state, it will wait forever for a mutex which is not owned by anyone.
There's another approach by using timeout to assume the mutex is abandoned if unable to obtain the mutex for certain time, but it is not a perfect solution compared against abandoned mutex way. In the worst case, accidentally two processes can access to the object locked by the mutex.
Is there any mutex support abandoned state in Mac OS X/Linux?
I researched for the boost library, the boost library has a named mutex, but that one is based on a shared file so it does not have abandoned state.
Please give me some advise.
Well maybe a little late but you can use the pthread_mutexattr_t to set your mutex attribute to be shared between pthread_mutexattr_setpshared(&mutexAttr, PTHREAD_PROCESS_SHARED); API. This mutex value needs to be shared between processes by storing it into a named shared memory.
Here is a code snippet:
int key = ftok(NAMED_MEMORY, ID_TAG);
if (-1 == key)
{
printf("Unable to name shared memory\n");
exit(1);
}
// Create the segment exclusively (if the segment already exists then a combination of IPC_CREAT | IPC_EXCL returns an error EEXIST)
int m_iShmid = shmget(key, TOTAL_SIZE, READ_WRITE_PERMISSIONS | IPC_CREAT | IPC_EXCL);
if (m_iShmid < 0)
{
if (EEXIST == errno)
{
// if the shared memory already exists we only fetch the id to that memory
m_iShmid = shmget(key, TOTAL_SIZE, READ_WRITE_PERMISSIONS);
}
if (m_iShmid < 0)
{
printf("Unable to create shared memory - %s\n",strerror(errno));
exit(1);
}
else
printf("Attached to the existing shared memory\n");
}
else
printf("Created new shared memory\n");
// Now we attach the segment to our data space.
mutex = reinterpret_cast<pthread_mutex_t*>(shmat(m_iShmid, NULL, 0));
if (reinterpret_cast<pthread_mutex_t*>(-1) == mutex)
{
printf("Unable to attach shared memory to the process - %s\n",strerror(errno));
exit(1);
}
// Now we can set this mutex to be shared between processes
pthread_mutex_t* mutex;
pthread_mutexattr_t mutexAttr;
ret = pthread_mutexattr_init(&mutexAttr);
if(ret != 0)
{
printf("pthread_mutexattr_init failed - err=%d\n",ret);
exit(1);
}
ret = pthread_mutexattr_setpshared(&mutexAttr, PTHREAD_PROCESS_SHARED);
if(ret != 0)
{
printf("pthread_mutexattr_setpshared failed - err=%d\n",ret);
exit(1);
}
ret = pthread_mutexattr_setrobust_np(&mutexAttr, PTHREAD_MUTEX_ROBUST_NP);
if(ret != 0)
{
printf("pthread_mutexattr_setrobust_np failed - err=%d\n",ret);
exit(1);
}
ret = pthread_mutex_init(mutex, &mutexAttr);
if(ret != 0)
{
printf("pthread_mutex_init failed - err=%d\n",ret);
exit(1);
}
// ------ Use the mutex from here on between processes
Given that the only answer here uses the ancient ftok()/shmget() approach to acquiring shared memory, I'm going to point you at a library I develop and maintain:
https://github.com/cubiclesoft/cross-platform-cpp
Specifically, you want the 'sync/sync_mutex.*' and 'sync/sync_util.*' files. You get Windows, Mac, Linux, and probably a few variants of those with a single Sync::Mutex class using modern POSIX pthreads and POSIX shared memory for named objects on *NIX-style OSes. The code also handles scenarios such as only letting one thread create and initialize an object and other threads wait until the object is fully initialized before continuing on.
The Sync portion of the library used to use POSIX semaphores, but I discovered that those are quite broken on some OSes while shared memory and pthreads share wider support on those same OSes.
As far as abandoned states go, the OS itself has to take ownership of synchronization objects to handle that particular scenario. That is, a process exits and all acquired synchronization objects get marked as abandoned. The Windows kernel generally handles cleanup of synchronization objects when a process exits. Other OS kernels won't/can't do that. One possible option is to write a system service or driver for other OSes whose sole purpose is to handle abandoned states and clean up and reinitialize objects. Of course, if you figure out a nifty way to handle abandoned objects from within an application itself, let me know/submit a patch/etc.
i am new to multithreading in C(LINUX)
i am doing a multiple client server(single) program ,here i am using threads for execution of the server,so i need when a client is waiting for reply other threads(other server threads) should not run,
while(n = read(conn->sock, buffer, sizeof(buffer)) > 0 )
{
//HERE I NEED THE LOCK THE OTHER THREADS FROM THEIR EXECUTION
//process
//process
//end of process
//HERE I NEED TO RELEASE LOCK FOR THE OTHER THREADS EXECUTION
}
}
i did not find anything specific on the net,even some example URL will be helpful
For this you can use e.g. condition variables, where you can notify all waiting threads.
You can use conditions variables from POSIX pthreads if you need pure C.
https://computing.llnl.gov/tutorials/pthreads/#ConditionVariables
You should use Mutual Exclusion (mutex). If you're on Windows I would use EnterCriticalSection on a critical section object.
You can also use std::mutex, which has been added in C++11 to try and create a standardized technique for this sort of thing.
Basically you have the thread that needs to transmit or access something take 'ownership' of the mutex and all other threads would check before taking action to see if the mutex is already owned. If the mutex is owned the other threads would wait until it is released thus waiting their turn to take action.
It is highly advisable to use the operating system built-in method for doing this like my suggestion for Windows. If you don't, you will not have the same level of fairness. Most operating systems have built in optimizations for this while the STL objects may not.
Edit:
I some how missed the Linux tag but AlexBG provided the link to POSIX built in mutex usage: https://computing.llnl.gov/tutorials/pthreads/#ConditionVariables
Search for thread synchronization.
global variables will be shared by threads so use a global variable and then can acquire lock using that variable.
To protect the concurrent access to a shared resource a simple (fast) mutex would do.
#include <pthread.h>
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
[...]
void some_func(void)
{
[...]
pthread_mutex_lock(&mutex);
/* Access to shared resource here (print to stdout for example). */
pthread_mutex_unlock(&mutex);
[...]
}
Please note that this code lacks error checking for the sake of readability.
How can i lock a file for read and write operation. That is If "ABC" file name is in Write lock, it also provide Read Lock on the same locked file. In normal case we want to wait till write operation completion.So if there any ways to acquire this kind of locking
Many programs simply use a lock file to signify that a certain file is currently in use for writing.
The lock file is later removed when done writing.
For example, when process #1 is about to start writing to file example, it creates file example.lock. Later when done writing, it simply removes example.lock.
When process #2 want to read from file example it first checks if file example.lock exists. If it does then the file is locked for write operations and process #2 will have to wait.
shared_mutex from Boost implements read/write locking.
I read a few documents about Mutex and still the only Idea I have got is that it helps preventing threads from accessing a resource that is already being used by another resource.
I got from Code snippet and executed which works fine:
#include <windows.h>
#include <process.h>
#include <iostream>
using namespace std;
BOOL FunctionToWriteToDatabase(HANDLE hMutex)
{
DWORD dwWaitResult;
// Request ownership of mutex.
dwWaitResult = WaitForSingleObject(
hMutex, // handle to mutex
5000L); // five-second time-out interval
switch (dwWaitResult)
{
// The thread got mutex ownership.
case WAIT_OBJECT_0:
__try
{
// Write to the database.
}
__finally {
// Release ownership of the mutex object.
if (! ReleaseMutex(hMutex)) {
// Deal with error.
}
break;
}
// Cannot get mutex ownership due to time-out.
case WAIT_TIMEOUT:
return FALSE;
// Got ownership of the abandoned mutex object.
case WAIT_ABANDONED:
return FALSE;
}
return TRUE;
}
void main()
{
HANDLE hMutex;
hMutex=CreateMutex(NULL,FALSE,"MutexExample");
if (hMutex == NULL)
{
printf("CreateMutex error: %d\n", GetLastError() );
}
else if ( GetLastError() == ERROR_ALREADY_EXISTS )
printf("CreateMutex opened existing mutex\n");
else
printf("CreateMutex created new mutex\n");
}
But What I don't understand is where is the thread and where is the shared resource? Can anyone please explain or provide a better article or document?
A mutex provides mutually exclusive access to a resource; in your case, a database. There aren't multiple threads in your program, but you can have multiple instances of your program running, which is what your mutex is protecting against. Effectively, it is still protecting against access from more than one thread, it's just that those threads can be in separate processes.
Your code is creating a named mutex that can be shared across multiple instances of your application. This is a form of interprocess communication. MSDN documentation on CreateMutex has additional helpful information about named mutexes:
Two or more processes can call
CreateMutex to create the same named
mutex. The first process actually
creates the mutex, and subsequent
processes with sufficient access
rights simply open a handle to the
existing mutex...
Multiple processes can have handles of
the same mutex object, enabling use of
the object for interprocess
synchronization.
A mutex is only necessary here if the database you're working against doesn't inherently support multithreaded access.
Maybe It will be the best source to you
http://en.wikipedia.org/wiki/Mutual_exclusion
You can refer this SO post for comparison of various thread synchronization mechanisms
Difference between Locks, Mutex and Critical Sections
If you want specific information Mutex then wikipedia will give you enough details.
This link in msdn provides a similar example as yours with threads made in the main() function. But again the shared resource, which is supposed to be a database is not included.
Anyway, a shared resource is whatever that needs to be accessed from multiple threads: settingsfiles, drivers, database,...
Mind you that the counter in the example is written while protected by the mutex, while it is been read while not being protected. While in this case, there is probably no problem, it is a bit sloppy.