Using the boost thread library I pass the open ofstream file to each thread as a reference, after half the threads write there is some kind of crash and the program terminates. My assumption is the function reaches the end, closes the file and the remaining threads are trying to write to a closed file. As a test, I added a join statement for the last thread and more of the threads managed to write to file. My experience mutithreading is 2 days - yesterday I got the boost library built and not much more experience in c++ or any language.
I read through the "Questions that may already have your answer" but none answer this question. There are so many posts addressing versions of this problem and just as many approaches to the solution - that is seems more like overthinking and there is a clean way to ensure all the threads have finished before the file closes and that the writes from each thread are buffered in a queue to prevent write clashes.
Some possible solutions:
Rather than pass an open file, pass the file reference and let each
thread open the file, append, and close ofstream myfile("database",
ios::out | ios::app); - from this posted solution "How do I
add..."; this did not work
Reading through the boost thread documentation, there is a
join_all() function but compiled in vs2008 from a boost 1.53.0
version, "error C2039: 'join_all' : is not a member of
'boost::thread'"
Use boost mutex or locks this explanation seems like it is the
asnwer, but I'd first like to know if the crash is due to multiple
writes conflicting or the file closes before the threads have
finished writing.
This c++ ostream::write page referenced multithreading, but just
states that there are no guarantees
This one states something about boost::thread and
boost::function work like no other, but reviewing the
boost::function literature did not help explain what the comment
means
Back to if this is a problem with waiting for all threads to complete
and not a write clash, this discussion provides a solution that
requires storing all the threads, and calling join()
This discusses the windows specific WaitForMultipleObjects but
this is windows specific - the last solution in this post sounds like
the answer but it has no votes and I cannot tell if it is windows
specific or not.
Buffer everything to memory and write-out in a separate function - this solution is in C# but the approach seems plausible. The specifics of their discussion don't make sense to me.
create threads in a loop - seems to have the clearest review; solved using boost::thread_group method given in this thread
There are more forum discussions but they sound like more versions of the previous examples
I'd like a solution that works in both windows and linux; my intuition is to pass the file reference and let each thread append to the file.
What results do you expect to see after uncoordinated writes from several threads to the same stream? That will be garbage after all, even if the stream did survive this torture... You will need to implement some kind of coordination between the writes.
Related
I'm using QReadWriteLock in recursive mode.
This code doesn't by itself make sense, but the issues I have arise from here:
lock->lockForWrite();
lock->lockForRead();
lockForRead is blocked. Note that this is in recursive mode.
The way i see it is that Write is a "superior" lock, it allows me to read and write to the protected data, where Read lock only allows reading.
Also, i think that write lock should not be blocked if the only reader is the same one asking for the write lock.
I can see from the qreadwritelock.cpp source codes that there is no attempt to make it work like what i would like. So it's not a bug, but a feature I find missing.
My question is this, should this kind of recursion be allowed? Are there any problems that arise from this kind of implementation and what would they be?
From QReadWriteLock docs:
Note that the lock type cannot be changed when trying to lock
recursively, i.e. it is not possible to lock for reading in a thread
that already has locked for writing (and vice versa).
So, like you say, it's just the way it works. I personally can't see how allowing reads on the same thread as write locked item would cause problems, but perhaps it requires an inefficient lock implementation?
You could try asking on the QT forums but I doubt you'll get a definitive answer.
Why don't you take the QT source as a starter and have a go at implementing yourself if it's something you need. Writing synchronisation objects can be tricky, but it's a good learning exercise.
I found this question while searching for the same functionality myself.
While thinking about implementing this on my own, I realized that there definitely is a problem arising doing so:
So you want to upgrade your lock from shared (read) to exclusive (write). Doing
lock->unlock();
lock->lockForWrite();
is not what you want, since you want no other thread to gain the write lock right after the current thread released the read lock. But if there Was a
lock->changeModus(WRITE);
or something like that, you will create a deadlock. To gain a write lock, the lock blocks until all current read locks are released. So here, multiple threads will block waiting for each other.
If I have
1. mainThread: write data A,
2. Thread_1: read A and write it to into a Buffer;
3. Thread_2: read from the Buffer.
how to synchronize these three threads safely, with not much performance loss? Is there any existing solution to use? I use C/C++ on linux.
IMPORTANT: the goal is to know the synchronization mechanism or algorithms for this particular case, not how mutex or semaphore works.
First, I'd consider the possibility of building this as three separate processes, using pipes to connect them. A pipe is (in essence) a small buffer with locking handled automatically by the kernel. If you do end up using threads for this, most of your time/effort will be spent on creating nearly an exact duplicate of the pipes that are already built into the kernel.
Second, if you decide to build this all on your own anyway, I'd give serious consideration to following a similar model anyway. You don't need to be slavish about it, but I'd still think primarily in terms of a data structure to which one thread writes data, and from which another reads the data. By strong preference, all the necessary thread locking necessary would be built into that data structure, so most of the code in the thread is quite simple, reading, processing, and writing data. The main difference from using normal Unix pipes would be that in this case you can maintain the data in a more convenient format, instead of all the reading and writing being in text.
As such, what I think you're looking for is basically a thread-safe queue. With that, nearly everything else involved becomes borders on trivial (at least the threading part of it does -- the processing involved may not be, but at least building it with multiple threads isn't adding much to the complexity).
It's hard to say how much experience with C/C++ threads you have. I hate to just point to a link but have you read up on pthreads?
https://computing.llnl.gov/tutorials/pthreads/
And for a shorter example with code and simple mutex'es (lock object you need to sync data):
http://students.cs.byu.edu/~cs460ta/cs460/labs/pthreads.html
I would suggest Boost.Thread for this purpose. This is quite good framework with mutexes and semaphores, and it is multiplatform. Here you can find very good tutorial about this.
How exactly synchronize these threads is another problem and needs more information about your problem.
Edit The simplest solution would be to put two mutexes -- one on A and second on Buffer. You don't have to worry about deadlocks in this particular case. Just:
Enter mutex_A from MainThread; Thread1 waits for mutex to be released.
Leave mutex from MainThread; Thread1 enters mutex_A and mutex_Buffer, starts reading from A and writes it to Buffer.
Thread1 releases both mutexes. ThreadMain can enter mutex_A and write data, and Thread2 can enter mutex_Buffer safely read data from Buffer.
This is obviously the simplest solution, and probably can be improved, but without more knowledge about the problem, this is the best I can come up with.
i have created two classes. One for input reading (through an istream object) and parsing and the other one for processing the output of the parser.
There is one instance of each of those.
I have the parser running in a loop calling istream::get() and then creating commands for the second object based upon the input. These commands are then put on a queue which the second object processes in a separate thread.
Now it is quite obvious that I eventually need to be able to send a "Quit" command. Here the problem arises though: The "Quit" command needs to end the parsing loop as well but I can't find a way to signal the parser that it should quit because it is caught within istream::get().
I would need a way to wake it from that method, but I cannot find any...
I have thought of writing some sort of "termination sequence" to the istream object (which in this case is cin) by creating an ostream object from istream::rdbuf(). But that doesn't work - The badbit is set after the attempt to write to the buffer.
In another question at StackOverflow I saw the asio class of the Boost library mentioned, but I'd rather not depend on third party libraries.
Is there a way to wake the thread from istream::get() - i.e. is there a way to write to the istream buffer (maybe assuming it actually is cin) from within the program?
Another approach would be to kill the thread which I could find acceptable as well since there is no cleanup needed in that specific place. But how can this be done? (I'm relying on a POSIX thread implementation)
You will have to depend on something other than the standard iostream classes, because they don't provide select()-style behaviour.
Also, killing the thread is impossible with POSIX (and utterly broken in Windows). You can issue a cancellation request via pthread_cancel(), but in your case, it may be stuck in an un-cancellable system call. Of particular interest to you, read() may or may not be cancellable, depending on the environment. At least one environment says that a cancellation point may occur in read(), though admittedly it is a Windows POSIX layer. Also, Mac OS X, as recently as Leopard 10.5.1, had a broken read() implementation with respect to cancellability.
Once past this hurdle, you also have to consider the uneasy relationship between C++ destructors and pthread_cancel. Not all environments guarantee that destructors will be called, so you have to be extremely cautions when using pthread_cancel in C++ code.
In short, for interruptible I/O, use low-level I/O and select(): one fd for I/O, a second fd (created by pipe()) for signalling. Or, if you're brave, use AIO, but you're probably better off using a high level interface such as Boost.Asio.
Any chance this is implemented in .NET? - if so take a look at the Reactive Framework.
It provides a very elegant way of handling streams and especially cancelling them on the fly.
On top of this, you get a very extensible library of Linq extension for all sorts of stuff, like Buffering, Memoization, Zip ect..
We use it a lot for transforming (and parsing), modelling of streamed data.
Jeff from the Reative team has a couble of nice blogs about Streaming and Reative here:
I'm looking for a way to do asynchronous and thread-safe logging in my C++ project, if possible to one file. I'm currently using cerr and clog for the task, but since they are synchronous, execution shortly pauses every time something is logged. It's a relatively graphics-heavy app, so this kind of thing is quite annoying.
The new logger should use asynchronous I/O to get rid of these pauses. Thread-safety would also be desirable as I intend to add some basic multithreading soon.
I considered a one-file-per-thread approach, but that seemed like it would make managing the logs a nightmare. Any suggestions?
I noticed this 1 year+ old thread. Maybe the asynchronous logger I wrote could be of interest.
http://www.codeproject.com/KB/library/g2log.aspx
G2log uses a protected message queue to forward log entries to a background worker that the slow disk accesses.
I have tried it with a lock-free queue which increased the average time for a LOG call but decreased the worst case time, however I am using the protected queue now as it is cross-platform. It's tested on Windows/Visual Studio 2010 and Ubuntu 11.10/gcc4.6.
It's released as public domain so you can do with it what you want with no strings attached.
This is VERY possible and practical. How do I know? I wrote exactly that at my last job. Unfortunately (for us), they now own the code. :-) Sadly, they don't even use it.
I intend on writing an open source version in the near future. Meanwhile, I can give you some hints.
I/O manipulators are really just function names. You can implement them for your own logging class so that your logger is cout/cin compatible.
Your manipulator functions can tokenize the operations and store them into a queue.
A thread can be blocked on that queue waiting for chunks of log to come flying through. It then processes the string operations and generates the actual log.
This is intrinsically thread compatible since you are using a queue. However, you still would want to put some mutex-like protection around writing to the queue so that a given log << "stuff" << "more stuff"; type operation remains line-atomic.
Have fun!
I think the proper approach is not one-file-per-thread, but one-thread-per-file. If any one file (or resource in general) in your system is only ever accessed by one thread, thread-safe programming becomes so much easier.
So why not make Logger a dedicated thread (or several threads, one per file, if you're logging different things in different files), and in all other threads, writing to log would place the message on the input queue in the appropriate Logger thread, which would get to it after it's done writing the previous message. All it takes is a mutex to protect the queue from adding an event while Logger is reading an event, and a condvar for Logger to wait on when its queue is empty.
Have you considered using a log library.
There are several available, I discovered Pantheios recently and it really seems to be quite incredible.
It's more a front-end logger, you can customize which system is used. It can interact with ACE or log4cxx for example and it seems really easy to use and configure. The main advantage is that it use typesafe operators, which is always great.
If you just want a barebone logging library:
ACE
log4c*
Boost.Log
Pick any :)
I should note that it's possible to implement lock-free queues in C++ and that they are great for logging.
I had the same issue and I believe I have found the perfect solution. I present to you, a single-header library called loguru: https://github.com/emilk/loguru
It's simple to use, portable, configurable, macro-based and by default doesn't #include anything (for that sweet, sweet compilation times).
I am wondering if you can : lock only a line or a single character in a file in linux and the rest of the file should remain accessible for other processes?
I received a task regarding simulating transaction on a file with c/c++ under linux .
Please give me an answer and if this answer is yes ,give me some links from where i could take a peek to make this task.
Thanks,
Madicemickael
fcntl() is the one API to choose, since it is the least broken and is POSIX. It is the only one that works across NFS. That said it is a complete disaster, too, since locks are bound to processes, not file descriptors. That means that if you lock a file and then some other thread or some library function locks/unlocks it, your lock will be broken too. Also, you cannot use file system locks to protect two threads of the same process to interfere with each other. Also, you should not use file locks on files that are accessible to more than one user, because that effectively enables users to freeze each others processes.
In summary: file locking on Unix creates more problems than it solves. Before you use it you need to be really sure you fully understand the semantics.
Yes, this is possible.
The Unix way to do this is via fcntl or lockf.
Whatever you choose, make sure to use only it and not mix the two. Have a look at this question (with answer) about it: fcntl, lockf, which is better to use for file locking?.
If you can, have a look at section 14.3 in Advanced Programming in the UNIX Environment.
lockf(3) can apply a lock to a section of a file.