I want to add logging support to a COM object (DLL) of mine, and there are usually at least two instances of this object loaded. I want both of the DLLs to be able to write lines to the same text file but am worried that this is going to give me problems. Is this possible to achieve? Am I right in thinking that a single Windows API WriteFile call is atomic? Can two processes both open the same file for writing?
I'd like to use the STL for the file handling here (std::ofstream) if possible. My other idea is to use a separate log per-DLL but a single log would be much easier to manage.
#include <iostream>
#include <fstream>
#include <windosws.h>
struct Mutex {
Mutex () {
h = ::CreateMutex(0, false, "{any-GUID-1247965802375274724957}");
}
~Mutex () {
::CloseHandle(h);
}
HANDLE h;
};
Mutex mutex; // GLOBAL mutex
void dll_write_func() {
::WaitForSingleObject(mutex.h, INFINITE);
////////////////
// Write here
std::ofstrem f("output.txt");
f << "Test" << std::endl;
////////////////
::ReleaseMutex(mutex.h);
}
Or
struct MutexLock {
explicit MutexLock(Mutex & m) : m(m) {
::WaitForSingleObject(m.h, INFINITE);
}
~MutexLock() {
::ReleaseMutex(m.h);
}
Mutex & m;
};
void dll_write_func2() {
MutexLock mlock(mutex);
// Write here
std::ofstrem f("output.txt");
f << "Auto mutex Release" << std::endl;
}
Related
I have the following mutex manager which I aims to lock/unlock mutex given a topic name. I want to be able to lock/unlock mutexes depending on a specific tag (in this example a string). What I am doing is simply mapping a string to a mutex. Then, the outside world would invoke MutexManager::lock on tag name, then the MutexManager would lock the correct mutex.
Is this the way to do it, or should I instead be creating a map of std::unique_lock<std::mutex>
#include <iostream>
#include <unordered_map>
#include <mutex>
class MutexManager {
public:
std::unordered_map<std::string, std::mutex> mutexes;
std::unique_lock<std::mutex> lock_mutex(const std::string& name) {
try {
std::unique_lock<std::mutex> lock(mutexes.at(name));
return lock;
} catch (...) {
std::cout << "Failed to acquire lock";
}
}
void unlock_mutex(std::unique_lock<std::mutex> locked_mutex)
{
try {
locked_mutex.unlock();
} catch (...) {
std::cout << "Failed to release lock.";
}
}
void add_mutex(std::string topic) {
mutexes[topic]; // is that really the solution?
}
};
int main()
{
MutexManager mutexManager;
mutexManager.add_mutex("test");
auto& mutexx = mutexManager.mutexes.at("test");
return 0;
}
My concern with the above is if I got two threads where thread 1 runs lock followed by thread2 :
thread 1:
mutexManager.lock("test");
thread 2:
mutexManager.lock("test");
Will thread two be blocked untill thread 1 has released the lock ? In other words, does the locks above target the same mutex given we got the same topic?
I'm trying to solve the next problem:
There are two types of functions. The first type can execute from different threads simultaneously (for example - read data from container). The second one must block all threads inside first-type-function until some operation done (for example - change container).
I am sure this is an easy task, but I can't find a solution. Can anyone please help me?
This is simple example. I expect threads to stop output for 1 second every 3 seconds (I'm aware that this code is unsafe (cout from different threads), but it's just an simple example)
#include <iostream>
#include <thread>
#include <mutex>
using namespace std;
void f() {
// TODO: wait if g() locked this
cout << "Hello from thread " << this_thread::get_id() << endl;
this_thread::sleep_for(100ms);
}
void g() {
// TODO: some lock for f()
cout << "All threads must sleep at night" << endl;
this_thread::sleep_for(1000ms);
// TODO: unlock
}
int main() {
thread t1([]() { for (;;) f(); });
thread t2([]() { for (;;) f(); });
thread t3([]() { for (;;) f(); });
for (;;) {
this_thread::sleep_for(3000ms);
g();
}
return 0;
}
This is a common pattern best solved using reader/writer locks. In C++ you want to use std::shared_lock and imagine that f is a reader and g is a writer.
Declare variable:
std::shared_mutex mutex;
In f:
// Multiple threads can enter the shared lock (readers).
std::shared_lock lock(mutex);
In g:
// Only one thread can enter the unique lock (writers).
std::unique_lock lock(mutex);
I am programming in C++ with the intention to provide some client/server communication between Unreal Engine 4 and my server.
I am in need of a logging system but the current ones are flooded by system messages.
So I made a Logger class with a ofstream object which I do file << "Write message." << endl.
Problem is that each object makes another instance of the ofstream and several longer writes to the file get cut off by newer writes.
I am looking for a way to queue writing to a file, this system/function/stream being easy to include and call.
Bonus points: the ofstream seems to complain whenever I try to write std::string and Fstring :|
log asynchronously using i.e. g2log or using a non-blocking socket wrapper, such as zeromq
ofstream can't be used across multiple threads. It needs to be synchronized using mutex or similar objects. Check the below thread for details:ofstream shared by mutiple threads - crashes after awhile
I wrote a quick example of how you can implement something like that. Please keep in mind that this may not be a final solution and still requires additional error checking and so on ...
#include <concurrent_queue.h>
#include <string>
#include <thread>
#include <fstream>
#include <future>
class Message
{
public:
Message() : text_(), sender_(), quit_(true)
{}
Message(std::string text, std::thread::id sender)
: text_(std::move(text)), sender_(sender), quit_(false)
{}
bool isQuit() const { return quit_; }
std::string getText() const { return text_; }
std::thread::id getSender() const { return sender_; }
private:
bool quit_;
std::string text_;
std::thread::id sender_;
};
class Log
{
public:
Log(const std::string& fileName)
: workerThread_(&Log::threadFn, this, fileName)
{}
~Log()
{
queue_.push(Message()); // push quit message
workerThread_.join();
}
void write(std::string text)
{
queue_.push(Message(std::move(text), std::this_thread::get_id()));
}
private:
static void threadFn(Log* log, std::string fileName)
{
std::ofstream out;
out.open(fileName, std::ios::out);
assert(out.is_open());
// Todo: ... some error checking here
Message msg;
while(true)
{
if(log->queue_.try_pop(msg))
{
if(msg.isQuit())
break;
out << msg.getText() << std::endl;
}
else
{
std::this_thread::yield();
}
}
}
concurrency::concurrent_queue<Message> queue_;
std::thread workerThread_;
};
int main(int argc, char* argv[])
{
Log log("test.txt");
Log* pLog = &log;
auto fun = [pLog]()
{
for(int i = 0; i < 100; ++i)
pLog->write(std::to_string(i));
};
// start some test threads
auto f0 = std::async(fun);
auto f1 = std::async(fun);
auto f2 = std::async(fun);
auto f3 = std::async(fun);
// wait for all
f0.get();
f1.get();
f2.get();
f3.get();
return 0;
}
The main idea is to use one Log class that has a thread safe write() method that may be called from multiple threads simultaneously. The Log class uses a worker thread to put all the file access to another thread. It uses a threadsafe (possibly lock-free) data structure to transfer all messages from the sending thread to the worker thread (I used concurrent_queue here - but there are others as well). Using a small Message wrapper it is very simple to tell the worker thread to shut down. Afterwards join it and everything is fine.
You have to make sure that the Log is not destroyed as long as any thread that may possibly write to it is still running.
I am implementing some file system in C++. Up to now I was using fstream but I realized that it is impossible to open it in exclusive mode. Since there are many threads I want to allow multiple reads, and when opening file in writing mode I want to open the file in exclusive mode?
What is the best way to do it? I think Boost offers some features. And is there any other possibility? I would also like to see simple example. If it is not easy / good to do in C++ I could write in C as well.
I am using Windows.
On many operating systems, it's simply impossible, so C++
doesn't support it. You'll have to write your own streambuf.
If the only platform you're worried about is Windows, you can
possibly use the exclusive mode for opening that it offers.
More likely, however, you would want to use some sort of file
locking, which is more precise, and is available on most, if not
all platforms (but not portably—you'll need LockFileEx
under Windows, fcntl under Unix).
Under Posix, you could also use pthread_rwlock. Butenhof
gives an implementation of this using classical mutex and
condition variables, which are present in C++11, so you could
actually implement a portable version (provided all of the
readers and writers are in the same process—the Posix
requests will work across process boundaries, but this is not
true for the C++ threading primitives).
if your app only works on Windows, the Win32 API function CreateFile() is your choice.
For example:
HANDLE hFile = ::CreateFileW(lpszFileFullPathName, GENERIC_WRITE, FILE_SHARE_READ, NULL, OPEN_EXISTING, NULL, NULL);
If you are open to using boost, then I would suggest you use the file_lock class. This means you want to keep the filename of the files you open/close because fstream does not do so for you.
They have two modes lock() that you can use for writing (i.e. only one such lock at a time, the sharable lock prevents this lock too) and lock_sharable() that you can use for reading (i.e. any number of threads can obtain such a lock).
Note that you will find it eventually complicated to manage both, read and write, in this way. That is, if there is always someone to read, the sharable lock may never get released. In that case, the exclusive lock will never be given a chance to take....
// add the lock in your class
#include <boost/interprocess/sync/file_lock.hpp>
class my_files
{
...
private:
...
boost::file_lock m_lock;
};
Now when you want to access a file, you can lock it one way or the other. If the thread is in charge of when they do that, you could add functions for the user to have access to the lock. If your implementation of the read and write functions in my_files are in charge, you want to get a stack based object that locks and unlocks for you (RAII):
class safe_exclusive_lock
{
public:
safe_exclusive_lock(file_lock & lock)
: m_lock_ref(lock)
{
m_lock_ref.lock();
}
~safe_exclusive_lock()
{
m_lock_ref.unlock();
}
private:
file_lock & m_lock_ref;
};
Now you can safely lock the file (i.e. you lock, do things that may throw, you always unlock before exiting your current {}-block):
ssize_t my_files::read(char *buf, size_t len)
{
safe_exclusive_lock guard(m_lock);
...your read code here...
return len;
} // <- here we get the unlock()
ssize_t my_files::write(char const *buf, size_t len)
{
safe_exclusive_lock guard(m_lock);
...your write code here...
return len;
} // <- here we get the unlock()
The file_lock uses a file, so you will want to have the fstream file already created whenever the file_lock is created. If the fstream file may not be created in your constructor, you probably will want to transform the m_lock variable in a unique pointer:
private:
std::unique_ptr<file_lock> m_lock;
And when you reference it, you now need an asterisk:
safe_exclusive_lock guard(*m_lock);
Note that for safety, you should check whether the pointer is indeed allocated, if not defined, it means the file is not open yet so I would suggest you throw:
if(m_lock)
{
safe_exclusive_lock guard(*m_lock);
...do work here...
}
else
{
throw file_not_open();
}
// here the lock was released so you cannot touch the file anymore
In the open, you create the lock:
bool open(std::string const & filename)
{
m_stream.open(...);
...make sure it worked...
m_lock.reset(new file_lock(filename));
// TODO: you may want a try/catch around the m_lock and
// close the m_stream if it fails or use a local
// variable and swap() on success...
return true;
}
And do not forget to release the lock object in your close:
void close()
{
m_lock.reset();
}
Well you can manually prevent yourself from opening a file if it has been opened in write mode already. Just keep track internally of which files you've opened in write mode.
Perhaps you could hash the filename and store it in a table upon open with write access. This would allow fast lookup to see if a file has been opened or not.
You could rename the file, update it under the new name, and rename it back. I've done it, but it's a little heavy.
Since C++17 there are two options:
In C++23 by using the openmode std::ios::noreplace.
In C++17 by using the std::fopen mode x (exclusive).
Note: The x mode was added to c in C11.
C++23 and later:
#include <cerrno>
#include <cstring>
#include <fstream>
#include <iostream>
int main() {
std::ofstream ofs("the_file", std::ios::noreplace);
if (ofs) {
std::cout << "success\n";
} else {
std::cerr << "Error: " << std::strerror(errno) << '\n';
}
}
Demo
C++17 and later:
#include <cerrno>
#include <cstdio>
#include <cstring>
#include <fstream>
#include <iostream>
#include <memory>
struct FILE_closer {
void operator()(std::FILE* fp) const { std::fclose(fp); }
};
// you may want overloads for `std::filesystem::path`, `std::string` etc too:
std::ofstream open_exclusively(const char* filename) {
bool excl = [filename] {
std::unique_ptr<std::FILE, FILE_closer> fp(std::fopen(filename, "wx"));
return !!fp;
}();
auto saveerr = errno;
std::ofstream stream;
if (excl) {
stream.open(filename);
} else {
stream.setstate(std::ios::failbit);
errno = saveerr;
}
return stream;
}
int main() {
std::ofstream ofs = open_exclusively("the_file");
if (ofs) {
std::cout << "success\n";
} else {
std::cout << "Error: " << std::strerror(errno) << '\n';
}
}
Demo
i have thread pool that create threads each thread worker calculate some work and when it done
it writes the result to file , there is only 1 result file , that each worker thread needs to write to .
now my question is how i guarantee that there wouldn't be any locks or missing write data to the file where allot of threads trying to write to single file ? what is the right strategy for such scenario?
mybe keep all result in memory ? or chanks of results
im using already the QThreadPool framework , and i need to find solution with it.
also i wander , does writing to single file from worker threads , i will have to use
singleton file manager or static class , is it good idea? for multithreaded app ?
So you have many concurrent threads competing for one shared resource. That begs a synchronization primitive of some sort, for example a mutex.
Here's (non-Qt specific) code showcasing 10 threads simultaneously writing to a single file. On a side note, C++11 introduced a lot of goodies like std::mutex (and std::thread too, so that can help eliminate some Qt-specific threading code).
#include <fstream>
#include <mutex>
#include <thread>
#include <vector>
std::mutex m;
std::ofstream file;
int main() {
file.open("file.txt");
std::vector<std::thread> workers;
for (int i = 0; i < 10; ++i) {
workers.push_back(std::thread([=i] {
for (int j = 0; j < 10; ++j) {
std::lock_guard<std::mutex> lock(m);
file << "thread " << i << ": " << j << endl;
}
}));
}
for (auto& worker : workers) {
worker.join();
}
file.close();
return 0;
}
Of course, if you have a lot of places in your code accessing the shared resource, it's better to encapsulate it, along with the mutex, in some "accessor" class that would manage all state-modifying calls to the resource.
P.S. If you're not on C++11 compiler, you can use boost::mutex or Qt-specific mutex wrapper. The key thing is that you need some synchronization primitive associated with the shared resource.
As I assume, you crate your own Runnable, deriving from QRunnable
You can pass some context information when constructing your Runnable class. You need to pass device to write in and mutex to lock device, for example.
class Runnable: public QRunnable
{
public:
Runnable(QIOdevice* device, QMutex* mutex):_d(device), _m(mutex){}
void run()
{
saveInFile();
}
private:
void saveInFile()
{
QMutexLocker lock(_m);
//now only one thread can write to file in the same moment of time
device->write(...);
}
QIOdevice* _d;
QMutex* _m;
};