Losing messages when multiprocess logging with easylogging++ in C++ - c++

Question

I'm using easylogging++ in my app to log messages for control and I've noticed that in production env (which runs under Linux) some messages were disappearing or missing from the log files. I managed to simulate this problem with a simple example in the test environment (on Windows). I made an infinite thread that just keeps on logging a counter and then I executed two instances of my program, here is a resumed example of my code:
#include "Log/Log.h"
#include <chrono>
#include <thread>
INITIALIZE_EASYLOGGINGPP
void log_test() {
long int count = 0;
while (true) {
log_info("Logando..." + std::to_string(count)); // this is defined in Log.h
boost::this_thread::sleep(boost::posix_time::milliseconds(10));
count++;
}
}
int main(){
std::thread t(log_test);
t.detach();
// rest of the code
}
and Log.h/Log.cpp are:
#pragma once
#include "easylogging++.h"
#include <mutex>
static std::mutex mtx;
void log_info(std::string s);
void log_error(std::string s);
Log.cpp:
#include "Log.h"
void log_info(std::string s)
{
mtx.lock();
LOG(INFO, ELPP_THREAD_SAFE) << s;
mtx.unlock();
}
void log_error(std::string s)
{
mtx.lock();
LOG(ERROR, ELPP_THREAD_SAFE) << s;
mtx.unlock();
}
and the both executable files are using the same .conf file with the following configurations:
* GLOBAL:
FORMAT = "%datetime %msg"
FILENAME = "C:/logs/%datetime{%Y-%M-%d}/msgs.log"
ENABLED = true
TO_FILE = true
TO_STANDARD_OUTPUT = false
SUBSECOND_PRECISION = 6
PERFORMANCE_TRACKING = true
MAX_LOG_FILE_SIZE = 2097152 ## 2MB - Comment starts with two hashes (##)
in the msgs.log file I've noticed this sample:
2022-06-22 18:51:24,886631 Logando...288
2022-06-22 18:51:24,901856 Logando...289
2022-06-22 18:51:24,917820 Logando...5
2022-06-22 18:51:24,932827 Logando...291
2022-06-22 18:51:24,948248 Logando...292
Where the log 290 is missing from the first process and there's just this blank line instead. I guess that one solution could be just using different log files for each process, however it doesn't happen in one single process with multiple threads (instantiating thread t1, t2,t3 as in the code example before). I can't just change one log file to each process in production at the moment since it will have a high impact, so how can I solve it to I don't lose any message at all? Thanks in advance!

Answer 1

I guess that one solution could be just using different log files for each process, however it doesn't happen in one single process with multiple threads (instantiating thread t1, t2,t3 as in the code example before)
Well, threads in a single process share an instance of std::mutex mtx, so they're properly synchronized. Perhaps more importantly, the thing being correctly synchronized is access to a single buffer, which is the only buffer writing to your file.
Two processes will have completely independent instances of std::mutex mtx, which doesn't matter if they're single-threaded, because only one thread is writing to each process's buffer. The problem is that the two buffers are not synchronized with each other when writing to the file, and as mentioned in comments, these writes are apparently not atomic appends.
Solutions are:
Just use threads, since this works already.
Use a shared mutex - this is generally platform specific, but Boost.Interprocess is a good place to start.
Use two files - have some other process tail them both and combine into a single file if you need that
Use two FIFOs for the output files, and have some other process reading them both and combining them into a single file. It avoids duplicating the file storage on disk, but is probably *NIX specific.
Use network sinks (see the easyloggingc++ documentation), and have a process listening to two ports on localhost ... and combining them into a single file. More portable than FIFOs, but also more coding.