Guys, I am a beginner in threading and logging.
Btw, I am not a native English speaker, so pardon me if there is any mistake in my English.
I have created a multiple-thread software, where each thread uses logging module like the following:
Each thread uses different log files, so I believe that the chances of data conflicts occurred are 0.
__inline void print_logW(int _level,const wchar_t *domain,const wchar_t *msg)
{
wchar_t mess[200] = _T("");
if(_level<=traceLevel)
{
__time64_t timer;
struct tm t_st;
_time64(&timer);
localtime_s(&t_st,&timer);
if (domain == NULL)
{
domain = _T("");
}
if (msg != NULL)
{
if (showTimeStampFlag == true)
{
swprintf_s(mess,200,_T("%s : %ld"),msg,GetTickCount());
}
else
{
wcscpy_s(mess,200,msg);
}
}
if(oldTime.tm_year != t_st.tm_year || oldTime.tm_mon != t_st.tm_mon || oldTime.tm_mday != t_st.tm_mday)
{
oldTime = t_st;
print_log_preparebyDateW();
}
FILE* fp;
errno_t err = _wfopen_s(&fp, this->m_pathW, _T("at+, ccs=UTF-8"));
if (err != 0)
{
// error
return;
}
fwprintf_s(fp, m_logFormatW,
_level,
1900 + t_st.tm_year, t_st.tm_mon + 1, t_st.tm_mday,
t_st.tm_hour, t_st.tm_min, t_st.tm_sec,
domain, mess
);
fflush(fp);
fclose(fp);
}
}
When I see the log of the software that I made, I found a problem where sometimes the thread process becomes so slow (a process (such as getting pointer of an image) that usually only take 16 ms max, would take 0.2 seconds or more to finish). I am still investigating the cause of this problem, but at first, I would like to know whether the logging module is already thread safe or not.
By the way, for the parameters,
"_level" is the logging level to print or unprint the details of the process
I use "domain" to show the class where the logging is performed
"msg" is the content of the log (e.g. "process 1 started")
And as for the m_logFormatW,
m_logFormatW = _T("[%.2d][%.4d-%.2d-%.2dT%.2d:%.2d:%.2d][%s] %s\n");
If there is any question or anything unclear, feel free to ask.
As long as you are linking to the multi-threaded runtime libraries and oldTime is not a global or static variable your log function will be thread safe. If oldTime is a global or static variable you will need to serialize access to it when you access or modify it otherwise you risk a race condition. The only other thing that may not be thread safe is print_log_preparebyDateW but it's hard to say since you haven't included the code for it. As long as oldTime is not global or static and all the runtime library functions that you use are marked as thread safe or are part of a library marked as thread safe in the MSDN you'll be OK.
The only other problem I can see is when you open the file. If the file is already open and another thread attempts to log information the open call will fail causing the information to be lost. This is because _wfopen_s opens the file without any sharing modes. You can fix this by using std::mutex and locking it while the file is open and unlocking it after the file is closed.
One possible reason your worker threads are taking longer to execute than expected is that opening the log file, writing the information, flushing the file and closing it can take a bit of extra time. This can happen any time file I/O occurs even when caching is involved. Usually you can reduce the time by opening the log file once and then closing it when your application terminates.
Another possible solution to reduce the time it takes your worker threads to execute is to use a pipe. In this scenario you write the log text to a pipe and have an additional thread that reads from the pipe and writes to the log file. This will eliminate any disk I/O that may occur when your worker threads log information. You may encounter some instances where the logging takes a bit of extra time if the pipe is full but it won't happen as often.
Your approach is good for single threaded application and will not work in multithreaded environment as you are not serializing the log message requests.
Its better you look into some well written logger class in opensource such as
a) AsynchronousAndSynchronouslogger - http://www.codeproject.com/Articles/288827/g2log-An-efficient-asynchronous-logger-using-Cplus
b) Simplethreadsafe - http://cpplogging.codeplex.com/
c) Log4Cpp - http://log4cpp.sourceforge.net/
Related
I observe pretty strange behavior in my application.
I have following functions:
bool File::Exists(const std::string & Path)
{
struct stat S;
if(stat(Path.c_str(), &S) != 0)
return false;
if(!S_ISREG(S.st_mode))
return false;
return true;
}
void File::Remove(const std::string & Path)
{
if(unlink(Path.c_str()) != 0)
throw Exceptions::Exception(EXCEPTION_PARAMS, errno);
}
The code i use is:
...
const std::string Path = ...;
if(File::Exists(Path))
File::Remove(Path);
...
And at this point exception is thrown:
No such file or directory (2)
Key facts:
Happens once every 1k-10k calls
All files being removed are binary files, about 20MB
This is called in application thread, however this is the only thread accessing those files (no other threads / process access them). Even no other process / user access partition they are located on.
File being removed is located on mounted CIFS (SMB) (Network) mount point
Why does stat() report file being present, but unlink() sometimes fails?
Why does stat() report file being present, but unlink() sometimes fails?
Because the calls don't happen at the same time.
This bug is so common it has its own name and even a Wikipedia page:
In software development, time of check to time of use (TOCTTOU or
TOCTOU, pronounced "TOCK too") is a class of software bug caused by
changes in a system between the checking of a condition (such as a
security credential) and the use of the results of that check. This is
one example of a race condition.
How intensively are these calls being made? Are you trying to delete thousands of files at a time? Or one or two a second/minute? Over a network you could get all sorts of timeouts that could be interpreted as "No such file or directory", but in actuality is another problem.
And if you're about to delete them anyway, why bother checking for their existence at all? Just delete them, and check the exception to find out if it didn't exist, or whether there was a different error - or not even do that. Either they're gone, or you can't delete them anyway...
In some rare cases (in fact on a single client's computer) code below throws an exception "library_error":
namespace ipc = boost::interprocess;
ipc::shared_memory_object m_shm;
...
bool initAsServer(size_t sharedMemSize)
{
ipc::permissions perm;
perm.set_unrestricted();
try
{
m_shm = ipc::shared_memory_object(
ipc::create_only,
CNameGenHelper::genUniqueNameUtf8().c_str(), // static std::string genUniqueNameUtf8()
ipc::read_write, perm);
}
catch(const ipc::interprocess_exception& ex)
{
logError("failed with exception \"%s\"", ex.what());
return false;
}
...
}
In log file:
[ERR] failed with exception "boost::interprocess_exception::library_error"
Boost v1.58, platform win32, vs13.
I'll be very grateful if you help me in solving this problem. Thank you in advance!
Reason of problem is events with Event ID = "6005" and source name is "EventLog" in "System" Windows log.
Event Viewer - Windows Logs - System.
If the system log does not contain at least one such event, then method boost::interprocess::winapi::get_last_bootup_time() returns false and boost::interprocess::ipcdetail::windows_bootstamp constructor throws exception.
(define BOOST_INTERPROCESS_HAS_KERNEL_BOOTTIME is used).
So it seems that it is enough to clear the "System" windows event log and any application that uses the Boost shared memory will stop working.
What a terrible logic: use the contents of the windows event log.
It seems this boost ipc implementation bug that has not yet been fixed (boost_1_61_0).
My temporary workaround for this case (w/o reboot of computer):
bool fixBoostIpcSharedMem6005issue() const
{
bool result = false;
HANDLE hEventLog = ::RegisterEventSourceA(NULL, "EventLog");
if(hEventLog)
{
const char* msg = "simple boost shared memory fix for 6005";
if(::ReportEventA(hEventLog, EVENTLOG_INFORMATION_TYPE, 0, 6005, NULL, 1, 0, &msg, NULL))
result = true;
::DeregisterEventSource(hEventLog);
}
return result;
}
Use it and try to use ipc::shared_memory_object again :)
Many detailed explanations about by the problem, by one of the authors of the library: Boost interprocess: Getting boot-up time is unreliable on Windows and here: Interprocess get_last_bootup_time use of Event Log on Windows is completely unreliable
Apparently, a reliable solution is to define the preprocessor constant BOOST_INTERPROCESS_SHARED_DIR_PATH to a function call, which always returns the same directory path as a string, once the machine is booted. For example by formatting the update time-stamp of a file, written to at start-up.
You can #define BOOST_INTERPROCESS_BOOTSTAMP_IS_SESSION_MANAGER_BASED or BOOST_INTERPROCESS_BOOTSTAMP_IS_LASTBOOTUPTIME to switch to either registry or WMI based boot time detection.
Alternatively, you can use BOOST_INTERPROCESS_SHARED_DIR_PATH, but it's kind of useless on Windows since it uses hardcoded path. BOOST_INTERPROCESS_SHARED_DIR_FUNC is much better option since it lets you define a function that returns path to the shared directory.
I'm making a multiple-thread program and I want to monitor each threads process using Pantheios.
Since each thread has lots of things going on, I need to separate the logs for each threads. (if not, the log file size would be monstrous)
Using the following function, I tried to separate the log files for each threads
void threadClass::prepareLog()
{
static int counter = 1;
CString strPath = _T("");
strPath.Format(_T("log%d.log"),counter++);
pantheios_be_file_setFilePath(strPath, PANTHEIOS_BEID_LOCAL);
}
But as the result, all the threads logs would only be recorded to the last thread's log file.
I am still searching, but I wonder if Pantheios itself doesn't support multiple log files generation.
By the way, I am using Unicode character set.
If there is anything unclear or if there is any question, feel free to ask.
Thank you for your attention.
My native C++ Win32 program spawns a worker process and needs to pass a huge configuration string to it. Currently it just passes the string as a command line to CreateProcess(). The problem is the string is getting longer and now it doesn't fit into the 32K characters limitation imposed by Windows.
Of course I could do something like complicating the worker process start - I use the RPC server in it anyway and I could introduce an RPC request for passing the configuration string, but this will require a lot of changes and make the solution not so reliable. Saving the data into a file for passing is also not very elegant - the file could be left on the filesystem and become garbage.
What other simple ways are there for passing long strings to a worker process started by my program on Windows?
One possible strategy is to create a named Pipe and pass the handle ( or pipe name) to the other process. Then use normal Read\Write operations on Pipe to extract the data.
There are several good answers already, but the easiest way is to save it in a file, and pass the filename in the command line.
As well as being simple, an advantage of this approach is that the apps will be very loosely coupled (you'll potentially be able to use the child application stand-alone in other ways, rather than always having to launch it from a program that knows how to pipe data into it via a specialised interface)
If you want to be sure that the file is cleaned up after processing, mark it for deletion on the next reboot. THen if anybody forgets to clean it up, the OS will deal with it for you on the next reboot.
I would prefer Boost's message queue. It's extremely simple yet sophisticated. Here's example:
#include <boost/interprocess/ipc/message_queue.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/shared_ptr.hpp>
using namespace boost::interprocess;
// ------------------------------------------------------------------------------
// Your worker:
// ------------------------------------------------------------------------------
try {
message_queue::remove("NAME_OF_YOUR_QUEUE");
boost::shared_ptr<message_queue> mq(new message_queue(create_only, "NAME_OF_YOUR_QUEUE", 65535, 32));
char message[1024];
std::size_t size_received;
unsigned int priority;
if (mq->timed_receive(&message, sizeof(message), size_received, priority, boost::posix_time::ptime(boost::posix_time::second_clock::universal_time()) + boost::posix_time::seconds(1))) {
std::string s(message); // s now contains the message.
}
} catch (std::exception &) {
// ...
}
// ------------------------------------------------------------------------------
// And the sender:
// ------------------------------------------------------------------------------
try {
boost::shared_ptr<message_queue> mq(new message_queue(create_only, "NAME_OF_YOUR_QUEUE", 1024, 1024));
std::stringstream message;
message << "the very very very long message you wish to send over";
while (!mq.try_send(message.str().c_str(), message.str().length(), 0))
::Sleep(33);
} catch (std::exception &) {
// ...
}
Use shared memory. Pass to a worker process name of shared memory object. Another solution is to use WM_COPYDATA message.
How about reading it from stdin :) It seems to work for the Unix folks.
Guaranteed a lot easier than passing pipe names/handles around!
Here is some official code from MSDN for creating child processes with I/O pipes.
Is it a possibility to set up a named shared memory segment?
http://msdn.microsoft.com/en-us/library/aa366551(VS.85).aspx
You could use an inheritable handle to a section object. In your parent process create a section object (CreateFileMapping) and specify that its handle is to be inherited by the child process; then pass the handle value to the child process on the command line. The child process can then open the section object (OpenFileMapping). Though I would prefer a named section object as the semantics of using it are easier to understand.
I am trying to open an output file which I am sure has a unique name but it fails once in a while. I could not find any information for what reasons the ofstream constructor would fail.
EDIT:
It starts failing at some point of time and after that it continuously fails until I stop the running program which write this file.
EDIT:
once in a while = 22-24 hours
code snippet ( I don't this would help but still someone asked for it )
ofstream theFile( sLocalFile.c_str(), ios::binary | ios::out );
if ( theFile.fail() )
{
std::string sErr = " failed to open ";
sErr += sLocalFile;
log_message( sErr );
return FILE_OPEN_FAILED;
}
Too many file handles open? Out of space? Access denied? Intermittent network drive problem? File already exists? File locked? It's awfully hard to say without more details. Edit: Based on the extra details you gave, it sounds like you might be leaking file handles (opening files and failing to close them and so running out of a per-process file handle limit).
I assume that you're familiar with using the exceptions method to control whether iostream failures are communicated as exceptions or as status flags.
In my experience, the iostream classes give very little details on what went wrong when they fail during an I/O operation. However, because they're generally implemented using lower-level Standard C and OS API functions, you can often get at the underlying C or OS error code for more details. I've had good luck using the following function to do this.
std::string DescribeIosFailure(const std::ios& stream)
{
std::string result;
if (stream.eof()) {
result = "Unexpected end of file.";
}
#ifdef WIN32
// GetLastError() gives more details than errno.
else if (GetLastError() != 0) {
result = FormatSystemMessage(GetLastError());
}
#endif
else if (errno) {
#if defined(__unix__)
// We use strerror_r because it's threadsafe.
// GNU's strerror_r returns a string and may ignore buffer completely.
char buffer[255];
result = std::string(strerror_r(errno, buffer, sizeof(buffer)));
#else
result = std::string(strerror(errno));
#endif
}
else {
result = "Unknown file error.";
}
boost::trim_right(result); // from Boost String Algorithms library
return result;
}
You could be out of space, or there could be a permission issue. The OS may have locked the file as well. Try a different name/path for kicks and see if it works then.
One possibility is that you have another instance of the same program running.
Another is that perhaps you run two instances (for debugging purposes?) right after each other, and the OS hasn't finished closing the file and resetting the locks before your next instance of the program comes along and asks for it.