I developed a logger for testing our modules in c++, Win32 console, visual studio(Windows)
Logger is running in one thread.
While it displays output in console window, thread is getting preempted.
Some other module thread is running.
So output of other modules is getting mixed with output of Logger module in Console window.
Is there any way to avoid preemption of logger thread, so that entire logger output can be at one place in console output window ?
Writing to file instead of output window is one solution. But as the drive names may be different on different machines, its difficult to hardcode the logger output file path. Even then, we can still write the code for finding the drives available on any machine and write to first drive etc. But tester may not understand where to search for the logger output file.
Add the thread Id to the logger output, and then use a log viewer that can filter.
DebugView (under windows) allows you to add highlight filters to dynamic logging.
The standard solution is to use a mutex . After formatting, but before starting the output to the console, you lock the mutex. When all output is sent, you unlock the mutex again. If a second thread comes in, its attempt to lock the mutex will cause that thread to be preempted until the first thread is done.
CriticalSections in Windows behave mutex-like and are also usable. They use a slightly different terminology. You don't "lock" them, you "enter" and "leave" a critical section with EnterCriticalSection and LeaveCriticalSection.
Preventing Thread preemption is generally dangerous. You can try to temporarily increase the thread priority, but i dont advise it (Dangerous; will not work on multiprocessor, ...).
Other Ways:
rewrite all modules to only use your logger for output
if other modules only write to cout/stdout: logger should write to cerr/stderr. This will not prevent the intermingled output in the console. But when redirecting the program output to different files, it will.
I think the best solution is to simply separate the logger output from the rest of your program's output. You mentioned the possibility of writing the logging to a file. If your only hang-up with this solution is coding an appropriate path, then you can choose the output path dynamically:
TCHAR buffer[ MAX_PATH ];
SHGetSpecialFolderPath( NULL, buffer, CSIDL_LOCAL_APPDATA, TRUE );
This will give you the local app data folder for the current user. You can then just append your log file name.
Related
In my c++ windows app I start multiple child processes and I want them to inherit parent's stdout/stderr, so that if output of my app is redirected to some file then that file would also contain output of all child processes that my app creates.
Currently I do that using CreateProcess without output redirection. MSDN has a sample how to redirect output: Creating a Child Process with Redirected Input and Output, but I want to see what alternative do I have. Simplest is to use system and call it from a blocking thread that waits for child to exit. All output is then piped back to parent's stdout/stderr, however in parent process I do not have a chance to process stdout data that comes from child.
There are also other functions to start processes on windows: spawn, exec, which might be easier to port to posix systems.
What should I use if I want it to work on linux/osx? What options do I have if I want it to work on UWP aka WinRT? I might be totally ok with system called from a blocking thread, but perhaps I'd prefer to be able to have more control on process PID (to be able to terminate it) and process stdout/stderr, to prepend each line with child##: for example.
The boost libraries recently released version 1.64 which includes a new boost::process library.
In it, you're given a C++ way to be able to redirect output to a pipe or asio::streambuf, from which you can create a std::string or std::istream to read whatever your child process wrote.
You can read up on boost::process tutorials here, which shows some simple examples of reading child output. It does make heavy use of boost::asio, so I highly recommend you read up on that too.
I'm writing a program with C++ and by native Win32 API. I'm creating a process from a SFX archive EXE in silent mode that no GUI is shown to user. But I want to show a progress bar in my application, while the SFX archive extracting.
How can I do that?
Thanks.
If the process you create produces some textual output to the standard output then you can probably parse that output somehow and show the progress. To know if it does, activate it in a command line windows and watch what you get from it.
win32's CreateProcess() allows you to redirect the standard output of the process to a pipe. This way you can receive the output as soon as it is produced.
If the process you're creating doesn't report its progress somehow then there's really not much you can do. You can try to come up with function between the size of the file and the average time it takes to extract it and then fake a progress bar. That will serve the purpose of setting the user's mind at ease but nothing more.
--Edit
The call to CreateProcess() returns as soon as the process is created. CreateProcess() fills up the struct PROCESS_INFORMATION with the handles of the process it creates. it contains the handle of the main thread of the process. If you want to wait for the process to finish you can WaitForSingleEvent() on that thread handle which gets signaled when the thread terminates. Don't forget to CloseHandle() those handles when you're done with them.
What's the best/proper method to collect log messages from multiple threads and have them all be displayed using a window? (while the threads are running).
I am currently trying to redirect stdout (cout) in to a wxTextCtrl but failing miserably when attempting to do so over multiple threads. Any help would be appreciated.
Logging has had a few major updates recently in the wxWidgets trunk, you can read about them here. One of them is to add support for logging from threads other than the main thread.
In what way is it failing? I'm not familiar with the wxTextCtrl, but unless it has built in synchronization (ie. its thread safe) that could be a big issue. The simplest way to protect a single resource like this is via a named 'mutex'. The following example is what you can do in each thread to make sure that only one accesses this resource (the output window) at a time.
// In each thread's initialization:
HANDLE mutexHandle = CreateMutex(0,FALSE,"__my_protecting_mutex__");
// Whenever you use the debug output:
WaitForSingleObject(mutexHandle, /* Timeout if you like. */ 0xFFFFFFFF );
// Do our printing here.
ReleaseMutex(mutexHandle);
// In each thread's cleanup:
CloseHandle(mutexHandle);
So this basically guarantees that only one thread can be in between the wait and the release. Now if your issue is actually routing to the wxTextCtrl, I would need some more details.
Edit: I just realized that what I posted is Windows specific, and maybe you aren't on windows! If you aren't I don't have experience with other platform's synchronization methods, but boost has some generic libraries which are not platform specific.
I have this tool in which a single log-like file is written to by several processes.
What I want to achieve is to have the file truncated when it is first opened, and then have all writes done at the end by the several processes that have it open.
All writes are systematically flushed and mutex-protected so that I don't get jumbled output.
First, a process creates the file, then starts a sequence of other processes, one at a time, that then open the file and write to it (the master sometimes chimes in with additional content; the slave process may or may not be open and writing something).
I'd like, as much as possible, not to use more IPC that what already exists (all I'm doing now is writing to a popen-created pipe). I have no access to external libraries other that the CRT and Win32 API, and I would like not to start writing serialization code.
Here is some code that shows where I've gone:
// open the file. Truncate it if we're the 'master', append to it if we're a 'slave'
std::ofstream blah(filename, ios::out | (isClient ? ios:app : 0));
// do stuff...
// write stuff
myMutex.acquire();
blah << "stuff to write" << std::flush;
myMutex.release();
Well, this does not work: although the output of the slave process is ordered as expected, what the master writes is either bunched together or at the wrong place, when it exists at all.
I have two questions: is the flag combination given to the ofstream's constructor the right one ? Am I going the right way anyway ?
If you'll be writing a lot of data to the log from multiple threads, you'll need to rethink the design, since all threads will block on trying to acquire the mutex, and in general you don't want your threads blocked from doing work so they can log. In that case, you'd want to write your worker thread to log entries to queue (which just requires moving stuff around in memory), and have a dedicated thread to pull entries off the queue and write them to the output. That way your worker threads are blocked for as short a time as possible.
You can do even better than this by using async I/O, but that gets a bit more tricky.
As suggested by reinier, the problem was not in the way I use the files but in the way the programs behave.
The fstreams do just fine.
What I missed out is the synchronization between the master and the slave (the former was assuming a particular operation was synchronous where it was not).
edit: Oh well, there still was a problem with the open flags. The process that opened the file with ios::out did not move the file pointer as needed (erasing text other processes were writing), and using seekp() completely screwed the output when writing to cout as another part of the code uses cerr.
My final solution is to keep the mutex and the flush, and, for the master process, open the file in ios::out mode (to create or truncate the file), close it and reopen it using ios::app.
I made a 'lil log system that has it's own process and will handle the writing process, the idea is quite simeple. The proccesses that uses the logs just send them to a pending queue which the log process will try to write to a file. It's like batch procesing in any realtime rendering app. This way you'll grt rid of too much open/close file operations. If I can I'll add the sample code.
How do you create that mutex?
For this to work this needs to be a named mutex so that both processes actually lock on the same thing.
You can check that your mutex is actually working correctly with a small piece of code that lock it in one process and another process which tries to acquire it.
I suggest blocking such that the text is completely written to the file before releasing the mutex. I've had instances where the text from one task is interrupted by text from a higher priority thread; doesn't look very pretty.
Also, put the format into Comma Separated format, or some format that can be easily loaded into a spreadsheet. Include thread ID and timestamp. The interlacing of the text lines shows how the threads are interacting. The ID parameter allows you to sort by thread. Timestamps can be used to show sequential access as well as duration. Writing in a spreadsheet friendly format will allow you to analyze the log file with an external tool without writing any conversion utilities. This has helped me greatly.
One option is to use ACE::logging. It has an efficient implementation of concurrent logging.
I am wrapping existing C++ code from a BSD project in our own custom wrapper and I want to integrate it to our code with as few changes as possible. This code uses fprintf to print to stderr in order to log / report errors.
I want to redirect this to an alternative place within the same process. On Unix I have done this with a socketpair and a thread: one end of the socket is where I send stderr (via a call to dup2) and the other end is monitored in a thread, where I can then process the output.
This does not work on Windows though because a socket is not the same as a file handle.
All documents I have found on the web show how to redirect output from a child process, which is not what I want. How can I redirect stderr within the same process getting a callback of some sort when output is written? (and before you say so, I've tried SetStdHandle but cannot find any way to make this work)...
You can use a similar technique on Windows, you just need to use different words for the same concepts. :) This article: http://msdn.microsoft.com/en-us/library/ms682499.aspx uses a win32 pipe to handle I/O from another process, you just have to do the same thing with threads within the same process. Of course, in your case all output to stderr from anywhere in the process will be redirected to your consumer.
Actually, other pieces of the puzzle you may need are _fdopen and _open_osfhandle. In fact, here's a related example from some code I released years ago:
DWORD CALLBACK DoDebugThread(void *)
{
AllocConsole();
SetConsoleTitle("Copilot Debugger");
// The following is a really disgusting hack to make stdin and stdout attach
// to the newly created console using the MSVC++ libraries. I hope other
// operating systems don't need this kind of kludge.. :)
stdout->_file = _open_osfhandle((long)GetStdHandle(STD_OUTPUT_HANDLE), _O_TEXT);
stdin->_file = _open_osfhandle((long)GetStdHandle(STD_INPUT_HANDLE), _O_TEXT);
debug();
stdout->_file = -1;
stdin->_file = -1;
FreeConsole();
CPU_run();
return 0;
}
In this case, the main process was a GUI process which doesn't start with stdio handles at all. It opens a console, then shoves the right handles into stdout and stdin so the debug() function (which was designed as a stdio interactive function) can interact with the newly created console. You should be able to open some pipes and do the same sort of thing to redirect stderr.
You have to remember that what MSVCRT calls "OS handles" are not Win32 handles, but another layer of handles added just to confuse you. MSVCRT tries to emulate the Unix handle numbers where stdin = 0, stdout = 1, stderr = 2 and so on. Win32 handles are numbered differently and their values always happen to be a multiple of 4. Opening the pipe and getting all the handles configured properly will require getting your hands messy. Using the MSVCRT source code and a debugger is probably a requirement.
You mention that you don't want to use a named pipe for internal use; it's probably worth poining out that the documentation for CreatePipe() states, "Anonymous pipes are implemented using a named pipe with a unique name. Therefore, you can often pass a handle to an anonymous pipe to a function that requires a handle to a named pipe." So, I suggest that you just write a function that creates a similar pipe with the correct settings for async reading. I tend to use a GUID as a string (generated using CoCreateGUID() and StringFromIID()) to give me a unique name and then create the server and client ends of the named pipe with the correct settings for overlapped I/O (more details on this, and code, here: http://www.lenholgate.com/blog/2008/02/process-management-using-jobs-on-windows.html).
Once I have that I wire up some code that I have to read a file using overlapped I/O with an I/O Completion Port and, well, then I just get async notifications of the data as it arrives... However, I've got a fair amount of well tested library code in there that makes it all happen...
It's probably possible to set up the named pipe and then just do an overlapped read with an event in your OVERLAPPED structure and check the event to see if data was available... I don't have any code available that does that though.