I'm making a multiple-thread program and I want to monitor each threads process using Pantheios.
Since each thread has lots of things going on, I need to separate the logs for each threads. (if not, the log file size would be monstrous)
Using the following function, I tried to separate the log files for each threads
void threadClass::prepareLog()
{
static int counter = 1;
CString strPath = _T("");
strPath.Format(_T("log%d.log"),counter++);
pantheios_be_file_setFilePath(strPath, PANTHEIOS_BEID_LOCAL);
}
But as the result, all the threads logs would only be recorded to the last thread's log file.
I am still searching, but I wonder if Pantheios itself doesn't support multiple log files generation.
By the way, I am using Unicode character set.
If there is anything unclear or if there is any question, feel free to ask.
Thank you for your attention.
Related
Being a beginner in C++, I found myself facing a problem when attempting to limit log file size using the ezlogger library: http://axter.com/ezlogger/
My take at it was to:
1) check log file size every n seconds
2) if size is too big, start logging to a second file (clearing it beforehand) Then switch between the files every n seconds.
I did 1. And I tackled 2 by changing the symlink used by the logging library as logging output file location (the app is running on Linux). However, it seems that the library retains a reference to the original file and never starts logging to the new file after changing the link.
The reason I decided to go this way was because I didn't want to touch the library. For an experienced programmer it would probably make more sense to somehow modify the library to enable switching log files. But with all the static variables and methods and hpp files containing actual code, I couldn't make sense of it and didn't know where to start.
So I guess I'm looking for opinions on my current approach, help with getting it to work and/or advice on how to do it differently/better.
Thanks.
Edit: I'm working on an existing older project which already uses ezlogger so I'd like to avoid using a different library if possible.
Either use logrotate (if you use unix like system) as it was suggested or modify your logging library. Those static variable you mention appear to be located in get_log_stream(). The modification would require checking on each get_log_stream call, the size of the current logging file. If the size exceeds some number of bytes then reopen stream. I don't see this logging library to be thread safe, so it probably isn't so you don't have to worry about it. But if your application is multithreaded then make a note of it.
The modification of get_log_stream would look as follows (its pseudocode):
// ...
if (logfile_is_open) {
if (logfile.tellp() > 1024*1024*10 /*10MB*/) {
logfile.close();
logfile.clear(); //clears flags
// TODO: update FileName accordingly, ie. add a count to it.
// TODO: remove older log files, etc.
logfile.open(FileName.c_str(), std::ios_base::out);
}
}
// Below is old code.
if (logfile_is_open) return logfile;
return std::cout;
The company I'm working with has a program written in ye olde vb6, which is updated pretty frequently, and most clients run the executable from a mapped network drive. This actually has surprisingly few issues, the biggest of which is automatic updates. Currently the updater program (written in c++) renames the existing exe, then downloads and places the new version into the old version's place. This generally works fine, but in some environments it simply fails.
The solution is running this command from microsoft:
for /f "skip=4 tokens=1" %a in ('net files') do net files %a /close
This command closes all network files that are shared (well... most) and then the updater can replace the exe.
In C++ I can use the System(""); function to run that command, or I could redirect the output of net files, and iterate through the results looking for the particular file in question and run net file /close command to close them. But it would be much much nicer if there were winapi functions that have similar capabilities for better reliability and future safety.
Is there any way for me to programmatically find all network shared files and close relevant ones?
You can programmatically do what net file /close does. Just include lmshare.h and link to Netapi32.dll. You have two functions to use: NetFileEnum to enumerate all open network files (on a given computer) and NetFileClose to close them.
Quick (it assumes program is running on same server and there are not too many open connections, see last paragraph) and dirty (no error checking) example:
FILE_INFO_2* pFiles = NULL;
DWORD nRead = 0, nTotal = 0;
NetFileEnum(
NULL, // servername, NULL means localhost
"c:\\directory\\path", // basepath, directory where VB6 program is
NULL, // username, searches for all users
2, // level, we just need resource ID
(LPBYTE*)&pFiles, // bufptr, need to use a double pointer to get the buffer
MAX_PREFERRED_LENGTH, // prefmaxlen, collect as much as possible
&nRead, // entriesread, number of entries stored in pFiles
&nTotal, // totalentries, ignore this
NULL //resume_handle, ignore this
);
for (int i=0; i < nRead; ++i)
NetFileClose(NULL, pFiles[i].fi2_id);
NetApiBufferFree(pFiles);
Refer to MSDN for details about NetFileEnum and NetFileClose. Note that NetFileEnum may return ERROR_MORE_DATA if more data is available.
I am trying to understand how basic I/O with files is handled in c++ or c. My aim is to read file line by line and send the lines across to a remote server. If the line is sent, I want to delete it from the file.
One way I tried was that I kept a count of the lines read and called an system() system call to delete the 'count' number of lines. I used the bash command: sed -i -e 1,'count'd filename.
After that I continued reading the file and surprisingly it worked as planned.
I have two questions:
Is this way reliable?
And why does this work at all, when while
reading the file I deleted a part of it and yet it works? What if I
did a seek to a previous position, what then?
Best,Digvijay
PS:
I would be glad if somebody could suggest a better way.
Also here is the code for the program I wrote:
#include<iostream>
#include<fstream>
#include<string>
#include<sstream>
#include<cstdlib>
int main(){
std::ifstream f;
std::string line;
std::stringstream ss;
int i=0;
f.open("in.txt");
if(f.is_open()){
while(getline(f,line)){
std::cout<<line<<std::endl;
i++;
if(i==2)break;
}
ss<<"sed -i -e 1,"<<i<<"d in.txt";
system(ss.str().c_str());
while(getline(f,line)){
std::cout<<line<<std::endl;
}
}
return 0;
}
Edit:
Firstly thanks for taking the time to write answers. But here is some extra information which I missed out on earlier. The files I am dealing with are log files. So they are constantly being appended with information from devices. The reason why I want to avoid creating a copy is, because the log file themselves are very big(at times) and plus this would help to keep the log file short. Since they would be divided into parts and archived on the server.
Solution
I have found the way to deal with the problem. Apparently Thomas is right, that sed does create a new file. So the old file remains as is. Using this, I can read n lines, call the system function, close the file pointer and open it again. I do this on small chunks of the log, repeatedly until it becomes small and hence efficient to deal with. The server while archives the logs in 1gb files.
However I have a new question, due to memory constraint, I need to know if it is possible to split a log file into two efficiently. (Which possibly would be another question on SO)
Most modern file systems don't support deleting lines at the beginning of the file, so doing so would be very inefficient.
The normal solution to your actual problem is to stop writing to your log file when it reaches some size, then start writing to a new file. The code that copies the files can delete a whole file once it has been written (this is an efficient operation).
sed writes a new version of the file, while the program keeps reading the same version that it opened. This is the usual behavior of Unix and Linux when a program writes a file that another program has open.
You can see this for yourself with this small C program:
#include <stdlib.h>
#include <stdio.h>
int main(void) {
FILE *f = fopen("in.txt", "r");
while (1) {
rewind(f);
int lines = 0;
int c;
while ((c = getc(f)) != EOF)
if (c == '\n')
++lines;
printf("Number of lines in file: %d\n", lines);
}
return 0;
}
Run that program in one window, and then use sed in another window to edit the file. The number of lines printed by the program will stay the same, even if the file on disk has been edited, and this is because Unix keeps the old, open version, even if it is no longer accessible to other programs.
As to your first question, how reliable your solution is, as far as I can see it should be reliable, except with the usual caveats about the system crashing or running out of memory in the middle of an update, someone else accessing the file, and of course all the problems with the system call. It is not very efficient, though, and for large data sets you might want to do it differently.
sujin's comment about using a temporary file for the lines you want to keep seems reasonable. It would be both faster and safer. Keep the original file, so if the system crashes you'll still have your data, and wait until you have finished to rename the old file to "in.txt.bak", and then rename your temporary file to "in.txt".
First off, avoid use of system calls as much as you can (if possible, don't use it at all) as they create race conditions and other problems which drastically (and often) detrimentally affect the outcome of your program. This especially true if access to files are involved.
Given your problem, there are a number of ways to do this, each with its own caveats.
I'll cover three possible solutions:
1) If the file is small enough:
you can read in the entire thing in a data structure (vector, list, deque, etc.)
delete the original file
determine how many lines to read (and send off via server protocol)
then write the remaining lines as the name of the original file.
If you intend to parallelize your program later on, this may be a better solution, provided that the file is small. Note: small is a relative term, but is generally limited by how much memory you have available.
2) If the file is quite large or you're limited by memory constraints, you will have to get creative by using buffers. Once you've read a line and successfully sent it off via your program, you determine where the file pointer is and copy the remaining information until the end of the current file as a new file. Once done, close and delete the old file, then close and rename the new file the same name as the old file.
3) If your solution doesn't have to be in C++, you can use shell-scripting or (controversially) another language to get the job done.
1) No, it's not reliable.
2) The C++ runtime library reads your file in blocks (internally) which are then parceled out to your (higher level) input requests until the block(s) is(are) exhausted, forcing it to (internally) read more blocks from disk. Since one or more physical blocks are read in before you make any call to sed, it/they cannot be altered if sed happens to change that first part of the file.
To see your code fail, you would need to make the input file big enough that there are remaining blocks of the file that have not been read in (internally by the runtime library) before you call sed. By "fail" I mean your program would not see all the characters that were originally in the file before sed clobbered some lines.
As the other guys said, you have to make another file with the records you need after read the original file and then delete it. But in this application perhaps you will see more useful a fifo than a file. If you are on a *NIX platform check up about the makefifo statement from the console.
It is like a file with the singularity that after read a line it gets deleted.
Guys, I am a beginner in threading and logging.
Btw, I am not a native English speaker, so pardon me if there is any mistake in my English.
I have created a multiple-thread software, where each thread uses logging module like the following:
Each thread uses different log files, so I believe that the chances of data conflicts occurred are 0.
__inline void print_logW(int _level,const wchar_t *domain,const wchar_t *msg)
{
wchar_t mess[200] = _T("");
if(_level<=traceLevel)
{
__time64_t timer;
struct tm t_st;
_time64(&timer);
localtime_s(&t_st,&timer);
if (domain == NULL)
{
domain = _T("");
}
if (msg != NULL)
{
if (showTimeStampFlag == true)
{
swprintf_s(mess,200,_T("%s : %ld"),msg,GetTickCount());
}
else
{
wcscpy_s(mess,200,msg);
}
}
if(oldTime.tm_year != t_st.tm_year || oldTime.tm_mon != t_st.tm_mon || oldTime.tm_mday != t_st.tm_mday)
{
oldTime = t_st;
print_log_preparebyDateW();
}
FILE* fp;
errno_t err = _wfopen_s(&fp, this->m_pathW, _T("at+, ccs=UTF-8"));
if (err != 0)
{
// error
return;
}
fwprintf_s(fp, m_logFormatW,
_level,
1900 + t_st.tm_year, t_st.tm_mon + 1, t_st.tm_mday,
t_st.tm_hour, t_st.tm_min, t_st.tm_sec,
domain, mess
);
fflush(fp);
fclose(fp);
}
}
When I see the log of the software that I made, I found a problem where sometimes the thread process becomes so slow (a process (such as getting pointer of an image) that usually only take 16 ms max, would take 0.2 seconds or more to finish). I am still investigating the cause of this problem, but at first, I would like to know whether the logging module is already thread safe or not.
By the way, for the parameters,
"_level" is the logging level to print or unprint the details of the process
I use "domain" to show the class where the logging is performed
"msg" is the content of the log (e.g. "process 1 started")
And as for the m_logFormatW,
m_logFormatW = _T("[%.2d][%.4d-%.2d-%.2dT%.2d:%.2d:%.2d][%s] %s\n");
If there is any question or anything unclear, feel free to ask.
As long as you are linking to the multi-threaded runtime libraries and oldTime is not a global or static variable your log function will be thread safe. If oldTime is a global or static variable you will need to serialize access to it when you access or modify it otherwise you risk a race condition. The only other thing that may not be thread safe is print_log_preparebyDateW but it's hard to say since you haven't included the code for it. As long as oldTime is not global or static and all the runtime library functions that you use are marked as thread safe or are part of a library marked as thread safe in the MSDN you'll be OK.
The only other problem I can see is when you open the file. If the file is already open and another thread attempts to log information the open call will fail causing the information to be lost. This is because _wfopen_s opens the file without any sharing modes. You can fix this by using std::mutex and locking it while the file is open and unlocking it after the file is closed.
One possible reason your worker threads are taking longer to execute than expected is that opening the log file, writing the information, flushing the file and closing it can take a bit of extra time. This can happen any time file I/O occurs even when caching is involved. Usually you can reduce the time by opening the log file once and then closing it when your application terminates.
Another possible solution to reduce the time it takes your worker threads to execute is to use a pipe. In this scenario you write the log text to a pipe and have an additional thread that reads from the pipe and writes to the log file. This will eliminate any disk I/O that may occur when your worker threads log information. You may encounter some instances where the logging takes a bit of extra time if the pipe is full but it won't happen as often.
Your approach is good for single threaded application and will not work in multithreaded environment as you are not serializing the log message requests.
Its better you look into some well written logger class in opensource such as
a) AsynchronousAndSynchronouslogger - http://www.codeproject.com/Articles/288827/g2log-An-efficient-asynchronous-logger-using-Cplus
b) Simplethreadsafe - http://cpplogging.codeplex.com/
c) Log4Cpp - http://log4cpp.sourceforge.net/
My native C++ Win32 program spawns a worker process and needs to pass a huge configuration string to it. Currently it just passes the string as a command line to CreateProcess(). The problem is the string is getting longer and now it doesn't fit into the 32K characters limitation imposed by Windows.
Of course I could do something like complicating the worker process start - I use the RPC server in it anyway and I could introduce an RPC request for passing the configuration string, but this will require a lot of changes and make the solution not so reliable. Saving the data into a file for passing is also not very elegant - the file could be left on the filesystem and become garbage.
What other simple ways are there for passing long strings to a worker process started by my program on Windows?
One possible strategy is to create a named Pipe and pass the handle ( or pipe name) to the other process. Then use normal Read\Write operations on Pipe to extract the data.
There are several good answers already, but the easiest way is to save it in a file, and pass the filename in the command line.
As well as being simple, an advantage of this approach is that the apps will be very loosely coupled (you'll potentially be able to use the child application stand-alone in other ways, rather than always having to launch it from a program that knows how to pipe data into it via a specialised interface)
If you want to be sure that the file is cleaned up after processing, mark it for deletion on the next reboot. THen if anybody forgets to clean it up, the OS will deal with it for you on the next reboot.
I would prefer Boost's message queue. It's extremely simple yet sophisticated. Here's example:
#include <boost/interprocess/ipc/message_queue.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/shared_ptr.hpp>
using namespace boost::interprocess;
// ------------------------------------------------------------------------------
// Your worker:
// ------------------------------------------------------------------------------
try {
message_queue::remove("NAME_OF_YOUR_QUEUE");
boost::shared_ptr<message_queue> mq(new message_queue(create_only, "NAME_OF_YOUR_QUEUE", 65535, 32));
char message[1024];
std::size_t size_received;
unsigned int priority;
if (mq->timed_receive(&message, sizeof(message), size_received, priority, boost::posix_time::ptime(boost::posix_time::second_clock::universal_time()) + boost::posix_time::seconds(1))) {
std::string s(message); // s now contains the message.
}
} catch (std::exception &) {
// ...
}
// ------------------------------------------------------------------------------
// And the sender:
// ------------------------------------------------------------------------------
try {
boost::shared_ptr<message_queue> mq(new message_queue(create_only, "NAME_OF_YOUR_QUEUE", 1024, 1024));
std::stringstream message;
message << "the very very very long message you wish to send over";
while (!mq.try_send(message.str().c_str(), message.str().length(), 0))
::Sleep(33);
} catch (std::exception &) {
// ...
}
Use shared memory. Pass to a worker process name of shared memory object. Another solution is to use WM_COPYDATA message.
How about reading it from stdin :) It seems to work for the Unix folks.
Guaranteed a lot easier than passing pipe names/handles around!
Here is some official code from MSDN for creating child processes with I/O pipes.
Is it a possibility to set up a named shared memory segment?
http://msdn.microsoft.com/en-us/library/aa366551(VS.85).aspx
You could use an inheritable handle to a section object. In your parent process create a section object (CreateFileMapping) and specify that its handle is to be inherited by the child process; then pass the handle value to the child process on the command line. The child process can then open the section object (OpenFileMapping). Though I would prefer a named section object as the semantics of using it are easier to understand.