I am writing a scientific computation code in C++. There are outputs that I want to write in a console and outputs that I write into a file. However, when debugging after implementing a new feature, it is useful to print out much more information than usual. So far I was just sending more information to std::cout/clog and commented these lines out when not needed.
What I want is like std::clog, which would go into a file when needed, or not do anything at all, when not needed. It is ok, if I need to recompile the code to switch between the two regimes. It is important that nothing happens when not needed, because for a real large calculation the log file would be enormous (or the the console full of rubbish) and all the writing would slow the calculation down.
I am looking for the smallest possible implementation, ideally using only standard libraries for portability.
The obvious solution is to have a global variable, redirect clog to a file and then use an if statement.
bool DEBUG = true;
std::ofstream out("logfile");
std::clog.rdbuf(out.rdbuf());
...
if (DEBUG) std::clog << "my message" << std::endl;
...
Is there a more elegant way of doing this?
Edit:
I would like to avoid using non-standard libraries and preprocessor macros (program is spread across many files and also a bad programming habit in general). One way I could imagine this working, but I don't know how to do it, is to create a globally accessible object that would be able to accept messages using << and would save them to a file. Then I could just comment out the line inside this object class that saves it to a file. However, I don't know how much performance impact may result from passing messages to such a disfunctional object.
You may use any external logging library for C/C++.
Or create your own small implementation with only utilities what you need.
A traditional logging mechanism is build on macros (or inline functions) and looks like:
#define LOG_MESSAGE(msg) \
{
#ifdef DEBUG
// your debug logging
#else
// your release logging, may be leaved empty
#endif // DEBUG
}
It's also useful to add different logging levels: Error, Warning, Info, etc.
Related
I am building somewhat larger c++ code bases than I'm used to. I have a need for both good logging and debugging, at least to the console, and also speed.
Generally, I like to do something like this
// Some header file
bool DEBUG = true;
And then in some other file
if (DEBUG) cout << "Some debugging information" << endl;
The issue with this (among others) is that the branching lowers the speed of the final executable. In order to fix this, I'd have to go into the files at the end and remove all these, and then I couldn't use them again later without saving them to some other file and then putting them back in.
What is the most efficient solution to this quandry? Python decorators provide a nice approach that I'm not certain exists in CPP.
Thanks!
The classic way is to make that DEBUG not a variable, but a preprocessor macro. Then you can have two builds: one with the macro defined to 1, the other with it defined to 0 (or not defined at all, depending on how you plan to use it). Then you can either do #ifdef to completely remove the debug code from being seen by the compiler, or just put it into a regular if, the optimizer will take care of removing the branch with the constant conditional.
I'd like to have the possibility to increase the verbosity for debug purposes of my program. Of course I can do that using a switch/flag during runtime. But that can be very inefficient, due to all the 'if' statements I should add to my code.
So, I'd like to add a flag to be used during compilation in order to include optional, usually slow debug operations in my code, without affecting the performance/size of my program when not needed. here's an example:
/* code */
#ifdef _DEBUG_
/* do debug operations here
#endif
so, compiling with -D_DEBUG_ should do the trick. without it, that part won't be included in my program.
Another option (at least for i/o operations) would be to define at least an i/o function, like
#ifdef _DEBUG_
#define LOG(x) std::clog << x << std::endl;
#else
#define LOG(x)
#endif
However, I strongly suspect this probably isn't the cleanest way to do that. So, what would you do instead?
I prefer to use #ifdef with real functions so that the function has an empty body if _DEBUG_ is not defined:
void log(std::string x)
{
#ifdef _DEBUG_
std::cout << x << std::endl;
#endif
}
There are three big reasons for this preference:
When _DEBUG_ is not defined, the function definition is empty and any modern compiler will completely optimize out any call to that function (the definition should be visible inside that translation unit, of course).
The #ifdef guard only has to be applied to a small localized area of code, rather than every time you call log.
You do not need to use lots of macros, avoiding pollution of your code.
You can use macros to change implementation of the function (Like in sftrabbit's solution). That way, no empty places will be left in your code, and the compiler will optimize the "empty" calls away.
You can also use two distinct files for the debug and release implementation, and let your IDE/build script choose the appropriate one; this involves no #defines at all. Just remember the DRY rule and make the clean code reusable in debug scenario.
I would say that his actually is very dependent on the actual problem you are facing. Some problems will benefit more of the second solution, whilst the simple code might be better with simple defines.
Both snippets that you describe are correct ways of using conditional compilation to enable or disable the debugging through a compile-time switch. However, your assertion that checking the debug flags at runtime "can be very inefficient, due to all the 'if' statements I should add to my code" is mostly incorrect: in most practical cases a runtime check does not influence the speed of your program in a detectable way, so if keeping the runtime flag offers you potential advantages (e.g. turning the debugging on to diagnose a problem in production without recompiling) you should go for a run-time flag instead.
For the additional checks, I would rely on the assert (see the assert.h) which does exactly what you need: check when you compile in debug, no check when compiled for the release.
For the verbosity, a more C++ version of what you propose would use a simple Logger class with a boolean as template parameter. But the macro is fine as well if kept within the Logger class.
For commercial software, having SOME debug output that is available at runtime on customer sites is usually a valuable thing to have. I'm not saying everything has to be compiled into the final binary, but it's not at all unusual that customers do things to your code that you don't expect [or that causes the code to behave in ways that you don't expect]. Being able to tell the customer "Well, if you run myprog -v 2 -l logfile.txt and do you usual thing, then email me logfile.txt" is a very, very useful thing to have.
As long as the "if-statement to decide if we log or not" is not in the deepest, darkest jungle in peru, eh, I mean in the deepest nesting levels of your tight, performance critical loop, then it's rarely a problem to leave it in.
So, I personally tend to go for the "always there, not always enabled" approach. THat's not to say that I don't find myself adding some extra logging in the middle of my tight loops sometimes - only to remove it later on when the bug is fixed.
You can avoid the function-like macro when doing conditional compilation. Just define a regular or template function to do the logging and call it inside the:
#ifdef _DEBUG_
/* ... */
#endif
part of the code.
At least in the *Nix universe, the default define for this kind of thing is NDEBUG (read no-debug). If it is defined, your code should skip the debug code. I.e. you would do something like this:
#ifdef NDEBUG
inline void log(...) {}
#else
inline void log(...) { .... }
#endif
An example piece of code I use in my projects. This way, you can use variable argument list and if DEBUG flag is not set, related code is cleared out:
#ifdef DEBUG
#define PR_DEBUG(fmt, ...) \
PR_DEBUG(fmt, ...) printf("[DBG] %s: " fmt, __func__, ## __VA_ARGS__)
#else
#define PR_DEBUG(fmt, ...)
#endif
Usage:
#define DEBUG
<..>
ret = do_smth();
PR_DEBUG("some kind of code returned %d", ret);
Output:
[DBG] some_func: some kind of code returned 0
of course, printf() may be replaced by any output function you use. Furthermore, it can be easily modified so additional information, as for example time stamp, is automatically appended.
For me it depends from application to application.
I've had applications where I wanted to always log (for example, we had an application where in case of errors, clients would take all the logs of the application and send them to us for diagnostics). In such a case, the logging API should probably be based on functions (i.e. not macros) and always defined.
In cases when logging is not always necessary or you need to be able to completely disable it for performance/other reasons, you can define logging macros.
In that case I prefer a single-line macro like this:
#ifdef NDEBUG
#define LOGSTREAM /##/
#else
#define LOGSTREAM std::clog
// or
// #define LOGSTREAM std::ofstream("output.log", std::ios::out|std::ios::app)
#endif
client code:
LOG << "Initializing chipmunk feeding module ...\n";
//...
LOG << "Shutting down chipmunk feeding module ...\n";
It's just like any other feature.
My assumptions:
No global variables
System designed to interfaces
For whatever you want verbose output, create two implementations, one quiet, one verbose.
At application initialisation, choose the implementation you want.
It could be a logger, or a widget, or a memory manager, for example.
Obviously you don't want to duplicate code, so extract the minimum variation you want. If you know what the strategy pattern is, or the decorator pattern, these are the right direction. Follow the open closed principle.
Some investigation on existing C/C++ logging solutions turned out that Pantheios might be the best in my case, that is lowest overhead if logging is disabled.
All the loggers seem to support a kind of a print log message. However, in my case I have a function call that should be avoided if logging is disabled (since it's quite expensive).
At the moment I use a very simple logging setup like
#ifdef DEBUG_L1
cout << "msg 1" << endl // log level 1
#ifdef DEBUG_L2
printBuffer() // log level 2
#endif
#endif
It serves my needs (for now) since i pay zero overhead if logging is disabled. However the code quickly looks ugly and it is not very flexible.
This should be realized with a C++ logger. As said, the function body of printBuffer() is quite expensive. It would be good if calling it could be avoided if logging is turned off.
Is it possible to declare a whole function call only to be carried out when above a certain log level? Or do i still need the preprocessor in this case?
Edit:
Thanks #BobTFish. I was actually thinking about using the kind of setup you are describing. I am wondering how flexible one can realize this kind of thing. Typically i log a set of strings and values (int, float, and pointers). In the style
cout << "name1=" << int << " name2=" << (void*)(ptr) << endl;
Now, I really do not like switching to a printf like syntax at this point. How would the macro approach deal with this (since it's templated it with just one class parameter)?
What me come in mind are c++ specific template based logging frameworks like easylogging or spdlog. In spdlog for example you can create custom log targets by implementing a sink interface. Another (may be the better) option is to use it's log level feature.
Here an example (copied from spdlog manual):
//
// Runtime log levels
//
spd::set_level(spd::level::info); //Set global log level to info
console->debug("This message shold not be displayed!");
console->set_level(spd::level::debug); // Set specific logger's log level
console->debug("Now it should..");
By implementing the << operator for an own custom class, you can control what data is being dumped to the log. With logger->should_log() you can test if specified log level is enabled.
I think you can use Google logging library from here
The typical use of glog with condition
#include <glog/logging.h>
{
// Initialize logging. There are multiple options, so read the documentation
google::InitGoogleLogging();
void* p;
LOG_IF(INFO, p == nullptr) << "We have nullptr. Bomb detected!";
// Don't forget to shut that down
google::ShutdownGoogleLogging();
}
If you concerned with performance and runtime overhead, take a look at zf_log library. Things you could like about it:
It evaluates logging arguments and invokes actual logging function only when necessary (when log level allows that).
It has run-time log level AND compile-time log level. LOG() statements below compile-time log level are compiled out and have no run-time overhead.
Run-time log level could be changed in run-time, but when message log level is below run-time log level, arguments will not be evaluated and actual log function will not be called. The only thing that will be evaluated is if (msg_log_level >= runtime_log_level).
It has extremely small call site (amount of code generated per each LOG() line), 3x-20x times less than other libraries.
It doesn't slow down compilation of your sources that include its headers (unlike some header-only libraries).
Having used gprof and callgrind many times, I have reached the (obvious) conclusion that I cannot use them efficiently when dealing with large (as in a CAD program that loads a whole car) programs. I was thinking that maybe, I could use some C/C++ MACRO magic and somehow build a simple (but nice) logging mechanism. For example, one can call a function using the following macro:
#define CALL_FUN(fun_name, ...) \
fun_name (__VA_ARGS__);
We could add some clocking/timing stuff before and after the function call, so that every function called with CALL_FUN gets timed, e.g
#define CALL_FUN(fun_name, ...) \
time_t(&t0); \
fun_name (__VA_ARGS__); \
time_t(&t1);
The variables t0, t1 could be found in a global logging object. That logging object can also hold the calling graph for each function called through CALL_FUN. Afterwards, that object can be written in a (specifically formatted) file, and be parsed from some other program.
So here comes my (first) question: Do you find this approach tractable ? If yes, how can it be enhanced, and if not, can you propose a better way to measure time and log callgraphs ?
A collegue proposed another approach to deal with this problem, which is annotating with a specific comment each function (that we care to log). Then, during the make process, a special preprocessor must be run, parse each source file, add logging logic for each function we care to log, create a new source file with the newly added (parsing) code, and build that code instead. I guess that reading CALL_FUN... macros (my proposal) all over the place is not the best approach, and his approach would solve this problem. So what is your opinion about this approach?
PS: I am not well versed in the pitfalls of C/C++ MACROs, so if this can be developed using another approach, please say it so.
Thank you.
Well you could do some C++ magic to embed a logging object. something like
class CDebug
{
CDebug() { ... log somehow ... }
~CDebug() { ... log somehow ... }
};
in your functions then you simply write
void foo()
{
CDebug dbg;
...
you could add some debug info
dbg.heythishappened()
...
} // not dtor is called or if function is interrupted called from elsewhere.
I am bit late, but here is what I am doing for this:
On Windows there is a /Gh compiler switch which makes the compiler to insert a hidden _penter function at the start of each function. There is also a switch for getting a _pexit call at the end of each function.
You can utilizes this to get callbacks on each function call. Here is an article with more details and sample source code:
http://www.johnpanzer.com/aci_cuj/index.html
I am using this approach in my custom logging system for storing the last few thousand function calls in a ring buffer. This turned out to be useful for crash debugging (in combination with MiniDumps).
Some notes on this:
The performance impact very much depends on your callback code. You need to keep it as simple as possible.
You just need to store the function address and module base address in the log file. You can then later use the Debug Interface Access SDK to get the function name from the address (via the PDB file).
All this works suprisingly well for me.
Many nice industrial libraries have functions' declarations and definitions wrapped into void macros, just in case. If your code is already like that -- go ahead and debug your performance problems with some simple asynchronous trace logger. If no -- post-insertion of such macros can be an unacceptably time-consuming.
I can understand the pain of running an 1Mx1M matrix solver under valgrind, so I would suggest starting with so called "Monte Carlo profiling method" -- start the process and in parallel run pstack repeatedly, say each second. As a result you will have N stack dumps (N can be quite significant). Then, the mathematical approach would be to count relative frequencies of each stack and make a conclusion about the ones most frequent. In practice you either immediately see the bottleneck or, if no, you switch to bisection, gprof, and finally to valgrind's toolset.
Let me assume the reason you are doing this is you want to locate any performance problems (bottlenecks) so you can fix them to get higher performance.
As opposed to measuring speed or getting coverage info.
It seems you're thinking the way to do this is to log the history of function calls and measure how long each call takes.
There's a different approach.
It's based on the idea that mainly the program walks a big call tree.
If time is being wasted it is because the call tree is more bushy than necessary,
and during the time that's being wasted, the code that's doing the wasting is visible on the stack.
It can be terminal instructions, but more likely function calls, at almost any level of the stack.
Simply pausing the program under a debugger a few times will eventually display it.
Anything you see it doing, on more than one stack sample, if you can improve it, will speed up the program.
It works whether or not the time is being spent in CPU, I/O or anything else that consumes wall clock time.
What it doesn't show you is tons of stuff you don't need to know.
The only way it can not show you bottlenecks is if they are very small,
in which case the code is pretty near optimal.
Here's more of an explanation.
Although I think it will be hard to do anything better than gprof, you can create a special class LOG for instance and instantiate it in the beginning of each function you want to log.
class LOG {
LOG(const char* ...) {
// log time_t of the beginning of the call
}
~LOG(const char* ...) {
// calculate the total time spent,
//by difference between current time and that saved in the constructor
}
};
void somefunction() {
LOG log(__FUNCTION__, __FILE__, ...);
.. do other things
}
Now you can integrate this approach with the preprocessing one you mentioned. Just add something like this in the beginning of each function you want to log:
// ### LOG
and then you replace the string automatically in debug builds (shoudn't be hard).
May be you should use a profiler. AQTime is a relatively good one for Visual Studio. (If you have VS2010 Ultimate, you already have a profiler.)
I have the following code snippet:
#ifdef DO_LOG
#define log(p) record(p)
#else
#define log(p)
#endif
void record(char *data){
.....
.....
}
Now if I call log("hello world") in my code and DO_LOG isn't defined, will the line be compiled, in other words will it eat up the memory for the string "hello world"?
P.S. There are a lot of record calls in the program and it is memory sensitive, so is there any other way to conditionally compile so that it only depends on the #define DO_LOG?
This should be trivial to verify for yourself by inspecting the resulting binary.
I would say "no", since the expression totally goes away, the compiler will never see the string (it's removed by the preprocessor's macro expansion).
Since the preprocessor runs before the compiler, the line will not even exist when the compiler runs. So the answer is no, it does not use any memory at all.
No, it will not be in the binary. It will not even be compiled - the preprocessor will expand it into an empty string prior to the compilation, so the compiler will not even see it.
No. The preprocessor is executed prior to compilation, and so the code will never even be seen. I would like to add, though, that if you are interested in adding logging to your C++ application, you might want to use the Log4Cxx library. It uses similar macros which you can completely elide from your application, but when logging is enabled, it supports several different levels of logging (based on importance/severity) as well as multiple different "appenders" to which to send logging output (e.g. syslog, console, files, network I/O, etc.).
The full API documentation may be found at Log4Cxx API docs. Also, if you have any Java developers on board who have used Log4J, they should feel right at home with Log4Cxx (and convince you to use it).