I am curious to know how memory leaks are detected in a C/C++ based product under heavy load (linux platform).
I am aware of Valgrind does a great job in finding memory leak, invalid access etc.
But with valgrind, product need to operate at low load. With valgrind, you can not expect to run product at high load.
Under high load, product code execution path may be different. In that case if memory leak is there, how to catch that memory leak.
Is there any such tool available?
Followed these steps to override system functions.
1. Made my own .so file, that implements system function
2. Pre loaded the .so file in system using LD_PRELOAD
3. Thats all, after that when I execute any program it was call my custom function instead of system function.
But I had one issue, recursive calling.
When my custom function is called, internally I call system function, again it calls my function so on...
To stop it, I did not call system function directly. Instead I searched as below
static void * (*func)();
if(!func)
func = (void *(*)()) dlsym(RTLD_NEXT, "malloc");
return(func(size));
Thanks all for helping to resolve this.
Related
I'm optimizing the performance of some codes, and I find that the call of func() in third-party shared library is too slow every a few minutes. For example, if I called it for 10 times in a loop, the first call was very slow compared the other calls. A few minutes later I repeated the same operation and got the same result. I suspect that the root cause is cache missing(data and instructions). So if I can load the instructions to cache before calling this function, maybe the first call of this function will be faster than before.
I plan to using memcpy() to access the address of func() before calling it. But the problem is that func() calls other functions which also are defined in shared library, and I can't access the address of those functions.
According to this question: How to make backtrace()/backtrace_symbols() print the function names?, I know backtrace()/backtrace_symbols() can be used to print stack backtrace, but is it possible print stack backtrace of function call in third-party shared library?
BTW: I would appreciate any advice and suggestions about preloading instructions to cache.
Can I allocate a block on the heap, set its bytes to values that correspond to a function call and its parameters, then use the function call and dereference operators to execute that sequence?
So if I read you right you want to dynamically create CPU assembly instructions on the heap and execute them. A bit like self-modifying code. In theory that's possible, but in practice maybe not.
The problem is that the heap is in a data segment, and CPU's/operating systems nowadays have measures to prevent exactly this kind of behavior (it's called the NX bit, or No-eXecute bit for x86 CPUs). If a segement is marked as NX, you can't execute code from it. This was invented to stop computer virusses from using buffer overflows to place exectuable code in data/heap/stack memory and then try the calling program to execute such code.
Note that DLL's and libraries are loaded in the code segment, which of course allows code execution.
Yes. How else could Dynamic loading and Linking work? Remembering that some (most?) Operating Systems, and some (most?) Linkers are also written in C/C++. For example,
#include <dlfcn.h>
void* initializer = dlsym(sdl_library,"SDL_Init");
if (initializer == NULL) {
// report error ...
} else {
// cast initializer to its proper type and use
}
Also, I believe that a JIT (e.g. GNU lightning and others) in general performs those operations.
In windows, for example, this is now very hard to do when it was once very easy. I used to be able to take an array of bytes in C and then cast it to a function pointer type to execute it... but not any more.
Now, you can do this if you can call Global or VirtualAlloc functions and specifically ask for executable memory. On most platforms its either completely open or massively locked down. Doing this sort of thing on iOS, for example, is a massive headache and it will cause a submission fail on the app store if discovered.
here is some fantastically out of date and crusty code where i did the original thing you described:
https://code.google.com/p/fridgescript/source/browse/trunk/src/w32/Code/Platform_FSCompiledCode.cpp
using bytes from https://code.google.com/p/fsassembler
you may notice in there that i need to provide platform (windows) specific allocation functions to get some executable memory:
https://code.google.com/p/fridgescript/source/browse/trunk/src/w32/Core/Platform_FSExecutableAlloc.cpp
Yes, but you must ensure that the memory is marked executable. How you do that depends on the architecture.
I have been looking for a way to dynamically load functions into c++ for some time now, and I think I have finally figure it out. Here is the plan:
Pass the function as a string into C++ (via a socket connection, a file, or something).
Write the string into file.
Have the C++ program compile the file and execute it. If there are any errors, catch them and return it.
Have the newly executed program with the new function pass the memory location of the function to the currently running program.
Save the location of the function to a function pointer variable (the function will always have the same return type and arguments, so
this simplifies the declaration of the pointer).
Run the new function with the function pointer.
The issue is that after step 4, I do not want to keep the new program running since if I do this very often, many running programs will suck up threads. Is there some way to close the new program, but preserve the memory location where the new function is stored? I do not want it being overwritten or made available to other programs while it is still in use.
If you guys have any suggestions for the other steps as well, that would be appreciated as well. There might be other libraries that do things similar to this, and it is fine to recommend them, but this is the approach I want to look into — if not for the accomplishment of it, then for the knowledge of knowing how to do so.
Edit: I am aware of dynamically linked libraries. This is something I am largely looking into to gain a better understanding of how things work in C++.
I can't see how this can work. When you run the new program it'll be a separate process and so any addresses in its process space have no meaning in the original process.
And not just that, but the code you want to call doesn't even exist in the original process, so there's no way to call it in the original process.
As Nick says in his answer, you need either a DLL/shared library or you have to set up some form of interprocess communication so the original process can send data to the new process to be operated on by the function in question and then sent back to the original process.
How about a Dynamic Link Library?
These can be linked/unlinked/replaced at runtime.
Or, if you really want to communicated between processes, you could use a named pipe.
edit- you can also create named shared memory.
for the step 4. we can't directly pass the memory location(address) from one process to another process because the two process use the different virtual memory space. One process can't use memory in other process.
So you need create a shared memory through two processes. and copy your function to this memory, then you can close the newly process.
for shared memory, if in windows, looks Creating Named Shared Memory
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366551(v=vs.85).aspx
after that, you still create another memory space to copy function to it again.
The idea is that the normal memory allocated only has read/write properties, if execute the programmer on it, the CPU will generate the exception.
So, if in windows, you need use VirtualAlloc to allocate the memory with the flag,PAGE_EXECUTE_READWRITE (http://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx)
void* address = NULL;
address= VirtualAlloc(NULL,
sizeof(emitcode),
MEM_COMMIT|MEM_RESERVE,
PAGE_EXECUTE_READWRITE);
After copy the function to address, you can call the function in address, but need be very careful to keep the stack balance.
Dynamic library are best suited for your problem. Also forget about launching a different process, it's another problem by itself, but in addition to the post above, provided that you did the virtual alloc correctly, just call your function within the same "loadder", then you shouldn't have to worry since you will be running the same RAM size bound stack.
The real problems are:
1 - Compiling the function you want to load, offline from the main program.
2 - Extract the relevant code from the binary produced by the compiler.
3 - Load the string.
1 and 2 require deep understanding of the entire compiler suite, including compiler flag options, linker, etc ... not just the IDE's push buttons ...
If you are OK, with 1 and 2, you should know why using a std::string or anything but pure char *, is an harmfull.
I could continue the entire story but it definitely deserve it's book, since this is Hacker/Cracker way of doing things I strongly recommand to the normal user the use of dynamic library, this is why they exists.
Usually we call this code injection ...
Basically it is forbidden by any modern operating system to access something for exceution after the initial loading has been done for sake of security, so we must fall back to OS wide validated dynamic libraries.
That's said, one you have valid compiled code, if you realy want to achieve that effect you must load your function into memory then define it as executable ( clear the NX bit ) in a system specific way.
But let's be clear, your function must be code position independant and you have no help from the dynamic linker in order to resolve symbol ... that's the hard part of the job.
I have a problem with an application I'm debugging. Steady state memory usage is a few hundred megabytes. Occasionally (after several hours) it gets into a state where its memory usage soars to many gigabytes. I would like to be able to stop the program as soon as memory usage this happens.
Where control passes through my own code, I can trap excessive memory use with code like this:
bool usingTooMuchMemory()
{
PROCESS_MEMORY_COUNTERS pmc;
if (GetProcessMemoryInfo(GetCurrentProcess(), &pmc, sizeof pmc))
return pmc.WorkingSetSize > 0x80000000u; // 2GB working set
return false;
}
This doesn't help me because I need to test working set size at the right point. I really want the program to break on the first malloc or new that takes either working set or heap size over some threshold. And ideally I'd like to have this done by the CRT heap itself with minimal overhead because the library likes to allocate huge numbers of small blocks.
The suspect code is in a DLL running in a thread created by my calling code. The DLL links statically to the CRT and has no special heap management. I have source code for the DLL.
Any ideas? Am I missing something obvious?
You can set memory allocation and deallocation hooks, using _CrtSetAllocHook.
You can hook the HeapAlloc function, which malloc calls internally, by using the Detours library.
http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx
If you clear the IMAGE_FILE_LARGE_ADDRESS_AWARE flag in VS's linker options, the program's heap will be limited to 2GB in size, and should crash if attempts are made to acquire memory that would put it over that limit.
I'm currently working on an exception-based error reporting system for Windows MSVC++ (9.0) apps (i.e. exception structures & types / inheritance, call stack, error reporting & logging and so on).
My question now is: how to correctly report & log an out-of-memory error?
When this error occurs, e.g. as an bad_alloc thrown by the new op, there may be many "features" unavailable, mostly concerning further memory allocation. Normally, I'd pass the exception to the application if it has been thrown in a lib, and then using message boxes and error log files to report and log it. Another way (mostly for services) is to use the Windows Event Log.
The main problem I have is to assemble an error message.
To provide some error information, I'd like to define a static error message (may be a string literal, better an entry in a message file, then using FormatMessage) and include some run-time info such as a call stack.
The functions / methods necessary for this use either
STL (std::string, std::stringstream, std::ofstream)
CRT (swprintf_s, fwrite)
or Win32 API (StackWalk64, MessageBox, FormatMessage, ReportEvent, WriteFile)
Besides being documented on the MSDN, all of them more (Win32) or less (STL) closed source in Windows, so I don't really know how they behave under low memory problems.
Just to prove there might be problems, I wrote a trivial small app provoking a bad_alloc:
int main()
{
InitErrorReporter();
try
{
for(int i = 0; i < 0xFFFFFFFF; i++)
{
for(int j = 0; j < 0xFFFFFFFF; j++)
{
char* p = new char;
}
}
}catch(bad_alloc& e_b)
{
ReportError(e_b);
}
DeinitErrorReporter();
return 0;
}
Ran two instances w/o debugger attached (in Release config, VS 2008), but "nothing happened", i.e. no error codes from the ReportEvent or WriteFile I used internally in the error reporting. Then, launched one instance with and one w/o debugger and let them try to report their errors one after the other by using a breakpoint on the ReportError line. That worked fine for the instance with the debugger attached (correctly reported & logged the error, even using LocalAlloc w/o problems)! But taskman showed a strange behaviour, where there's a lot of memory freed before the app exits, I suppose when the exception is thrown.
Please consider there may be more than one process [edit] and more than one thread [/edit] consuming much memory, so freeing pre-allocated heap space is not a safe solution to avoid a low memory environment for the process which wants to report the error.
Thank you in advance!
"Freeing pre-allocated heap space...". This was exactly that I thought reading your question. But I think you can try it. Every process has its own virtual memory space. With another processes consuming a lot of memory, this still may work if the whole computer is working.
pre-allocate the buffer(s) you need
link statically and use _beginthreadex instead of CreateThread (otherwise, CRT functions may fail) -- OR -- implement the string concat / i2a yourself
Use MessageBox (MB_SYSTEMMODAL | MB_OK) MSDN mentions this for reporting OOM conditions (and some MS blogger described this behavior as intended: the message box will not allocate memory.)
Logging is harder, at the very least, the log file needs to be open already.
Probably best with FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH, to avoid any buffering attempts. The first one requires that writes and your memory buffers are sector aligned (i.e. you need to query GetDiskFreeSpace, align your buffer by that, and write only to "multiple of sector size" file offsets, and in blocks that are multiples of sector size. I am not sure if this is necessary, or helps, but a system-wide OOM where every allocation fails is hard to simulate.
Please consider there may be more than one process consuming much memory, so freeing pre-allocated heap space is not a safe solution to avoid a low memory environment for the process which wants to report the error.
Under Windows (and other modern operating systems), each process has its own address space (aka memory) separate from every other running process. And all of that is separate from the literal RAM in the machine. The operating system has virtualized the process address space away from the physical RAM.
This is how Windows is able to push memory used by processes into the page file on the hard disk without those processes having any knowledge of what happened.
This is also how a single process can allocate more memory than the machine has physical RAM and yet still run. For instance, a program running on a machine with 512MB of RAM could still allocate 1GB of memory. Windows would just couldn't keep all of it in the RAM at the same time and some of it would be in the page file. But the program wouldn't know.
So consequently, if one process allocates memory, it does not cause another process to have less memory to work with. Each process is separate.
Each process only needs to worry about itself. And so the idea of freeing a pre-allocated chunk of memory is actually very viable.
You can't use CRT or MessageBox functions to handle OOM since they might need memory, as you describe. The only truly safe thing you can do is alloc a chunk of memory at startup you can write information into and open a handle to a file or a pipe, then WriteFile to it when you OOM out.