Linux system running an application. This application is a cpp binary without any debug symbols. Some how this application using 100% cpu. Would like to debug why it is running infinitely. If I stop and replace the binary with debug symbols, the issue may not be reproducible.
So, running the same application with debug symbols in another environment. Here it is running fine.
Can I compare them (with and without debug symbols binaries) and deduct what is the problem using GDB.
This application is a cpp binary without any debug symbols
You don't need any debug symbols to understand where it is spending time, you just need the application to not be fully-stripped (most binaries aren't).
Use perf record -p $pid to collect CPU profile, then perf report to analyse it.
If the application is fully stripped, you can still use perf record to collect program counter values, then perf record --symfs ... to point it to unstripped copy of the application. Documentation here.
Beware: both stripped and unstripped copies must be built with exactly the same build flags, or you'll get garbage. Best practice is to always save unstripped copy as part of the build process.
Related
I have a small Qt project written in c++ and when i debug it (Release) its taking like 7~10sec to start, even with the cache symbols saved into a local folder.
When I disable this option:
Debugging -> General -> [x] Load debug symbols in external process (Native only)
And select this option:
Debugging -> Symbols -> (*) Load only specified modules
It take around 1sec to start debugging the same project.
My computer specs:
Windows 10
Visual Studio 2022
cpu: i9 9900k
ssd: 970 evo plus (gen3 around 2~3k read/write speed)
Why such difference in time?
I'm asking because even 'disabling' the options i mentioned, intellisense is still working.
What are these symbols used for when debugging?
Does disabling these options impact in something when not debugging?
After the source code has been compiled, most of the actual names of variables will be stripped out to save space, because the actual machine code of the executable doesn't use the long string names of variables, it uses memory addresses.
So debugging symbols are used to provide information to the debugger which normally wouldn't be present in the final executable. Such as the names you provided in the source code to associate with various memory addresses. So if an error occurs, such as:
int* some_invalid_pointer = NULL;
// Segmentation fault should occur here
int some_variable = *some_invalid_pointer;
If you don't have debug symbols enabled, you may get a message such as Error: Segmentation Fault because the debugger has no idea what the name of each variable actually is. Debugging symbols give the debugger more information so it could print a message closer to Error: Segmentation Fault on Line 3: "int some variable = *some_invalid_pointer;" which is much easier to debug.
So it takes longer both because of the larger size of the executable needing to be loaded, and because debugging isn't as simple as running a version of the code compiled in debug mode. Debugging usually consists of running an actual separate Debugger executable, which then loads and runs your code. So when debugging you actually need to load and start running the Debugger, then the Debugger loads your program, which is especially slow because it could be the debugger loading it and not the OS like normal.
(To answer your final question, yes technically debugging info can be left in a release executable, but because of the way executables are usually Demand Paged into memory (You can look it up if you're interested), basically what it means is that info within an executable which is never actually referenced, shouldn't ever be loaded into memory and so it shouldn't have an impact on release runtime performance)
I have a very frustrating problem, I'm debugging my code, and for complete Call Stack I need to get symbols for msvcr120.dll (msvcr120.i386.pdb).
The thing is, that I can't get it.
tried loading it from MS Symbols Servers, result:
SYMSRV: http://msdl.microsoft.com/download/symbols/msvcr120.i386.pdb/16F5E2EF340A453ABC8B8F67DC6FD8082/msvcr120.i386.pdb not found
http://msdl.microsoft.com/download/symbols: Symbols not found on symbol server.
tried downloading manualy symbols packages for various Windows versions from MS, downloaded aprox. 2GB, got symbols for msvcr from 90 to 110, and two 120 (from Win 10 package):
msvcr120.i386.coresys.pdb
msvcr120_clr0400.i386.pdb
So, is there a way to get "msvcr120.i386.pdb" ?
In order to get the proper PDB for the file, you need to know which module is actually loaded. If you create a crashdump (.DMP) of the program while running (which you can do with taskmanager), it should show you the exact file loaded, including it's full path. You can also use a tool like dependency walker, which may do a better job at finding the full path.
Once you have the path, you can then specifically try to force-load symbols for that module by right-clicking, loading, and pointing to the exact file in question. That should get the proper hash and load the correct PDB from the MS symbol store into your local cache.
What's likely happening is that the program you're running either has a manifest calling out a specific SxS version of the DLL, OR your program has crashed enough that Windows has placed it in "compatibility mode". Either way, that means the program and VS are finding different versions of the DLL, thus the problem you're having. You can also see this when your program crashes on a different machine than the one you're debugging on, especially if they're not both exactly in the same place for windows updates.
If you do dev work regularly, you should disable Windows automatically putting things into compatibility mode. It will save you lots of headaches down the road.
I'm a developer of Windows desktop software, and from time to time our app crashes. In rare cases I'd like to get a customer to run a debug version of the app to send me a stack trace so I know where it crashed. I followed the instructions in here:
Windows C++ stack trace from a running app
...but while it works on my development machine, it doesn't work on any client machine or those of my colleagues, who don't have Visual Studio installed. So I presume that there's some .dll or something they need before it'll work. They're using the same .exe I'm using, i.e. the one I compiled in VC++ in debug mode.
After some painstaking "message window" debugging, I learnt it's failing in SymGetSymFromAddr64() - this returns FALSE. But when I walk the stack, this always returns FALSE or it returns garbage that doesn't make sense (random unrelated method names), as if it's the PC values which are invalid, not the mapping process. To reiterate, it's a debug mode .exe that produces a perfect symbolic stack trace on my development machine.
I did some research and found some mentions of "dbghelp.dll" and "imagehlp.dll" but I just ended up confused. "dbghelp.dll" ships with all versions of Windows, but with reduced functionality. There's some other things I could install, but it's a little scary to be installing some Windows "WDK" or "debug kits" which might overwrite important system .dll's or do god-knows-what to your computer.
So what I need to know is: "what's the simplest set of instructions I can give to these helper customers e.g. the minimum set of .dll's and where to stick them so that we can get proper symbolic information out of the stack traces when our program crashes?"
The most likely reason for failing to find the symbols is that the .pdb file cannot be found. Even if you generate a .exe in debug mode, the symbols are not in the .exe, they are in the .pdb file. Through a lot of empirical testing, it seems that the process has the pathname of the .pdb hardcoded in it, so if your clients don't have that file at that location, they won't necessarily find it. However, you can supply a "search path" to the SymInitialize() function - a folder or set of folders to use to search for the .pdb file.
In my configuration, I had an exe called "Edval.exe" and a .pdb called "DebugEdval.pdb". The process searches for "DebugEdval.pdb" in the search folders. This corresponds to what you've configured in "Properties > Linker > Debugging > Generate Program Database File".
What are the 'best practices' when it comes to debugging core dumps using GDB?
Currently, I am facing a problem:
The release version of my application is compiled without the '-g' compiler flag.
The debug version of my application (compiled with '-g') is archived (along with the source code, and a copy of the release binary).
Recently, when a user gave me a core dump, I tried debugging it using
gdb --core=./core.pid ./my_app_debug-bin
The core was created by my_app_release-bin. There seems to be some kind of mismatch between the core file and the binary.
On the other hand, if I try
gdb --core=./core.pid ./my_app_release-bin
the core matches but I am unable to get source code line numbers (although I get the function names).
Is this what is practised? Because I feel I am missing something here.
It sounds like there are other differences between your release and debug build then simply the absence/presence of the -g flag. Assuming that's the case, there is nothing you can do right now, but you can adjust your build to handle this better:
Here's what we do at my place of work.
Include the -g flag when building the release version.
Archive that version.
run strip --strip-unneeded on the binary before shipping it to customers.
Now, when we get a crash we can use the archived version with symbols to do debugging.
One thing to note is that if your release version includes optimization, debugging may be difficult even with symbols. For example, the optimizer can reorder your code so even though the debugger will say you crashed on line N, you can't assume that the code actually executed line N-1.
You need to do some additional stuff to create binaries with stripped debug information that you can then debug from cores. Best description I could find is here
No, you don't miss anything. debug and release are just different binaries, so the core files of release don't match the debug binary. You have to look at machine code to get something from the release core dump.
You probably have to ask your user how the crash happened and collect additional log information or whatever you app produces.
In a way following on from reading a windows *.dmp file
Having received a dump file from random customer, running the debug session to see the crash, you often find it is in a MS or other third party library. The next issue is that you may not have knowledge of the PC setup to such an extent that you can ensure you have the actually modules available.
For instance I'm currently stuck trying to get symbols to load for ntdll.dll (5.01.2600.5512). In MSVC 2005 the path column in the modules list window shows a * before the fully pathed file name, and refuses to load symbols I have downloaded for XP/SP1/SP1a/SP2/SP3.
I have the symbol server setup to download from the internet and store in a local cache which seems to have been working fine for modules that I do have on my PC.
Using GUI equivelant to the method
Set _NT_SYMBOL_PATH=srv*d:\SymbolCache*\\server1\Third-Party-PDB;srv*d:\SymbolCache*\\server2\Windows\Symbols*http://msdl.microsoft.com/download/symbols
Perhaps I have the wrong symbols, but as new ones are not downloading where do I go to next? Do I have to contact the customer and ask what SP they have installed, and any other patches? Do I have to install that machine and then run up the debugger with the dmp file to get the symbols I need?
If you are using WinDbg (part of the Debugging Tools for Windows package), then it's simple to have it pull the right symbols for you from Microsoft automatically. Configure the symbol path using the ".symfix" (or ".symfix+", to simply append to your existing symbol search path) command.
Once you have that done and you have the crash dump loaded in WinDbg, type ".reload /f" to cause WinDbg to reload the symbols. It will use the information within the dump file itself to pull the correct symbols from Microsoft's public symbol server, regardless of what DLLs you have on your machine.
If for some reason the symbols aren't loading properly after you have done this, enter "!sym noisy" into WinDbg's command window and reload the symbols again. As WinDbg attempts to load them, you will see it output any errors that it encounters in its search/load process. These error messages will help you further diagnose what is going wrong and why the correct symbols aren't being loaded.
This post has information that may also be of use.
If you're typing "Set _NT_SYMBOL_PATH = srv..." into a command prompt, there are two things to consider:
cmd.exe's set command does not ignore whitespace, so this defines a variable called "_NT_SYMBOL_PATH", not "_NT_SYMBOL_PATH".
You must start the debugger as a child of that command prompt. However, you don't have to do this if you use the Control Panel to set persistent environment variables, or if you use the setx command (in Windows Vista or one of the Windows Resource Kits).
If you're setting the symbol path some other way, then this doesn't apply.
What are you using to debug the minidump? I.e., WinDBG or Visual Studio? And how was the minidump generated?
There should be enough information in the minidump to resolve system dll symbols correctly. Are you using a local download of symbols or http://msdl.microsoft.com/?
Update: You should be able to add the public microsoft symbol store to Tools->Options->Debugging->Symbols->Symbol file (.pdb) locations, and then manually load the symbols by right clicking on the module in the Modules window and loading them if it isn't done automatically.
It's also possibly (likely) that VS 2005 doesn't look at _NT_SYMBOL_PATH to resolve minidump symbols.
Try following the instructions at this kb article, and make sure your symbol path is configured correctly, that WinDbg has access to it, and that the ntdll symbols (for example) are actually downloaded to your symbol cache. The article also provides instructions on how to manually download and verify symbols in you cache via the SymChk tool.