C++ - Struct packing and GCC debugging symbols mismatch - c++

I am working on a project where I need to parse the DWARF output of a compiler. I am working a Debian x64 (windows WSL) with GCC 8.3.0.
I face a compiler behaviour that I think might be a bug, but I am unsure if there's a subtlety I don't get.
I order to pack a structure, I use the following directive : #pragma pack(push,1). I think that GCC doesn't produce the right debugging symbol when the following conditions occurs (but not limited to them):
struct declaration in .h file
pragma directive in the .cpp file only, before the include
instance of struct delcared in the cpp file
Here's a structure :
struct StructD
{
unsigned int bitfieldA : 1;
unsigned int bitfieldB : 9;
unsigned int bitfieldC : 3;
unsigned int bitfieldD;
};
And here's a piece of code to test it:
file1StructDInstance.bitfieldA = 1;
file1StructDInstance.bitfieldB = 0b100111011;
file1StructDInstance.bitfieldC = 0b11;
file1StructDInstance.bitfieldD = 0b101001101;
unsigned char* ptr = (unsigned char*)(&file1StructDInstance);
printf("%02x %02x %02x %02x %02x %02x %02x %02x\n", ptr[0], ptr[1], ptr[2], ptr[3], ptr[4], ptr[5], ptr[6], ptr[7]);
Scenario 1 - No pragma
If I do not use the pragma anywhere, my struct is 32bits aligned and the debugging symbol matches. The software output is like this: 77 0e 00 00 4d 01 00 00.
<2><150>: Abbrev Number: 3 (DW_TAG_member)
<151> DW_AT_name : (indirect string, offset: 0xc2): bitfieldD
<155> DW_AT_decl_file : 2
<156> DW_AT_decl_line : 33
<157> DW_AT_decl_column : 15
<158> DW_AT_type : <0x83>
<15c> DW_AT_data_member_location: 4
The debugging sumbol report that bitfieldD is at the 4th byte in the structure, no bit offset. Which is right.
Scenario 2 - Pragma before struct declaration
If I put the pragma at the top of the .h file , I get this software output :
77 0e 4d 01 00 00 00 00
And the debugging symbol is as follow
<2><150>: Abbrev Number: 3 (DW_TAG_member)
<151> DW_AT_name : (indirect string, offset: 0xc2): bitfieldD
<155> DW_AT_decl_file : 2
<156> DW_AT_decl_line : 33
<157> DW_AT_decl_column : 15
<158> DW_AT_type : <0x83>
<15c> DW_AT_data_member_location: 2
So bitfieldD is at byte 2 without offset, which is again right and matches the memory layout.
Scenario 3 - Pragma in .cpp, but omitted in .h
When I put the pragma in the .cpp file, before the include of the .h file that define StructD, but I omit to put the pragma in the .h file, I get a mismatch between the compiled code and the debugging symbol.
software output : 77 0e 00 00 4d 01 00 00
And debugging symbol
<2><150>: Abbrev Number: 3 (DW_TAG_member)
<151> DW_AT_name : (indirect string, offset: 0xc2): bitfieldD
<155> DW_AT_decl_file : 2
<156> DW_AT_decl_line : 33
<157> DW_AT_decl_column : 15
<158> DW_AT_type : <0x83>
<15c> DW_AT_data_member_location: 2
Now the debugging symbols says that bitfieldD is at byte #2, but clearly the software put it at byte #4. I recognize that the usage of the pragma might not be proper, but I would expect GCC to produce debugging symbols that matches the generated code.
Is this a bug in GCC or am I misunderstanding how DWARF works?

Related

I have a question about memory management and the memory window from Visual Studio

Let's say I have this code:
#include<iostream>
using namespace std;
int main()
{
int a = 1633;
}
I put a breakpoint at line 4 so I can follow how the memory is allocated by using the Memory window from Visual Studio. I've noticed that when I plug in &a it will show this:
0x00CFF800 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 61 06 00 00 cc cc cc cc 3c
I understand that my variable is that 61 06 00 00 because it's highlighted.
Questions:
Why do I have all of these elements at that address and not only a : 61 06 00 00?
Why is it written like that? I know that there are 4 bytes, but I don't know how to convert that code to my actual number. So, why is 61 06 00 00 equal to a=1633?
Why do I have all of these elements at that adress
You don't. VS shows you bytes adjacent to your variable as well.
The left-most cc is at address 0x00CFF800, the next cc is at 0x00CFF801, and so on.
Why is a written like that?
Each pair of symbols represents one byte, the symbols are hexadecimal digits.
The bytes are arranged from least significant to most significant (the little-endian byte order).
The easiest way to interpret the value is to:
Reverse the byte order: 61 06 00 00 -> 00 00 06 61
Convert the resulting hexadecimal number to a decimal one: 00000661(hex) -> 1633(decimal)
This is equivalent to first converting each separate byte from hexadecimal to decimal: 97 6 0 0, and then multiplying the values by powers of 256: 97 * 256^0 + 6 * 256^1 + 0 * 256^2 + 0 * 256^3 = 97 + 6 * 256.

Finding address of the 'vftable' symbol in an .exe file

I am trying to find the address (or offset from base address) of the 'vftable' symbol for the class 'Greeter' by statically analyzing an executable file. While analyzing it dynamically in Visual Studio by adding a breakpoint and disassembling the constructor of the 'Greeter' class, I was able to find the address of the symbol.
00007FF76A891951 lea rcx,[Greeter::`vftable' (07FF76A89ACE0h)]
00007FF76A891958 mov qword ptr [rax],rcx
But this is it's runtime address. I need to find a way to figure out the offset by maybe using a tool like 'dumpbin' or something similar. I know I can figure out the offset by using the address above, but I need a way to automate this, so it will have to be through a tool.
I tried using 'dumpbin' on the exe to disassemble it and find the same instructions of the Greeter class:
0000000140011951: 48 8D 0D 88 93 00 lea rcx,[??_7Greeter##6B#]
0000000140011958: 48 89 08 mov qword ptr [rax],rcx
So, I set out to try to find references to this symbol ??_7Greeter##6B#
I tried using 'dumpbin' with the following command:
dumpbin /all ConsoleApplication.obj > cout
I got the following relevant output:
Section length 18, #relocs 3, #linenums 0, checksum 0, selection 6 (pick largest)
Relocation CRC BDB82F45
134 00000008 SECT43 notype External | ??_7Greeter##6B# (const Greeter::`vftable')
135 00000000 SECT44 notype Static | .rdata
I also got this output:
SECTION HEADER #43
.rdata name
0 physical address
0 virtual address
18 size of raw data
DA3B file pointer to raw data (0000DA3B to 0000DA52)
DA53 file pointer to relocation table
0 file pointer to line numbers
3 number of relocations
0 number of line numbers
40401040 flags
Initialized Data
COMDAT; sym= "const Greeter::`vftable'" (??_7Greeter##6B#)
8 byte align
Read Only
RAW DATA #43
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000010: 00 00 00 00 00 00 00 00 ........
RELOCATIONS #43
Symbol Symbol
Offset Type Applied To Index Name
-------- ---------------- ----------------- -------- ------
00000000 ADDR64 00000000 00000000 14C ??_R4Greeter##6B# (const Greeter::`RTTI Complete Object Locator')
00000008 ADDR64 00000000 00000000 8C ?sayHello#Greeter##UEAAX_J00#Z (public: virtual void __cdecl Greeter::sayHello(__int64,__int64,__int64))
00000010 ADDR64 00000000 00000000 8D ?initUser#Greeter##UEAAXXZ (public: virtual void __cdecl Greeter::initUser(void))
Does anybody have any idea how I would go about finding the offset of this symbol? Is there a specific 'dumpbin' option that I need to use to print offset of symbols that are mentioned in the disassembly?

Dwarf DW_AT_location objdump and dwarfdump inconsistent

I am playing around with CPython and trying to understand how a debugger works.
Specifically, I am trying to get the location of the last PyFrameObject so that I can traverse that and get the Python backtrace.
In the file ceval.c, line 689 has the definition of the function:
PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
What I am interested in getting is the location of f on the stack. When dumping the binary with dwarfdump I get that f is at $rbp-824, but if I dump the binary with objdump I get that the location is $rbp-808 - a discrepancy of 16. Also, when debugging with GDB, I get that the correct answer is $rbp-808 like objdump gives me. Why the discrepancy, and why is dwarfdump incorrect? What am I not understanding?
How to technically recreate the problem:
Download python-2.7.17.tgz from Python website. Extract.
I compiled python-2.7.17 from source with debug symbols (./configure --enable-pydebug && make). Run the following commands on the resulting python binary:
dwarfdump Python-2.7.17/python has the following output:
DW_AT_name f
DW_AT_decl_file 0x00000001 /home/meir/code/python/Python-2.7.17/Python/ceval.c
DW_AT_decl_line 0x000002b1
DW_AT_type <0x00002916>
DW_AT_location len 0x0003: 91c879: DW_OP_fbreg -824
I know this is the correct f because the line the variable is declared on is 689 (0x2b1). As you can see the location is:
DW_AT_location len 0x0003: 91c879: DW_OP_fbreg -824: Meaning $rbp-824.
Running the command objdump -S Python-2.7.17/python has the following output:
PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
{
f7577: 55 push %rbp
f7578: 48 89 e5 mov %rsp,%rbp
f757b: 41 57 push %r15
f757d: 41 56 push %r14
f757f: 41 55 push %r13
f7581: 41 54 push %r12
f7583: 53 push %rbx
f7584: 48 81 ec 38 03 00 00 sub $0x338,%rsp
f758b: 48 89 bd d8 fc ff ff mov %rdi,-0x328(%rbp)
f7592: 89 b5 d4 fc ff ff mov %esi,-0x32c(%rbp)
f7598: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax
f759f: 00 00
f75a1: 48 89 45 c8 mov %rax,-0x38(%rbp)
f75a5: 31 c0 xor %eax,%eax
Debugging this output will show you that the relevant line is:
f758b: 48 89 bd d8 fc ff ff mov %rdi,-0x328(%rbp) where you can clearly see that f is being loaded from -0x328(%rbp) which is $rbp-808. Also, GDB supports this finding.
So again, the question is, what am I missing and why the 16 byte discrepency between dwarfdump and reality?
Thanks
Edit:
The dwarfdump including the function above is:
< 1><0x00004519> DW_TAG_subprogram
DW_AT_external yes(1)
DW_AT_name PyEval_EvalFrameEx
DW_AT_decl_file 0x00000001 /home/meir/code/python/Python-2.7.17/Python/ceval.c
DW_AT_decl_line 0x000002b1
DW_AT_prototyped yes(1)
DW_AT_type <0x00000817>
DW_AT_low_pc 0x000f7577
DW_AT_high_pc <offset-from-lowpc>53969
DW_AT_frame_base len 0x0001: 9c: DW_OP_call_frame_cfa
DW_AT_GNU_all_tail_call_sites yes(1)
DW_AT_sibling <0x00005bbe>
< 2><0x0000453b> DW_TAG_formal_parameter
DW_AT_name f
DW_AT_decl_file 0x00000001 /home/meir/code/python/Python-2.7.17/Python/ceval.c
DW_AT_decl_line 0x000002b1
DW_AT_type <0x00002916>
DW_AT_location len 0x0003: 91c879: DW_OP_fbreg -824
According to the answer below, DW_OP_fbreg is offset from the frame base - in my case DW_OP_call_frame_cfa. I am having trouble identifying the frame base. My registers are as following:
(gdb) info registers
rax 0xfffffffffffffdfe -514
rbx 0x7f6a4887d040 140094460121152
rcx 0x7f6a48e83ff7 140094466441207
rdx 0x0 0
rsi 0x0 0
rdi 0x0 0
rbp 0x7ffd24bcef00 0x7ffd24bcef00
rsp 0x7ffd24bceba0 0x7ffd24bceba0
r8 0x7ffd24bcea50 140725219813968
r9 0x0 0
r10 0x0 0
r11 0x246 582
r12 0x7f6a48870df0 140094460071408
r13 0x7f6a48874b58 140094460087128
r14 0x1 1
r15 0x7f6a48873794 140094460082068
rip 0x5559834e99c0 0x5559834e99c0 <PyEval_EvalFrameEx+46153>
eflags 0x246 [ PF ZF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
As stated above, I already know that %rbp-808 works. What is the correct way to do it with the registers that I have?
Edit:
I finally understood the answer. I needed to unwind one more function, and find the place my function was called. There, the variable I was looking for really was in $rsp and $rsp-824 was correct
DW_OP_fbreg -824: Meaning $rbp-824
It does not mean that. It means, offset -824 from frame base (virtual) register, which is not necessarily (nor usually) equal to $rbp.
You need to look for DW_AT_frame_base to know what the frame base in the current function is.
Most likely it's defined as DW_OP_call_frame_cfa, which is the value of $RSP just before current function was called, and is equal to $RBP-16 (8 bytes for return address saved by the CALL instruction, and 8 bytes for previous $RBP saved by the first instruction of your function).

Misaligned address using virtual inheritance

The following apparently valid code produces a misaligned address runtime error using the UndefinedBehaviorSanitizer sanitiser.
#include <memory>
#include <functional>
struct A{
std::function<void()> data; // seems to occur only if data is a std::function
} ;
struct B{
char data; // occurs only if B contains a member variable
};
struct C:public virtual A,public B{
};
struct D:public virtual C{
};
void test(){
std::make_shared<D>();
}
int main(){
test();
return 0;
}
Compiling and executing on a macbook with
clang++ -fsanitize=undefined --std=c++11 ./test.cpp && ./a.out
produces the output
runtime error: constructor call on misaligned address 0x7fe584500028 for type 'C', which requires 16 byte alignment [...].
I would like to understand how and why the error occurs.
Since alignment of std::function<void()> is 16 and size is 48 lets simplify. This code has the same behavior but is easier to understand:
struct alignas(16) A
{ char data[48]; };
struct B
{ char data; };
struct C : public virtual A, public B
{};
struct D : public virtual C
{};
int main()
{
D();
}
We have the following alignments and sizes:
|__A__|__B__|__C__|__D__|
alignment (bytes): | 16 | 1 | 16 | 16 |
size (bytes): | 48 | 1 | 64 | 80 |
Now lets see how this looks like in memory. More explanation on that can be found in this great answer.
A: char[48] + no padding == 48B
B: char[1] + no padding == 1B
C: A* + B + A + 7 bytes of padding (align to 16) == 64B
D: C* + C + 8 bytes of padding (align to 16) == 80B
Now it is easy to see that the offset of C inside D is 8 bytes, but C is aligned to 16. Thus error, which is helpfully accompanied by this pseudo-graphic
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
^
Here each zero is 1 byte.
UPDATE:
Where and how to place padding is up to a C++ compiler. Standard does not specify it. It looks like with the size of padding it has, clang is unable to align everything in D. One way to mitigate the misalignment is to design your classes carefully so that they have the same alignment (e.g., 8 bytes).

Disable alignment on a 64-bit structure

I'm trying to align my structure and make it as small as possible using bit fields. I have to send this data back to a client, which will examine the fields to set a few data members.
The size of the structure is indeed the same, but when I set members it does not work at all.
Here's some example code:
#pragma pack(push, 1)
struct PW_INFO
{
char hash[16]; //Does not matter
uint32_t number; //Does not matter
uint32_t salt_id : 30; //Position: 0 bits
uint32_t enc_level : 7; //Position: 30 bits
uint32_t delta : 27; //Position: 37 bits
}; //Total size: 28 bytes
#pragma pack(pop)
void int64shrl(uint64_t& base, uint32_t to_shift, uint32_t position)
{
uint64_t res = static_cast<uint64_t>(to_shift);
res = Int64ShllMod32(res, position);
base |= res;
}
int32_t main()
{
std::cout << "Size of PW_INFO: " << sizeof(PW_INFO) << "\n"; //Returns 28 as expected (16 + sizeof(uint32_t) + 8)
PW_INFO pw = { "abc123", 0, 0, 0, 0 };
pw.enc_level = 105;
uint64_t base{ 0 };
&base; //debug purposes
int64shrl(base, 103, 30);
return 0;
}
Here's where it gets weird: setting the "salt_id" field (which is 30 bits into the bitfield) will yield the following result in memory:
0x003FFB8C 61 62 63 31 32 33 00 00 abc123..
0x003FFB94 00 00 00 00 00 00 00 00 ........
0x003FFB9C 00 00 00 00 00 00 00 00 ........
0x003FFBA4 69 00 00 00 i...
(Only the last 8 bytes are of concern since they represent the bit field.)
But, Int64ShllMod32 returns a correct result (the remote client undersands it perfectly):
0x003FFB7C 00 00 00 c0 19 00 00 00 ...À....
I'm guessing it has to do with alignment, if so how would I completely get rid of it? It seems even if the size is correct, it will try to align it (1 byte boundary as the #pragma directive suggests).
More information:
I use Visual Studio 2015 and its compiler.
I am not trying to write those in a different format, the reason I'm asking this is that I do NOT want to use my own format. They are reading from 64 bit bitfields everywhere, I don't have access to the source code but I see a lot of calls to Int64ShrlMod32 (from what I read, this is what the compiler produces when dealing with 8 byte structures).
The actual bitfield starts at "salt_id". 30 + 7 + 27 = 64 bits, I hope it is clearer now.