libgcc_s.so conflicts could lead to cpu overload using exceptions? - c++

Question

I developed a C++ server application for embedded i386 compatible environment, so no cross compiler was needed. A dynamic library developed by a collegue, making (large) use of exceptions tecnique. That library is demanded to implement network communications, and, once copied in the target file sytem, after the client connection, causes an abort with the common message:terminated after throwing an instance of... even if the libstdc++ is available on the embedded os.
After several tries, including static link of libraries, we apparently found a solution just copying the libgcc_s.so.1 used at compile time on a Fedora3 virtual machine to the embedded file system and launching the server with environment variable LD_LIBRARY_PATH=path to fedora lib.
On the embedded os we have a busybox with few and reduced tools, but we noticed, with the command uptime, that after the client connection, the cpu usage raised from 20% to 100% (and I don't know how... even more). The first impression is an application bug but it was never noticed during the debug sessions on the Fedora machine and if you see on /proc/task/status you will see this log:
Name: taskname
State: S (sleeping)
SleepAVG: 97%
Tgid: 589
Pid: 589
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups: 0
VmSize: 3396 kB
VmLck: 0 kB
VmRSS: 1604 kB
VmData: 492 kB
VmStk: 84 kB
VmExe: 84 kB
VmLib: 2512 kB
VmPTE: 20 kB
Threads: 1
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000080000000
SigIgn: 0000000000001004
SigCgt: 0000000380004a02
CapInh: 0000000000000000
CapPrm: 00000000fffffeff
CapEff: 00000000fffffeff
So I cannot figure out who's using the cpu massively, even if the client disconnects.
This behaviour is not present if the server is launched on the Fedora machine.
I suspect that mixing the Fedora3 libgcc_s.so.1 with embedded system could lead to some strange side effect but I don't have any clue.
So I started to find another way to deploy the server:
Copying others required libraries from Fedora3 to embedded so (libstdc++ and libc). Same behaviour
Reversing the process: copying the required libraries to source tree and forcing the linker to use those libraries. Launching the application (compiler side) the error message terminated after throwing an instance of... respawned.
Additional Infos:
If useful: applying ldd -v libgcc_s.so.1 (not available on embedded system) on the two libraries I had the following results:
HOST LIBRARY:
libc.so.6 => /lib/tls/libc.so.6 (0x00694000)
/lib/ld-linux.so.2 (0x0067b000)
Version information:
/lib/libgcc_s.so.1:
libc.so.6 (GLIBC_2.2.4) => /lib/tls/libc.so.6
libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
/lib/tls/libc.so.6:
ld-linux.so.2 (GLIBC_2.1) => /lib/ld-linux.so.2
ld-linux.so.2 (GLIBC_2.3) => /lib/ld-linux.so.2
ld-linux.so.2 (GLIBC_PRIVATE) => /lib/ld-linux.so.2
ld-linux.so.2 (GLIBC_2.0) => /lib/ld-linux.so.2
EMBEDDED LIBRARY:
libc.so.6 => /lib/tls/libc.so.6 (0xf6ec3000)
/lib/ld-linux.so.2 (0x0067b000)
Version information:
./libgcc_s.so.1:
libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
/lib/tls/libc.so.6:
ld-linux.so.2 (GLIBC_2.1) => /lib/ld-linux.so.2
ld-linux.so.2 (GLIBC_2.3) => /lib/ld-linux.so.2
ld-linux.so.2 (GLIBC_PRIVATE) => /lib/ld-linux.so.2
ld-linux.so.2 (GLIBC_2.0) => /lib/ld-linux.so.2
Someone have any explanation or suggestion?
Thank you
A. Cappelli
More info about processors type:
Compiler host /proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.40GHz
stepping : 1
cpu MHz : 3390.524
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss nx pni
bogomips : 6471.68
Embedded machine /proc/cpu_info:
processor : 0
vendor_id : AuthenticAMD
cpu family : 4
model : 9
model name : 486 DX/4-WB
stepping : 4
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu
bogomips : 65.40

Answer 1

If your embedded system has enough recent version of Linux kernel, you can try using Linux performance counters(perf). When you install them run perf record ./server on your embedded system. This will generate perf.data when the server exits. After that you can just analyze the file using perf report in the same directory as the file. It will show how much CPU% each library and executable symbol used. Then you can narrow down the issue to the libraries or your server code. More info about perf here