If I have a C or CPP program written like
#include <stdio.h>
int bss_section_object[10];
int main()
{
return 0;
}
After compiling the code with gcc/g++,
$g++ -save-temps aa.cpp
If I check the size of generated object and executable files
$ls -ltr
-rw-rw-r-- 1 dinesh dinesh 77 Feb 14 12:42 aa.cpp
-rw-rw-r-- 1 dinesh dinesh 17283 Feb 14 12:45 aa.ii
-rw-rw-r-- 1 dinesh dinesh 517 Feb 14 12:45 aa.s
-rw-rw-r-- 1 dinesh dinesh 1296 Feb 14 12:45 aa.o
-rwxrwxr-x 1 dinesh dinesh 8592 Feb 14 12:45 a.out
$size a.out
text data bss dec hex filename
1099 544 72 1715 6b3 a.out
$size aa.o
text data bss dec hex filename
67 0 40 107 6b aa.o
Now if I comment the object created on .bss section
#include <stdio.h>
//int bss_section_object[10];
int main()
{
return 0;
}
and check the size after compling it:-
$ls -ltr
-rw-rw-r-- 1 dinesh dinesh 79 Feb 14 12:46 aa.cpp
-rw-rw-r-- 1 dinesh dinesh 17256 Feb 14 12:46 aa.ii
-rw-rw-r-- 1 dinesh dinesh 378 Feb 14 12:46 aa.s
-rw-rw-r-- 1 dinesh dinesh 1232 Feb 14 12:46 aa.o
-rwxrwxr-x 1 dinesh dinesh 8552 Feb 14 12:46 a.out
$size a.out
text data bss dec hex filename
1099 544 8 1651 673 a.out
$size aa.o
text data bss dec hex filename
67 0 0 67 43 aa.o
I can see that their size varies based on "bss_section_object" size.
If I read the details of .bss section, then as per wikipedia:-
"The size that BSS will require at runtime is recorded in the object file,
but BSS (unlike the data segment) doesn't take up any actual space in the
object file".
We can see similar description of .bss section in other computer science
books as well.
Now, If the .bss section doesn't take up any space in object file, then why
is the size of object as well as executable file changing based on size of
object created on .bss section?
Can someone please explain the details about it?
You are perfectly right when saying :
"The size that BSS will require at runtime is recorded in the object file, but BSS (unlike the data segment) doesn't take up any actual space in the object file"
Because uninitialized data is placed in the .bss section of the program, which is of type SHT_NOBITS (see the elf(5) manpage).
To convince yourself that this section doesn't takes up any space in the ELF file, try readelf -Wl /bin/ls and look at the fourth segment :
[...]
LOAD 0x01a310 0x000000000061a310 0x000000000061a310 0x0012f0 0x002010 RW 0x200000
[...]
`03 .init_array .fini_array .jcr .data.rel.ro .dynamic .got .got.plt .data .bss
You can see that it contains (among others), the .bss section, and has file size 0x12f0 but memory size 0x2010. The difference is the size of the .bss section.
This section is not included in the file because the associated memory region will be zeroed out at program startup, and it would be unnecessary to store zeros in the file.
You're right, but your test is wrong
In the example you give, given that you disable optimization and debugger symbols, the difference you see in file size may be explained by the symbol table, and not by the change of the .bss section size.
Here's an experiment you can reproduce :
you$ cat main.cpp
#include <stdio.h>
#ifdef USE_BSS
int bss_section_object[10];
#endif
int main()
{
return 0;
}
you$ g++ -O0 main.cpp -o without.elf
you$ g++ -DUSE_BSS -O0 main.cpp -o with.elf
you$ ls -l *.elf
-rwxr-xr-x. 1 you you 8505 Feb 14 10:00 with.elf
-rwxr-xr-x. 1 you you 8462 Feb 14 10:00 without.elf
At this point, you may believe that changing the .bss section size affects the file size, but in fact the difference is due to the additional symbol bss_section_object in the symbol table (which you can remove using the strip command) :
you$ readelf -WS with.elf > with && readelf -WS without.elf > without
you$ diff with without
30c30
< [25] .bss NOBITS 0000000000601040 00102c 000048 00 WA 0 0 32
---
> [25] .bss NOBITS 000000000060102c 00102c 000004 00 WA 0 0 4
33,34c33,34
< [28] .symtab SYMTAB 0000000000000000 0018e8 000618 18 29 45 8
< [29] .strtab STRTAB 0000000000000000 001f00 000239 00 0 0 1
---
> [28] .symtab SYMTAB 0000000000000000 0018e8 000600 18 29 45 8
> [29] .strtab STRTAB 0000000000000000 001ee8 000226 00 0 0 1
# Yep, the .symtab and .strtab are also different
you$ strip *.elf
you$ ls -l *.elf
-rwxr-xr-x. 1 you you 6232 Feb 14 10:06 with.elf
-rwxr-xr-x. 1 you you 6232 Feb 14 10:06 without.elf
Yay ! By removing the symbol table section .symtab (it its associated string table .strtab), you end up with the same file size, but different .bss section sizes, which is what you expected.
Related
I discovered via my previous question that depth appears to work differently when I use it in ImageMagick's convert vs Magick++.
CLI version and result
Using:
$ convert /foo/bar.ppm -depth 1 /foo/out.ppm
I get an output image which, upon inspection, shows a 1-bit color depth:
$ identify /foo/out.ppm
out.ppm PPM (blah blah) 1-bit sRGB (blah blah)
C++ version and result
Using the code:
#include <Magick++.h>
int main(int argc, char **argv) {
Magick::InitializeMagick(*argv);
Magick::Image img;
img.read("/foo/bar.ppm");
Magick::Image temp_img(img);
temp_img.depth(1);
temp_img.write("/foo/out.ppm");
return 0;
}
Compiled using the command:
g++ -std=c++17 test.cpp -o test `Magick++-config --cppflags --cxxflags --ldflags --libs`
Produces the output:
$ identify /foo/out.ppm
out.ppm PPM (blah blah) 8-bit sRGB (blah blah)
Hardware
I have run this with the same result on on:
Raspberry Pi - Raspbian 10 (buster)
Laptop - Ubuntu 18.04 (bionic beaver)
Software (on the RPi)
$ apt list --installed | grep magick
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
graphicsmagick-libmagick-dev-compat/stable,now 1.4+really1.3.35-1~deb10u1 all [installed]
imagemagick-6-common/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,automatic]
imagemagick-6.q16/now 8:6.9.10.23+dfsg-2.1 armhf [installed,upgradable to: 8:6.9.10.23+dfsg-2.1+deb10u1]
imagemagick/now 8:6.9.10.23+dfsg-2.1 armhf [installed,upgradable to: 8:6.9.10.23+dfsg-2.1+deb10u1]
libgraphics-magick-perl/stable,now 1.4+really1.3.35-1~deb10u1 armhf [installed,automatic]
libgraphicsmagick++-q16-12/stable,now 1.4+really1.3.35-1~deb10u1 armhf [installed,automatic]
libgraphicsmagick++1-dev/stable,now 1.4+really1.3.35-1~deb10u1 armhf [installed,automatic]
libgraphicsmagick-q16-3/stable,now 1.4+really1.3.35-1~deb10u1 armhf [installed,automatic]
libgraphicsmagick1-dev/stable,now 1.4+really1.3.35-1~deb10u1 armhf [installed,automatic]
libmagick++-6-headers/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,auto-removable]
libmagick++-6.q16-8/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 armhf [installed,auto-removable]
libmagickcore-6-arch-config/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 armhf [installed,auto-removable]
libmagickcore-6-headers/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,auto-removable]
libmagickcore-6.q16-6-extra/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 armhf [installed,automatic]
libmagickcore-6.q16-6/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 armhf [installed,automatic]
libmagickwand-6-headers/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,auto-removable]
libmagickwand-6.q16-6/stable,now 8:6.9.10.23+dfsg-2.1+deb10u1 armhf [installed,automatic]
The Picture
I've tested with multiple input files with sRGB type. I convert everything to NetBPM format before starting my test e.g.:
convert yourimage.jpg /foo/bar.ppm
The Question
Why is the C++ different from the Bash version? They should be linking to the exact same code in the background. The input value for depth does not need to be a special type (Magick::Image.depth takes size_t). Is there something in my installation which is messing this up? I know most of it is based on ImageMagick v6 because debian repos are notoriously slow, but nothing has changed (to my knowledge) in the source code which should effect the depth.
What else doesn't work?
Quantization
Adding:
temp_img.quantizeColorSpace(Magick::GRAYColorspace);
temp_img.quantizeColors(1);
temp_img.quantize( );
to the code should also be a method which reduces the color depth. Again, this results in an 8-bit image in C++.
Monochrome
This results in an 8-bit image in both CLI and C++
Closest solution I can think of is to user the "PBM" format.
Generating a test image with the following.
convert -size 10x10 plasma: input.jpg && convert input.jpg input.ppm
Just using the Magick::Image.magick method.
#include <Magick++.h>
int main(int argc, char **argv) {
Magick::InitializeMagick(*argv);
Magick::Image img;
img.read("input.ppm");
Magick::Image temp_img(img);
temp_img.magick("PBM");
temp_img.depth(1);
temp_img.write("output.ppm");
return 0;
}
We get the following file structure...
$ hexdump -C output.ppm
00000000 50 34 0a 31 30 20 31 30 0a 00 00 00 00 00 00 06 |P4.10 10........|
00000010 00 ff 80 ff c0 ff c0 ff c0 ff c0 ff c0 |.............|
0000001d
If we want the ASCII representation of the binary data, just disable the compression.
Magick::Image temp_img(img);
temp_img.compressType(Magick::NoCompression);
temp_img.magick("PBM");
temp_img.depth(1);
temp_img.write("output.ppm");
Which would yield the following...
$ hexdump -C output2.ppm
00000000 50 31 0a 31 30 20 31 30 0a 30 20 30 20 30 20 30 |P1.10 10.0 0 0 0|
00000010 20 30 20 30 20 30 20 30 20 30 20 30 20 0a 30 20 | 0 0 0 0 0 0 .0 |
00000020 30 20 30 20 30 20 30 20 30 20 30 20 30 20 30 20 |0 0 0 0 0 0 0 0 |
00000030 30 20 0a 30 20 30 20 30 20 30 20 30 20 30 20 30 |0 .0 0 0 0 0 0 0|
00000040 20 30 20 30 20 30 20 0a 30 20 30 20 30 20 30 20 | 0 0 0 .0 0 0 0 |
00000050 30 20 31 20 31 20 30 20 30 20 30 20 0a 31 20 31 |0 1 1 0 0 0 .1 1|
00000060 20 31 20 31 20 31 20 31 20 31 20 31 20 31 20 30 | 1 1 1 1 1 1 1 0|
00000070 20 0a 31 20 31 20 31 20 31 20 31 20 31 20 31 20 | .1 1 1 1 1 1 1 |
00000080 31 20 31 20 31 20 0a 31 20 31 20 31 20 31 20 31 |1 1 1 .1 1 1 1 1|
00000090 20 31 20 31 20 31 20 31 20 31 20 0a 31 20 31 20 | 1 1 1 1 1 .1 1 |
000000a0 31 20 31 20 31 20 31 20 31 20 31 20 31 20 31 20 |1 1 1 1 1 1 1 1 |
000000b0 0a 31 20 31 20 31 20 31 20 31 20 31 20 31 20 31 |.1 1 1 1 1 1 1 1|
000000c0 20 31 20 31 20 0a 31 20 31 20 31 20 31 20 31 20 | 1 1 .1 1 1 1 1 |
000000d0 31 20 31 20 31 20 31 20 31 20 0a |1 1 1 1 1 .|
000000db
Don't know if that's exactly which you need, but should get you on track. Also might be worth reviewing WritePNMImage method in coders/pnm.c file (same file for ImageMagick-6).
Solution
It appears that this problem was solved by removing problem packages previously downloaded from the Debian apt repository. It is difficult to nail down which was the offending part, but I removed:
sudo apt remove graphicsmagick-libmagick-dev-compat imagemagick-6-common imagemagick-6.q16 imagemagick
Next, I built ImageMagick from source, following the directions here.
Explanation
The solution was not simply a version change, which would be an understandable confusion since during the source build, I upgraded from v6 of ImageMagick still in the Debian repository to v7. However, tests by #emcconville were performed on both v6 and v7 without reproducing the errors I experienced. Presumably, since he is involved with ImageMagick development, he uses a copy built from source rather than what is available from apt-get. Therefore, we can safely assume that the problem is either in one of the Debian packages or caused by some incorrect combination of packages on the affected machine.
I have two files having each two columns: (Count | Term). Where term column is in Unicode and some terms are emojis.
I am trying to join both files based on term column (second column) using this code:
join -j 2 -o 1.1,1.2,2.1,2.2 <(sort -k2 File1.txt) <(sort -k2 File2.txt) > join_File1_File2.txt
When I join the two files I have correct output except some wrong lines having emoji joined with letters and characters like :
11 ๐ ๐๐๐ฆ๐๐ฅ๐๐ฆโซ 4 ๐๏ธ๐๏ธ๐๏ธ๐๏ธ๐๏ธ
484 โฆโฆโฆ 79 โโโ
27 โโโโโโโโโโโโโโโโ 25 ๐จ๐ฆ๐จ๐ฆ๐จ๐ฆ๐จ๐ฆ๐จ๐ฆ๐จ๐ฆ
I have even some wrong matched emojis like this :
39 ๐ 726 ๐ด
And some wrong matched characters like :
2 ยฐโฒ 1 ยฐโ
Here is a sample of Files : File1, File2 and JoinFile
File1 :
1 โโฆโฆโฆโฆ
1369 i
1347 โฆ
1339 it
8 โ
๐๐ โ
1322 with
1 ๐โโพโ๐โพโโด
1302 are
1299 your
1276 my
39 ๐
1272 with
1261 from
1255 this
1244 what
File 2:
1437 to
1435 your
1433 are
1421 in
83 โ๏ธ๐ฎ๐น
1411 and
1404 for
1 โ๐ป๐ป๐ป
1384 you
1373 on
726 ๐ด
1347 โฆ
13 โค๏ธ๐๐งก๐๐๐
1333 this
1322 with
Join_File1_File2
1 โโฆโฆโฆโฆ 1 โ๐ป๐ป๐ป
1347 โฆ 1347 โฆ
39 ๐ 726 ๐ด
8 โ
๐๐ โ 83 โ๏ธ๐ฎ๐น
1 ๐โโพโ๐โพโโด 13 โค๏ธ๐๐งก๐๐๐
1302 are 1433 are
1255 this 1333 this
1272 with 1322 with
1322 with 1322 with
1299 your 1435 your
I'm trying to implement the method described here and here which override new and delete in a SWIG module in order to allocate memory on the Java heap instead of natively.
I added the SWIG prescription to my interface file. It compiles fine, and when I run my Java GUI, I can see from cout statements that it appears to be newing and deleteing without problems for a while (over 100k times). Eventually though, I always get this (same stack trace):
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fff8754c0e0, pid=61178, tid=81411
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C [libsystem_platform.dylib+0x50e0] _platform_memmove$VARIANT$Haswell+0x1c0
Using my cout statements, it looks like the sample code is using the same addresses inside delete as it used in new, so the code itself appears to be working. There are no known memory issues with my code (I've analyzed it using AddressSanitizer and Valgrind), and furthermore, if I comment out only the cached_jvm = jvm; line (which results in the code using malloc/free instead of on the JVM heap), then it works ok. I'm not running out of heap memory -- for my testing I'm using small sample data.
If I switch the ReleaseByteArrayElements line to use mode JNI_ABORT instead of 0, then the free does not fail in the same place, but other problems occur afterwards which I didnโt bother to figure out.
Why is it segfaulting? Are there certain kinds of memory operations that are not compatible with this approach? I'm not using malloc/free (directly). I'm linking to a 3rd party library in C++, and they might be up to something.
Here's a sample compile line:
/usr/bin/clang++ -DEngine_EXPORTS -m64 -W -Wall -Wconversion -Wshadow -Wno-error=shadow -fdiagnostics-show-option -Wno-sign-conversion -stdlib=libc++ -ftemplate-depth=1000 -O3 -fno-omit-frame-pointer -ffast-math -funroll-loops -dynamic -fPIC -fno-common -fvisibility=hidden -fvisibility-inlines-hidden -Werror -Wno-error -fno-strict-aliasing -g -fPIC -I/third/party/include -I/my/include -I/System/Library/Frameworks/JavaVM.framework/Headers -o CMakeFiles/Engine.dir/engineJAVA_wrap.cxx.o -c engineJAVA_wrap.cxx
Stack trace:
Thread 59 Crashed:: Java: Thread-22
0 libsystem_kernel.dylib 0x00007fff902fbf06 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff98e704ec pthread_kill + 90
2 libsystem_c.dylib 0x00007fff95c736df abort + 129
3 libjvm.dylib 0x000000010a282beb os::abort(bool) + 25
4 libjvm.dylib 0x000000010a3a6a2a VMError::report_and_die() + 2304
5 libjvm.dylib 0x000000010a3a6d9b crash_handler(int, __siginfo*, void*) + 106
6 libsystem_platform.dylib 0x00007fff8754952a _sigtramp + 26
7 ??? 000000000000000000 0 + 0
8 libjvm.dylib 0x000000010a27d0b1 os::print_location(outputStream*, long, bool) + 917
9 libjvm.dylib 0x000000010a283d89 os::print_register_info(outputStream*, void*) + 83
10 libjvm.dylib 0x000000010a3a588c VMError::report(outputStream*) + 2142
11 libjvm.dylib 0x000000010a3a6522 VMError::report_and_die() + 1016
12 libjvm.dylib 0x000000010a284816 JVM_handle_bsd_signal + 1131
13 libjvm.dylib 0x000000010a280a7b signalHandler(int, __siginfo*, void*) + 47
14 libsystem_platform.dylib 0x00007fff8754952a _sigtramp + 26
15 ??? 0x0000000000000001 0 + 1
16 libjvm.dylib 0x000000010a11d830 jni_ReleaseByteArrayElements + 124
17 libengineEngine.jnilib 0x000000012750e41b JNIEnv_::ReleaseByteArrayElements(_jbyteArray*, signed char*, int) + 75 (jni.h:1689)
18 libengineEngine.jnilib 0x000000012750e1f8 operator delete(void*) + 408 (engineJAVA_wrap.cxx:298)
19 libengineEngine.jnilib 0x000000012750f710 Java_engineJNI_delete_1VectorMol + 80 (engineJAVA_wrap.cxx:586)
20 ??? 0x000000010ad9e954 0 + 4477020500
21 ??? 0x000000010ad90c4d 0 + 4476963917
22 ??? 0x000000010ad90c4d 0 + 4476963917
23 ??? 0x000000010ad90c92 0 + 4476963986
24 ??? 0x000000010ad90c4d 0 + 4476963917
25 ??? 0x000000010ad90c4d 0 + 4476963917
26 ??? 0x000000010ad90c4d 0 + 4476963917
27 ??? 0x000000010ad90c4d 0 + 4476963917
28 ??? 0x000000010ad909d0 0 + 4476963280
29 ??? 0x000000010ad90c92 0 + 4476963986
30 ??? 0x000000010ad897a7 0 + 4476934055
31 libjvm.dylib 0x000000010a0eab0e JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 1710
32 libjvm.dylib 0x000000010a0eb29c JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*) + 356
33 libjvm.dylib 0x000000010a0eb448 JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*) + 74
34 libjvm.dylib 0x000000010a145965 thread_entry(JavaThread*, Thread*) + 124
35 libjvm.dylib 0x000000010a3612d3 JavaThread::thread_main_inner() + 155
36 libjvm.dylib 0x000000010a3629ce JavaThread::run() + 450
37 libjvm.dylib 0x000000010a28343a java_start(Thread*) + 246
38 libsystem_pthread.dylib 0x00007fff98e6d99d _pthread_body + 131
39 libsystem_pthread.dylib 0x00007fff98e6d91a _pthread_start + 168
40 libsystem_pthread.dylib 0x00007fff98e6b351 thread_start + 13
Edit:
I found that if I comment out all my .delete() calls (SWIG generated method) in my Java code, then I get a different error:
# SIGILL (0x4) at pc=0x00007fff95c7379f, pid=85146, tid=80651
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C [libsystem_c.dylib+0x5e79f] __abort+0xb1
and SIGILL is a new one to me...
When building the JSGFDemo using ant, everything works fine. Running the JSGFDemo.jar build artifact works without any errors. However, when using source folder imported in eclipse, and adding the jars in the lib/ directory to the build path, the program errors with the following message:
Problem configuring recognizerProperty exception component:'jsgfGrammar' property:'grammarLocation' - Can't locate resource:/edu/cmu/sphinx/demo/jsapi/jsgf/
edu.cmu.sphinx.util.props.InternalConfigurationException: Can't locate resource:/edu/cmu/sphinx/demo/jsapi/jsgf/
For some reason the call to ConfigurationManagerUtils.class.getResource(resourceName); in ConfigurationManagerUtils.resourceToURL(String location) seemingly returns different results for location = "resource:/edu/cmu/sphinx/demo/jsapi/jsgf/". (null, or a valid URL-object)
As a sidenote, I thought getResource("/path/to/a/dir/not/file/"); was invalid when it would resolve to a path inside a jar.
I've been banging my head against this for a while now and can't see what I'm doing wrong.
I believe to have found the issue. By default, Eclipse seems to construct the jar differently, leaving out entries for directories.
Investigating the archives with unzip -v reveals some interesting details.
File from building with Ant:
Archive: JSGFDemo.jar
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
0 Stored 0 0% 2013-03-31 03:13 00000000 META-INF/
284 Defl:N 210 26% 2013-03-31 03:13 ddd976ff META-INF/MANIFEST.MF
0 Stored 0 0% 2013-03-31 03:08 00000000 edu/
0 Stored 0 0% 2013-03-31 03:08 00000000 edu/cmu/
0 Stored 0 0% 2013-03-31 03:13 00000000 edu/cmu/sphinx/
0 Stored 0 0% 2013-03-31 03:12 00000000 edu/cmu/sphinx/demo/
0 Stored 0 0% 2013-03-31 03:13 00000000 edu/cmu/sphinx/demo/jsapi/
0 Stored 0 0% 2013-03-31 03:13 00000000 edu/cmu/sphinx/demo/jsapi/jsgf/
7391 Defl:N 3501 53% 2013-03-31 03:13 938438dd edu/cmu/sphinx/demo/jsapi/jsgf/JSGFDemo.class
798 Defl:N 326 59% 2013-03-31 03:13 647722fc edu/cmu/sphinx/demo/jsapi/jsgf/books.gram
204 Defl:N 140 31% 2013-03-31 03:13 789bb514 edu/cmu/sphinx/demo/jsapi/jsgf/commands.gram
9295 Defl:N 1500 84% 2013-03-31 03:13 3b519044 edu/cmu/sphinx/demo/jsapi/jsgf/jsgf.config.xml
1589 Defl:N 473 70% 2013-03-31 03:13 60075af0 edu/cmu/sphinx/demo/jsapi/jsgf/movies.gram
299 Defl:N 195 35% 2013-03-31 03:13 42e94d32 edu/cmu/sphinx/demo/jsapi/jsgf/music.gram
666 Defl:N 288 57% 2013-03-31 03:13 ca4b72f9 edu/cmu/sphinx/demo/jsapi/jsgf/news.gram
-------- ------- --- -------
20526 6633 68% 15 files
Jar exported using eclipse:
Archive: JSGFDemo-eclipse.jar
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
180 Defl:N 134 26% 2013-03-31 23:35 1e681d3b META-INF/MANIFEST.MF
7338 Defl:N 3537 52% 2013-03-31 23:29 ed8c4c3f edu/cmu/sphinx/demo/jsapi/jsgf/JSGFDemo.class
798 Defl:N 326 59% 2013-03-31 13:21 647722fc edu/cmu/sphinx/demo/jsapi/jsgf/books.gram
204 Defl:N 140 31% 2013-03-31 13:21 789bb514 edu/cmu/sphinx/demo/jsapi/jsgf/commands.gram
9295 Defl:N 1500 84% 2013-03-31 13:21 3b519044 edu/cmu/sphinx/demo/jsapi/jsgf/jsgf.config.xml
1589 Defl:N 473 70% 2013-03-31 13:21 60075af0 edu/cmu/sphinx/demo/jsapi/jsgf/movies.gram
299 Defl:N 195 35% 2013-03-31 13:21 42e94d32 edu/cmu/sphinx/demo/jsapi/jsgf/music.gram
666 Defl:N 288 57% 2013-03-31 13:21 ca4b72f9 edu/cmu/sphinx/demo/jsapi/jsgf/news.gram
-------- ------- --- -------
20369 6593 68% 8 files
After a quick google, I found the "Add directory entries" option in Eclipse's jar export wizard.
I am running a Qt application on embedded Linux platform. The system has 128 MB RAM, 512MB NAND, no swap. The application uses a custom library for the peripherals, the rest are all Qt and c/c++ libs. The application uses SQLITE3 as well.
After 2-3 hours, the machine starts running very slow, shell commands take 10 or so seconds to respond. Eventually the machine hangs, and finally OOM killer kills the application, and the system starts behaving at normal speed.
After some system memory observations using top command reveals that while application is running, the system free memory is decreasing, while slab keeps on increasing. These are the snaps of top given below. The application is named xyz.
At Application start :
Mem total:126164 anon:3308 map:8436 free:32456
slab:60936 buf:0 cache:27528 dirty:0 write:0
Swap total:0 free:0
PID VSZ VSZRW^ RSS (SHR) DIRTY (SHR) STACK COMMAND
776 29080 9228 8036 528 968 0 84 ./xyz -qws
781 3960 736 1976 1456 520 0 84 sshd: root#notty
786 3676 680 1208 764 416 0 88 /usr/libexec/sftp-server
770 3792 568 1948 1472 464 0 84 {sshd} sshd: root#pts/0
766 3792 568 956 688 252 0 84 /usr/sbin/sshd
388 1864 284 552 332 188 0 84 udevd --daemon
789 2832 272 688 584 84 0 84 top
774 2828 268 668 560 84 0 84 -sh
709 2896 268 556 464 80 0 84 /usr/sbin/inetd
747 2828 268 596 516 68 0 84 /sbin/getty -L ttymxc0 115200 vt100
777 2824 264 444 368 68 0 84 tee out.log
785 2824 264 484 416 68 0 84 sh -c /usr/libexec/sftp-server
1 2824 264 556 488 64 0 84 init
After some time :
Mem total:126164 anon:3312 map:8440 free:9244
slab:83976 buf:0 cache:27584 dirty:0 write:0
Swap total:0 free:0
PID VSZ VSZRW^ RSS (SHR) DIRTY (SHR) STACK COMMAND
776 29080 9228 8044 528 972 0 84 ./xyz -qws
781 3960 736 1976 1456 520 0 84 sshd: root#notty
786 3676 680 1208 764 416 0 88 /usr/libexec/sftp-server
770 3792 568 1948 1472 464 0 84 {sshd} sshd: root#pts/0
766 3792 568 956 688 252 0 84 /usr/sbin/sshd
388 1864 284 552 332 188 0 84 udevd --daemon
789 2832 272 688 584 84 0 84 top
774 2828 268 668 560 84 0 84 -sh
709 2896 268 556 464 80 0 84 /usr/sbin/inetd
747 2828 268 596 516 68 0 84 /sbin/getty -L ttymxc0 115200 vt100
777 2824 264 444 368 68 0 84 tee out.log
785 2824 264 484 416 68 0 84 sh -c /usr/libexec/sftp-server
1 2824 264 556 488 64 0 84 init
Funnily though, I can not see any major changes in the output of top involving the application itself. Eventually the application is killed, top output after that :
Mem total:126164 anon:2356 map:916 free:2368
slab:117944 buf:0 cache:1580 dirty:0 write:0
Swap total:0 free:0
PID VSZ VSZRW^ RSS (SHR) DIRTY (SHR) STACK COMMAND
781 3960 736 708 184 520 0 84 sshd: root#notty
786 3724 728 736 172 484 0 88 /usr/libexec/sftp-server
770 3792 568 648 188 460 0 84 {sshd} sshd: root#pts/0
766 3792 568 252 0 252 0 84 /usr/sbin/sshd
388 1864 284 188 0 188 0 84 udevd --daemon
819 2832 272 676 348 84 0 84 top
774 2828 268 512 324 96 0 84 -sh
709 2896 268 80 0 80 0 84 /usr/sbin/inetd
747 2828 268 68 0 68 0 84 /sbin/getty -L ttymxc0 115200 vt100
785 2824 264 68 0 68 0 84 sh -c /usr/libexec/sftp-server
1 2824 264 64 0 64 0 84 init
The dmesg shows :
sh invoked oom-killer: gfp_mask=0xd0, order=2, oomkilladj=0
[<c002d4c4>] (unwind_backtrace+0x0/0xd4) from [<c0073ac0>] (oom_kill_process+0x54/0x1b8)
[<c0073ac0>] (oom_kill_process+0x54/0x1b8) from [<c0073f14>] (__out_of_memory+0x154/0x178)
[<c0073f14>] (__out_of_memory+0x154/0x178) from [<c0073fa0>] (out_of_memory+0x68/0x9c)
[<c0073fa0>] (out_of_memory+0x68/0x9c) from [<c007649c>] (__alloc_pages_nodemask+0x3e0/0x4c8)
[<c007649c>] (__alloc_pages_nodemask+0x3e0/0x4c8) from [<c0076598>] (__get_free_pages+0x14/0x4c)
[<c0076598>] (__get_free_pages+0x14/0x4c) from [<c002f528>] (get_pgd_slow+0x14/0xdc)
[<c002f528>] (get_pgd_slow+0x14/0xdc) from [<c0043890>] (mm_init+0x84/0xc4)
[<c0043890>] (mm_init+0x84/0xc4) from [<c0097b94>] (bprm_mm_init+0x10/0x138)
[<c0097b94>] (bprm_mm_init+0x10/0x138) from [<c00980a8>] (do_execve+0xf4/0x2a8)
[<c00980a8>] (do_execve+0xf4/0x2a8) from [<c002afc4>] (sys_execve+0x38/0x5c)
[<c002afc4>] (sys_execve+0x38/0x5c) from [<c0027d20>] (ret_fast_syscall+0x0/0x2c)
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
Active_anon:424 active_file:11 inactive_anon:428
inactive_file:3 unevictable:0 dirty:0 writeback:0 unstable:0
free:608 slab:29498 mapped:14 pagetables:59 bounce:0
DMA free:692kB min:268kB low:332kB high:400kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:0kB present:24384kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 103 103
Normal free:1740kB min:1168kB low:1460kB high:1752kB active_anon:1696kB inactive_anon:1712kB active_file:40kB inactive_file:12kB unevictable:0kB present:105664kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 3*4kB 3*8kB 5*16kB 2*32kB 4*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 692kB
Normal: 377*4kB 1*8kB 4*16kB 1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1740kB
30 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
32768 pages of RAM
687 free pages
1306 reserved pages
29498 slab pages
59 pages shared
0 pages swap cached
Out of memory: kill process 774 (sh) score 339 or a child
Killed process 776 (xyz)
So it's obvious that there is a memory leak, it must be my app since my app is killed. But I am not doing any malloc s from the program. I have taken care as to limit the scope of variables so that they are deallocated after they are used. So I am at a complete loss as to why is slab increasing in the top output. I have tried http://valgrind.org/docs/manual/faq.html#faq.reports but didn't work.
Currently trying to use Valgrind on desktop (since I have read it only works for arm-cortex) to check my business logic.
Addittional info :
root#freescale ~/Application/app$ uname -a
Linux freescale 2.6.31-207-g7286c01 #2053 Fri Jun 22 10:29:11 IST 2012 armv5tejl GNU/Linux
Compiler : arm-none-linux-gnueabi-4.1.2 glibc2.5
cpp libs : libstdc++.so.6.0.8
Qt : 4.7.3 libs
Any pointers would be greatly appreciated...
I don't think the problem is directly in your code.
The reason is obvious: your application space does not increase (both RSS and VSW do not increase).
However, you do see the number of slabs increasing. You cannot use or increase the number of slabs from your application - it's a kernel-only thingie.
Some obvious causes of slab size increase from the top of my head:
you never really close network sockets
you read many files, but never close them
you use many ioctls
I would run strace and look at its output for a while. strace intercepts interactions with the kernel. If you have memory issues, I'd expect repeated calls to brk(). If you have other issues, you'll see repeated calls to open without close.
If you have some data structure allocation, check for the correctness of adding children and etc.. I had similar bug in my code. Also if you make big and large queries to the database it may use more ram memory. Try to find some memory leak detector to find if there is any leak.