When building the JSGFDemo using ant, everything works fine. Running the JSGFDemo.jar build artifact works without any errors. However, when using source folder imported in eclipse, and adding the jars in the lib/ directory to the build path, the program errors with the following message:
Problem configuring recognizerProperty exception component:'jsgfGrammar' property:'grammarLocation' - Can't locate resource:/edu/cmu/sphinx/demo/jsapi/jsgf/
edu.cmu.sphinx.util.props.InternalConfigurationException: Can't locate resource:/edu/cmu/sphinx/demo/jsapi/jsgf/
For some reason the call to ConfigurationManagerUtils.class.getResource(resourceName); in ConfigurationManagerUtils.resourceToURL(String location) seemingly returns different results for location = "resource:/edu/cmu/sphinx/demo/jsapi/jsgf/". (null, or a valid URL-object)
As a sidenote, I thought getResource("/path/to/a/dir/not/file/"); was invalid when it would resolve to a path inside a jar.
I've been banging my head against this for a while now and can't see what I'm doing wrong.
I believe to have found the issue. By default, Eclipse seems to construct the jar differently, leaving out entries for directories.
Investigating the archives with unzip -v reveals some interesting details.
File from building with Ant:
Archive: JSGFDemo.jar
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
0 Stored 0 0% 2013-03-31 03:13 00000000 META-INF/
284 Defl:N 210 26% 2013-03-31 03:13 ddd976ff META-INF/MANIFEST.MF
0 Stored 0 0% 2013-03-31 03:08 00000000 edu/
0 Stored 0 0% 2013-03-31 03:08 00000000 edu/cmu/
0 Stored 0 0% 2013-03-31 03:13 00000000 edu/cmu/sphinx/
0 Stored 0 0% 2013-03-31 03:12 00000000 edu/cmu/sphinx/demo/
0 Stored 0 0% 2013-03-31 03:13 00000000 edu/cmu/sphinx/demo/jsapi/
0 Stored 0 0% 2013-03-31 03:13 00000000 edu/cmu/sphinx/demo/jsapi/jsgf/
7391 Defl:N 3501 53% 2013-03-31 03:13 938438dd edu/cmu/sphinx/demo/jsapi/jsgf/JSGFDemo.class
798 Defl:N 326 59% 2013-03-31 03:13 647722fc edu/cmu/sphinx/demo/jsapi/jsgf/books.gram
204 Defl:N 140 31% 2013-03-31 03:13 789bb514 edu/cmu/sphinx/demo/jsapi/jsgf/commands.gram
9295 Defl:N 1500 84% 2013-03-31 03:13 3b519044 edu/cmu/sphinx/demo/jsapi/jsgf/jsgf.config.xml
1589 Defl:N 473 70% 2013-03-31 03:13 60075af0 edu/cmu/sphinx/demo/jsapi/jsgf/movies.gram
299 Defl:N 195 35% 2013-03-31 03:13 42e94d32 edu/cmu/sphinx/demo/jsapi/jsgf/music.gram
666 Defl:N 288 57% 2013-03-31 03:13 ca4b72f9 edu/cmu/sphinx/demo/jsapi/jsgf/news.gram
-------- ------- --- -------
20526 6633 68% 15 files
Jar exported using eclipse:
Archive: JSGFDemo-eclipse.jar
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
180 Defl:N 134 26% 2013-03-31 23:35 1e681d3b META-INF/MANIFEST.MF
7338 Defl:N 3537 52% 2013-03-31 23:29 ed8c4c3f edu/cmu/sphinx/demo/jsapi/jsgf/JSGFDemo.class
798 Defl:N 326 59% 2013-03-31 13:21 647722fc edu/cmu/sphinx/demo/jsapi/jsgf/books.gram
204 Defl:N 140 31% 2013-03-31 13:21 789bb514 edu/cmu/sphinx/demo/jsapi/jsgf/commands.gram
9295 Defl:N 1500 84% 2013-03-31 13:21 3b519044 edu/cmu/sphinx/demo/jsapi/jsgf/jsgf.config.xml
1589 Defl:N 473 70% 2013-03-31 13:21 60075af0 edu/cmu/sphinx/demo/jsapi/jsgf/movies.gram
299 Defl:N 195 35% 2013-03-31 13:21 42e94d32 edu/cmu/sphinx/demo/jsapi/jsgf/music.gram
666 Defl:N 288 57% 2013-03-31 13:21 ca4b72f9 edu/cmu/sphinx/demo/jsapi/jsgf/news.gram
-------- ------- --- -------
20369 6593 68% 8 files
After a quick google, I found the "Add directory entries" option in Eclipse's jar export wizard.
Related
If I have a C or CPP program written like
#include <stdio.h>
int bss_section_object[10];
int main()
{
return 0;
}
After compiling the code with gcc/g++,
$g++ -save-temps aa.cpp
If I check the size of generated object and executable files
$ls -ltr
-rw-rw-r-- 1 dinesh dinesh 77 Feb 14 12:42 aa.cpp
-rw-rw-r-- 1 dinesh dinesh 17283 Feb 14 12:45 aa.ii
-rw-rw-r-- 1 dinesh dinesh 517 Feb 14 12:45 aa.s
-rw-rw-r-- 1 dinesh dinesh 1296 Feb 14 12:45 aa.o
-rwxrwxr-x 1 dinesh dinesh 8592 Feb 14 12:45 a.out
$size a.out
text data bss dec hex filename
1099 544 72 1715 6b3 a.out
$size aa.o
text data bss dec hex filename
67 0 40 107 6b aa.o
Now if I comment the object created on .bss section
#include <stdio.h>
//int bss_section_object[10];
int main()
{
return 0;
}
and check the size after compling it:-
$ls -ltr
-rw-rw-r-- 1 dinesh dinesh 79 Feb 14 12:46 aa.cpp
-rw-rw-r-- 1 dinesh dinesh 17256 Feb 14 12:46 aa.ii
-rw-rw-r-- 1 dinesh dinesh 378 Feb 14 12:46 aa.s
-rw-rw-r-- 1 dinesh dinesh 1232 Feb 14 12:46 aa.o
-rwxrwxr-x 1 dinesh dinesh 8552 Feb 14 12:46 a.out
$size a.out
text data bss dec hex filename
1099 544 8 1651 673 a.out
$size aa.o
text data bss dec hex filename
67 0 0 67 43 aa.o
I can see that their size varies based on "bss_section_object" size.
If I read the details of .bss section, then as per wikipedia:-
"The size that BSS will require at runtime is recorded in the object file,
but BSS (unlike the data segment) doesn't take up any actual space in the
object file".
We can see similar description of .bss section in other computer science
books as well.
Now, If the .bss section doesn't take up any space in object file, then why
is the size of object as well as executable file changing based on size of
object created on .bss section?
Can someone please explain the details about it?
You are perfectly right when saying :
"The size that BSS will require at runtime is recorded in the object file, but BSS (unlike the data segment) doesn't take up any actual space in the object file"
Because uninitialized data is placed in the .bss section of the program, which is of type SHT_NOBITS (see the elf(5) manpage).
To convince yourself that this section doesn't takes up any space in the ELF file, try readelf -Wl /bin/ls and look at the fourth segment :
[...]
LOAD 0x01a310 0x000000000061a310 0x000000000061a310 0x0012f0 0x002010 RW 0x200000
[...]
`03 .init_array .fini_array .jcr .data.rel.ro .dynamic .got .got.plt .data .bss
You can see that it contains (among others), the .bss section, and has file size 0x12f0 but memory size 0x2010. The difference is the size of the .bss section.
This section is not included in the file because the associated memory region will be zeroed out at program startup, and it would be unnecessary to store zeros in the file.
You're right, but your test is wrong
In the example you give, given that you disable optimization and debugger symbols, the difference you see in file size may be explained by the symbol table, and not by the change of the .bss section size.
Here's an experiment you can reproduce :
you$ cat main.cpp
#include <stdio.h>
#ifdef USE_BSS
int bss_section_object[10];
#endif
int main()
{
return 0;
}
you$ g++ -O0 main.cpp -o without.elf
you$ g++ -DUSE_BSS -O0 main.cpp -o with.elf
you$ ls -l *.elf
-rwxr-xr-x. 1 you you 8505 Feb 14 10:00 with.elf
-rwxr-xr-x. 1 you you 8462 Feb 14 10:00 without.elf
At this point, you may believe that changing the .bss section size affects the file size, but in fact the difference is due to the additional symbol bss_section_object in the symbol table (which you can remove using the strip command) :
you$ readelf -WS with.elf > with && readelf -WS without.elf > without
you$ diff with without
30c30
< [25] .bss NOBITS 0000000000601040 00102c 000048 00 WA 0 0 32
---
> [25] .bss NOBITS 000000000060102c 00102c 000004 00 WA 0 0 4
33,34c33,34
< [28] .symtab SYMTAB 0000000000000000 0018e8 000618 18 29 45 8
< [29] .strtab STRTAB 0000000000000000 001f00 000239 00 0 0 1
---
> [28] .symtab SYMTAB 0000000000000000 0018e8 000600 18 29 45 8
> [29] .strtab STRTAB 0000000000000000 001ee8 000226 00 0 0 1
# Yep, the .symtab and .strtab are also different
you$ strip *.elf
you$ ls -l *.elf
-rwxr-xr-x. 1 you you 6232 Feb 14 10:06 with.elf
-rwxr-xr-x. 1 you you 6232 Feb 14 10:06 without.elf
Yay ! By removing the symbol table section .symtab (it its associated string table .strtab), you end up with the same file size, but different .bss section sizes, which is what you expected.
I am trying do a ARIMA model estimation for 5 different variables. The data consists of 16 months of Point of Sales. How do I approach this complicated ARIMA modelling?
Furthermore I would like to do:
A simple moving average of each product group
A Holt-Winters
exponential smoothing model
Data is as follows with date and product groups:
Date Gloves ShoeCovers Socks Warmers HeadWear
apr-14 11015 3827 3465 1264 772
maj-14 11087 2776 4378 1099 1423
jun-14 7645 1432 4490 674 670
jul-14 10083 7975 2577 1558 8501
aug-14 13887 8577 6854 1305 15621
sep-14 9186 5213 5244 1183 6784
okt-14 7611 4279 4150 977 6191
nov-14 6410 4033 2918 507 8276
dec-14 4856 3552 3192 450 4810
jan-15 17506 7274 3137 2216 3979
feb-15 21518 5672 8848 1838 2321
mar-15 17395 5200 5712 1604 2282
apr-15 11405 4531 5185 1479 1888
maj-15 11509 5690 4370 1145 2369
jun-15 9945 2610 4884 882 1709
jul-15 8707 5658 4570 1948 6255
Any skilled forecasters out there willing to help? Much appreciated!
I am experimenting with launching rsync from QProcess and although it runs, it behaves differently when run from QProcess compared to running the exact same command from the command line.
Here is the command and stdout when run from QProcess
/usr/bin/rsync -atv --stats --progress --port=873 --compress-level=9 --recursive --delete --exclude="/etc/*.conf" --exclude="A*" rsync://myhost.com/haast/tmp/mysync/* /tmp/mysync/
receiving incremental file list
created directory /tmp/mysync
A
0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=6/7)
B
0 100% 0.00kB/s 0:00:00 (xfer#2, to-check=5/7)
test.conf
0 100% 0.00kB/s 0:00:00 (xfer#3, to-check=4/7)
subdir/
subdir/A2
0 100% 0.00kB/s 0:00:00 (xfer#4, to-check=2/7)
subdir/C
0 100% 0.00kB/s 0:00:00 (xfer#5, to-check=1/7)
subdir/D
0 100% 0.00kB/s 0:00:00 (xfer#6, to-check=0/7)
Number of files: 7
Number of files transferred: 6
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 105
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 174
Total bytes received: 367
sent 174 bytes received 367 bytes 360.67 bytes/sec
total size is 0 speedup is 0.00
Notice that although I excluded 'A*', it still copied them! Now running the exact same command from the command line:
/usr/bin/rsync -atv --stats --progress --port=873 --compress-level=9 --recursive --delete --exclude="/etc/*.conf" --exclude="A*" rsync://myhost.com/haast/tmp/mysync/* /tmp/mysync/
receiving incremental file list
created directory /tmp/mysync
B
0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=4/5)
test.conf
0 100% 0.00kB/s 0:00:00 (xfer#2, to-check=3/5)
subdir/
subdir/C
0 100% 0.00kB/s 0:00:00 (xfer#3, to-check=1/5)
subdir/D
0 100% 0.00kB/s 0:00:00 (xfer#4, to-check=0/5)
Number of files: 5
Number of files transferred: 4
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 83
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 132
Total bytes received: 273
sent 132 bytes received 273 bytes 270.00 bytes/sec
total size is 0 speedup is 0.00
Notice that now the 'A*' exclude is respected! Can someone explain why they are performing differently?
A noticed that if I removed the quotes surrounding the excludes, then the QProcess run performs correctly.
In your command-line execution, bash interpreter performs a previous substitution and remove quotes, so they are not passed to rsync arg list.
Next script shows how bash substitution is performed:
[tmp]$ cat printargs.sh
#!/bin/bash
echo $*
[tmp]$ ./printargs.sh --exclude="A*"
--exclude=A*
Assume a 3Kb file that looks like this:
PdId1 Unit 1
Model 3244
Status: OK
Advanced Status OK
-----------------------
No errors found
Statistics...
...<arbitrary length values here>...
PdId2 Unit 1
Model 3222
Status: OK
Advanced Status OK
-----------------------
Error Log is as follows <arbitrary values here>
PdId3 Unit 1
Model 3243
Status: OK
Advanced Status OK
-----------------------
No errors found
So we can be certain that PdIdn can reliably used as a delimiter, that it's always at the start of a line and that it's always trailing a numebr. I want to parse the text between the delimiter for "No errors found" and if the string is missing, grab the delimiter and the next four lines (grep -A4), glue on an error message and echo the result.
I've been wracking my brain about how to approach this. I'm most comfortable in Bash with grep, but I don't think grep's going to cut it here. I've looked at using split to break the file into pieces, but this seems messy and hard to clean up after processing is done. I started to try to write something in awk / sed, but I don't understand how to split on the delimiters, then go back and parse each result, then break off the next piece and parse that.
I apologise for the general nature of this question, but I'm stumped and could use some guidance.
Edit: Technically, PdId isn't a delimiter as much as it's the start of the next record. The number of records is arbitrary.
Edit: We've now got real world data to work with:
-------------------------------------------------------------------------------
PdId: 1
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: SMART Off-line Data Collection executing in background (4)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 27/31 Celsius
Lifetime Min/Max Temperature: 16/41 Celsius
Under/Over Temperature Limit Count: 0/0
Self-test execution status: ( 0) The previous self-test routine
completed without error or no self-test
has ever been run.
has ever been run.
Error logging capability: (0x01) Error logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 251) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number: 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Error Log Version: 1
No Errors Logged
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
==============================================================================
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
==============================================================================
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always
- 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline
- 71
3 Spin_Up_Time 0x0007 169 169 024 Pre-fail Always
- 245 (Average 204)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always
- 746
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always
- 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always
- 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline
- 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always
- 1181
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 529
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 751
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always
- 751
194 Temperature_Celsius 0x0002 193 193 000 Old_age Always
- 31 (Lifetime Min/Max 16/41)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always
- 0
-------------------------------------------------------------------------------
PdId: 2
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: SMART Off-line Data Collection executing in background (4)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 27/31 Celsius
Lifetime Min/Max Temperature: 16/40 Celsius
Under/Over Temperature Limit Count: 0/0
Self-test execution status: ( 0) The previous self-test routine
completed without error or no self-test
has ever been run.
has ever been run.
Error logging capability: (0x01) Error logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 246) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number: 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Error Log Version: 1
No Errors Logged
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
==============================================================================
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
==============================================================================
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always
- 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline
- 72
3 Spin_Up_Time 0x0007 171 171 024 Pre-fail Always
- 243 (Average 201)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always
- 746
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always
- 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always
- 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline
- 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always
- 1181
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 529
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 749
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always
- 749
194 Temperature_Celsius 0x0002 193 193 000 Old_age Always
- 31 (Lifetime Min/Max 16/40)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always
- 0
-------------------------------------------------------------------------------
PdId: 3
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: SMART Off-line Data Collection executing in background (4)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 27/31 Celsius
Lifetime Min/Max Temperature: 17/40 Celsius
Under/Over Temperature Limit Count: 0/0
Self-test execution status: ( 0) The previous self-test routine
completed without error or no self-test
has ever been run.
has ever been run.
Error logging capability: (0x01) Error logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 241) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number: 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Error Log Version: 1
No Errors Logged
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
==============================================================================
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
==============================================================================
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always
- 0
2 Throughput_Performance 0x0005 140 140 054 Pre-fail Offline
- 67
3 Spin_Up_Time 0x0007 170 170 024 Pre-fail Always
- 234 (Average 213)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always
- 748
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always
- 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always
- 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline
- 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always
- 1188
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 529
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 750
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always
- 750
194 Temperature_Celsius 0x0002 193 193 000 Old_age Always
- 31 (Lifetime Min/Max 17/40)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always
- 0
-------------------------------------------------------------------------------
PdId: 4
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: SMART Off-line Data Collection executing in background (4)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 27/31 Celsius
Lifetime Min/Max Temperature: 15/40 Celsius
Under/Over Temperature Limit Count: 0/0
Self-test execution status: ( 0) The previous self-test routine
completed without error or no self-test
has ever been run.
has ever been run.
Error logging capability: (0x01) Error logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 254) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number: 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Error Log Version: 1
ATA Error Count: 165 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 165 occurred at disk power-on lifetime: 1176 hours (49 days + 0 hours)
When the command that caused the error occurred,
the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 50 b0 ee 81 0d
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 80 a8 80 ee 81 40 00 18:38:48.276 WRITE FPDMA QUEUED
61 80 a0 00 ee 81 40 00 18:38:48.276 WRITE FPDMA QUEUED
61 80 98 80 ed 81 40 00 18:38:48.276 WRITE FPDMA QUEUED
61 80 90 00 ed 81 40 00 18:38:48.276 WRITE FPDMA QUEUED
61 80 88 80 ec 81 40 00 18:38:48.275 WRITE FPDMA QUEUED
Error 164 occurred at disk power-on lifetime: 1175 hours (48 days + 23 hours)
When the command that caused the error occurred,
the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 10 f0 ad 6b 0d Error: ICRC, ABRT 16 sectors at LBA = 0x0d6badf0 = 225160688
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
35 00 80 80 ad 6b 40 00 18:36:07.145 WRITE DMA EXT
35 00 80 00 ae 6b 40 00 18:36:07.144 WRITE DMA EXT
35 00 80 00 ad 6b 40 00 18:36:07.144 WRITE DMA EXT
35 00 80 80 ab 6b 40 00 18:36:07.139 WRITE DMA EXT
35 00 80 00 ab 6b 40 00 18:36:07.139 WRITE DMA EXT
Error 163 occurred at disk power-on lifetime: 1175 hours (48 days + 23 hours)
When the command that caused the error occurred,
the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 f0 10 5e 5d 0d Error: ICRC, ABRT 240 sectors at LBA = 0x0d5d5e10 = 224222736
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
35 00 80 80 5b 5d 40 00 18:35:47.982 WRITE DMA EXT
35 00 80 80 5a 5d 40 00 18:35:47.982 WRITE DMA EXT
35 00 80 00 59 5d 40 00 18:35:47.981 WRITE DMA EXT
35 00 00 00 58 5d 40 00 18:35:47.979 WRITE DMA EXT
35 00 30 00 36 5d 40 00 18:35:47.960 WRITE DMA EXT
Error 162 occurred at disk power-on lifetime: 1175 hours (48 days + 23 hours)
When the command that caused the error occurred,
the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 20 e0 33 19 0d
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 80 30 00 33 19 40 00 18:34:50.672 WRITE FPDMA QUEUED
61 80 28 80 33 19 40 00 18:34:50.671 WRITE FPDMA QUEUED
61 80 20 00 34 19 40 00 18:34:50.671 WRITE FPDMA QUEUED
61 00 18 80 34 19 40 00 18:34:50.671 WRITE FPDMA QUEUED
61 80 10 80 36 19 40 00 18:34:50.670 WRITE FPDMA QUEUED
Error 161 occurred at disk power-on lifetime: 1133 hours (47 days + 5 hours)
When the command that caused the error occurred,
the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 d0 30 dd 3b 0a
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 80 38 80 dc 3b 40 00 06:26:51.414 WRITE FPDMA QUEUED
61 80 30 00 df 3b 40 00 06:26:51.413 WRITE FPDMA QUEUED
61 80 28 80 df 3b 40 00 06:26:51.413 WRITE FPDMA QUEUED
61 80 20 00 da 3b 40 00 06:26:51.402 WRITE FPDMA QUEUED
61 80 18 80 da 3b 40 00 06:26:51.402 WRITE FPDMA QUEUED
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
==============================================================================
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
==============================================================================
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always
- 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline
- 73
3 Spin_Up_Time 0x0007 170 170 024 Pre-fail Always
- 234 (Average 212)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always
- 747
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always
- 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always
- 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline
- 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always
- 1187
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 529
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 748
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always
- 748
194 Temperature_Celsius 0x0002 200 200 000 Old_age Always
- 30 (Lifetime Min/Max 15/40)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always
- 165
-------------------------------------------------------------------------------
PdId: 5
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: SMART Off-line Data Collection executing in background (4)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 27/31 Celsius
Lifetime Min/Max Temperature: 17/40 Celsius
Under/Over Temperature Limit Count: 0/0
Self-test execution status: ( 0) The previous self-test routine
completed without error or no self-test
has ever been run.
has ever been run.
Error logging capability: (0x01) Error logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 251) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number: 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Error Log Version: 1
No Errors Logged
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
==============================================================================
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
==============================================================================
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always
- 0
2 Throughput_Performance 0x0005 140 140 054 Pre-fail Offline
- 68
3 Spin_Up_Time 0x0007 133 133 024 Pre-fail Always
- 289 (Average 282)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always
- 748
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always
- 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always
- 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline
- 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always
- 1186
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 529
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 750
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always
- 750
194 Temperature_Celsius 0x0002 193 193 000 Old_age Always
- 31 (Lifetime Min/Max 17/40)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always
- 0
-------------------------------------------------------------------------------
PdId: 6
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: SMART Off-line Data Collection executing in background (4)
Current Temperature: 30 Celsius
Power Cycle Min/Max Temperature: 27/30 Celsius
Lifetime Min/Max Temperature: 17/40 Celsius
Under/Over Temperature Limit Count: 0/0
Self-test execution status: ( 0) The previous self-test routine
completed without error or no self-test
has ever been run.
has ever been run.
Error logging capability: (0x01) Error logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 243) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number: 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Error Log Version: 1
No Errors Logged
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
==============================================================================
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
==============================================================================
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always
- 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline
- 72
3 Spin_Up_Time 0x0007 130 130 024 Pre-fail Always
- 294 (Average 287)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always
- 748
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always
- 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always
- 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline
- 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always
- 1186
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 529
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 751
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always
- 751
194 Temperature_Celsius 0x0002 200 200 000 Old_age Always
- 30 (Lifetime Min/Max 17/40)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always
- 0
Read line by line, collect into an accumulator the lines you want, print the accumulated lines when you see the trigger message (otherwise just start over and overwrite the accumulator when you see the start of the next record).
We use a as the accumulator and the helper variable n to keep track of how many lines to accumulate:
awk '/^PdId: [1-9][0-9]*/ { a=$0; n=4; next }
n { --n; a=a "\n" $ 0; next }
/No Errors Logged/ { print a }' file
Put the following into an executable awk file:
#!/usr/bin/awk -f
BEGIN {no_errs=1}
c > 0 {a[c++]=$0}
/^----------/ {
logAnyErrors()
ata_err=""
no_errs=0
c=1
delete a
}
/^No Errors Logged/ {no_errs=1}
/^ATA Error Count:/ {ata_err=$0}
function logAnyErrors() {
if( ata_err!="" || !no_errs) {
for(i=1;i<=5;i++) print a[i]
if( ata_err!="" ) print ata_err
print "--" # separator
}
}
END { logAnyErrors() }
Your data actually has a delimiter of "^------------------"... before each PdId.
The breakdown:
Start off assuming no errors in the BEGIN block
add each record line to an array called a with a line counter c
Whenever a new record section occurs call logAnyErrors() and reset counters
In logAnyErrors(), if there are ATA or other errors, print the first 5 lines of the record and a delimiter similar to what I think grep -A4 would output.
At the end, log any errors in the final record.
When I put this into an executable file called awko and run like awko data I get the following output:
PdId: 4
Model Number: WD 1000
Drive Type: SATA
SMART Status: Enable
SMART Health Status: OK
ATA Error Count: 165 (device log contains only the most recent five errors)
----------------
It's possible that the delete a line is non-conforming for some awks. Works on my mac. It's not necessary unless you want to print out more information in each block when errors occur(since the first 5 lines will always be overwritten).
I am running a Qt application on embedded Linux platform. The system has 128 MB RAM, 512MB NAND, no swap. The application uses a custom library for the peripherals, the rest are all Qt and c/c++ libs. The application uses SQLITE3 as well.
After 2-3 hours, the machine starts running very slow, shell commands take 10 or so seconds to respond. Eventually the machine hangs, and finally OOM killer kills the application, and the system starts behaving at normal speed.
After some system memory observations using top command reveals that while application is running, the system free memory is decreasing, while slab keeps on increasing. These are the snaps of top given below. The application is named xyz.
At Application start :
Mem total:126164 anon:3308 map:8436 free:32456
slab:60936 buf:0 cache:27528 dirty:0 write:0
Swap total:0 free:0
PID VSZ VSZRW^ RSS (SHR) DIRTY (SHR) STACK COMMAND
776 29080 9228 8036 528 968 0 84 ./xyz -qws
781 3960 736 1976 1456 520 0 84 sshd: root#notty
786 3676 680 1208 764 416 0 88 /usr/libexec/sftp-server
770 3792 568 1948 1472 464 0 84 {sshd} sshd: root#pts/0
766 3792 568 956 688 252 0 84 /usr/sbin/sshd
388 1864 284 552 332 188 0 84 udevd --daemon
789 2832 272 688 584 84 0 84 top
774 2828 268 668 560 84 0 84 -sh
709 2896 268 556 464 80 0 84 /usr/sbin/inetd
747 2828 268 596 516 68 0 84 /sbin/getty -L ttymxc0 115200 vt100
777 2824 264 444 368 68 0 84 tee out.log
785 2824 264 484 416 68 0 84 sh -c /usr/libexec/sftp-server
1 2824 264 556 488 64 0 84 init
After some time :
Mem total:126164 anon:3312 map:8440 free:9244
slab:83976 buf:0 cache:27584 dirty:0 write:0
Swap total:0 free:0
PID VSZ VSZRW^ RSS (SHR) DIRTY (SHR) STACK COMMAND
776 29080 9228 8044 528 972 0 84 ./xyz -qws
781 3960 736 1976 1456 520 0 84 sshd: root#notty
786 3676 680 1208 764 416 0 88 /usr/libexec/sftp-server
770 3792 568 1948 1472 464 0 84 {sshd} sshd: root#pts/0
766 3792 568 956 688 252 0 84 /usr/sbin/sshd
388 1864 284 552 332 188 0 84 udevd --daemon
789 2832 272 688 584 84 0 84 top
774 2828 268 668 560 84 0 84 -sh
709 2896 268 556 464 80 0 84 /usr/sbin/inetd
747 2828 268 596 516 68 0 84 /sbin/getty -L ttymxc0 115200 vt100
777 2824 264 444 368 68 0 84 tee out.log
785 2824 264 484 416 68 0 84 sh -c /usr/libexec/sftp-server
1 2824 264 556 488 64 0 84 init
Funnily though, I can not see any major changes in the output of top involving the application itself. Eventually the application is killed, top output after that :
Mem total:126164 anon:2356 map:916 free:2368
slab:117944 buf:0 cache:1580 dirty:0 write:0
Swap total:0 free:0
PID VSZ VSZRW^ RSS (SHR) DIRTY (SHR) STACK COMMAND
781 3960 736 708 184 520 0 84 sshd: root#notty
786 3724 728 736 172 484 0 88 /usr/libexec/sftp-server
770 3792 568 648 188 460 0 84 {sshd} sshd: root#pts/0
766 3792 568 252 0 252 0 84 /usr/sbin/sshd
388 1864 284 188 0 188 0 84 udevd --daemon
819 2832 272 676 348 84 0 84 top
774 2828 268 512 324 96 0 84 -sh
709 2896 268 80 0 80 0 84 /usr/sbin/inetd
747 2828 268 68 0 68 0 84 /sbin/getty -L ttymxc0 115200 vt100
785 2824 264 68 0 68 0 84 sh -c /usr/libexec/sftp-server
1 2824 264 64 0 64 0 84 init
The dmesg shows :
sh invoked oom-killer: gfp_mask=0xd0, order=2, oomkilladj=0
[<c002d4c4>] (unwind_backtrace+0x0/0xd4) from [<c0073ac0>] (oom_kill_process+0x54/0x1b8)
[<c0073ac0>] (oom_kill_process+0x54/0x1b8) from [<c0073f14>] (__out_of_memory+0x154/0x178)
[<c0073f14>] (__out_of_memory+0x154/0x178) from [<c0073fa0>] (out_of_memory+0x68/0x9c)
[<c0073fa0>] (out_of_memory+0x68/0x9c) from [<c007649c>] (__alloc_pages_nodemask+0x3e0/0x4c8)
[<c007649c>] (__alloc_pages_nodemask+0x3e0/0x4c8) from [<c0076598>] (__get_free_pages+0x14/0x4c)
[<c0076598>] (__get_free_pages+0x14/0x4c) from [<c002f528>] (get_pgd_slow+0x14/0xdc)
[<c002f528>] (get_pgd_slow+0x14/0xdc) from [<c0043890>] (mm_init+0x84/0xc4)
[<c0043890>] (mm_init+0x84/0xc4) from [<c0097b94>] (bprm_mm_init+0x10/0x138)
[<c0097b94>] (bprm_mm_init+0x10/0x138) from [<c00980a8>] (do_execve+0xf4/0x2a8)
[<c00980a8>] (do_execve+0xf4/0x2a8) from [<c002afc4>] (sys_execve+0x38/0x5c)
[<c002afc4>] (sys_execve+0x38/0x5c) from [<c0027d20>] (ret_fast_syscall+0x0/0x2c)
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
Active_anon:424 active_file:11 inactive_anon:428
inactive_file:3 unevictable:0 dirty:0 writeback:0 unstable:0
free:608 slab:29498 mapped:14 pagetables:59 bounce:0
DMA free:692kB min:268kB low:332kB high:400kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:0kB present:24384kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 103 103
Normal free:1740kB min:1168kB low:1460kB high:1752kB active_anon:1696kB inactive_anon:1712kB active_file:40kB inactive_file:12kB unevictable:0kB present:105664kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 3*4kB 3*8kB 5*16kB 2*32kB 4*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 692kB
Normal: 377*4kB 1*8kB 4*16kB 1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1740kB
30 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
32768 pages of RAM
687 free pages
1306 reserved pages
29498 slab pages
59 pages shared
0 pages swap cached
Out of memory: kill process 774 (sh) score 339 or a child
Killed process 776 (xyz)
So it's obvious that there is a memory leak, it must be my app since my app is killed. But I am not doing any malloc s from the program. I have taken care as to limit the scope of variables so that they are deallocated after they are used. So I am at a complete loss as to why is slab increasing in the top output. I have tried http://valgrind.org/docs/manual/faq.html#faq.reports but didn't work.
Currently trying to use Valgrind on desktop (since I have read it only works for arm-cortex) to check my business logic.
Addittional info :
root#freescale ~/Application/app$ uname -a
Linux freescale 2.6.31-207-g7286c01 #2053 Fri Jun 22 10:29:11 IST 2012 armv5tejl GNU/Linux
Compiler : arm-none-linux-gnueabi-4.1.2 glibc2.5
cpp libs : libstdc++.so.6.0.8
Qt : 4.7.3 libs
Any pointers would be greatly appreciated...
I don't think the problem is directly in your code.
The reason is obvious: your application space does not increase (both RSS and VSW do not increase).
However, you do see the number of slabs increasing. You cannot use or increase the number of slabs from your application - it's a kernel-only thingie.
Some obvious causes of slab size increase from the top of my head:
you never really close network sockets
you read many files, but never close them
you use many ioctls
I would run strace and look at its output for a while. strace intercepts interactions with the kernel. If you have memory issues, I'd expect repeated calls to brk(). If you have other issues, you'll see repeated calls to open without close.
If you have some data structure allocation, check for the correctness of adding children and etc.. I had similar bug in my code. Also if you make big and large queries to the database it may use more ram memory. Try to find some memory leak detector to find if there is any leak.