I'm trying to find out if I can store values captured at irregular intervals into an RRD.
I have a script which connects to an ActiveMQ server subscribes to a queue or topic and looks at the message header time stamp, compares it with Time.now to give me a delta.
The data I get from my script is as below;
000000.681 Time Delta
000000.793 Time Delta
000000.583 Time Delta
000001.994 Time Delta
The issue I face is that messages from the ActiveMQ don't necessarily come in at a 'regular interval' (e.g 1/sec, 1/2sec) They could come in at peak times as 5 a second, and quite times 1 every 10 seconds.
I'd like to be able to capture the output into an RRD so I can graph against it but having a look around on the internet it's not clear is this can be done, or if I'd be better off using a.n.other database/store to capture the data into.
The eventual output I'd like would be a graph showing the time delta for each message.
It looks like I could set the RRD using --step to 1 second, and the hart beat to 2 seconds having had a read of the docs.
I found a couple of posts here and here which talk about being careful with the intervals and the fact my data might be averaged, smoothed or otherwise messed about with when written to the RRD. But nothing I've found online has a similar usage case to mine so its a bit hard to know where I should be looking. I'd like my data to be stored as point for each message received.
I have a couple of RRD's setup for testing; one is taking the AVERAGE the other is taking the LAST to produce some graphs. My heartbeat is set for 100 seconds, but the interval is set to 1. I'm now getting data which looks correct. I'm also guessing that empty spaces in graph from the LAST RRA are due to my data coming in slower that 1 per second?
I'll post my create code & output as an answer.
rrdtool will always store data at regular intervals. As data is handed over to rrdtool, it first gets re-sampled to the --step interval. and then further consolidated to the intervals setup in the RRAs.
The exact arrival time of the data (to the millisecond) is taken into account as the re-sampling takes place ...
If two data points are further apart than specified by mrhb, the data is considered non-continuous and rrdtool will store 'unknown' for the interval affected.
I ended up making two sets of RRD's to experiment with;
rrdtool create test1.rrd \
--step '1' \
'DS:ds0:GAUGE:5:0:U' \
'RRA:AVERAGE:0.5:1:86400' \
'RRA:MAX:0.5:1:86400' \
'RRA:AVERAGE:0.5:60:10080' \
'RRA:MAX:0.5:60:10080' \
'RRA:AVERAGE:0.5:120:21600' \
'RRA:MAX:0.5:120:21600' \
'RRA:AVERAGE:0.5:300:105120' \
'RRA:MAX:0.5:300:105120'
and
rrdtool create test.rrd \
--step '1' \
'DS:ds0:GAUGE:5:0:U' \
'RRA:AVERAGE:0.5:1:86400' \
'RRA:LAST:0.5:1:86400' \
'RRA:AVERAGE:0.5:60:10080' \
'RRA:LAST:0.5:60:10080' \
'RRA:AVERAGE:0.5:120:21600' \
'RRA:LAST:0.5:120:21600' \
'RRA:AVERAGE:0.5:300:105120' \
'RRA:MAX:0.5:300:105120'
Which allows me to store;
1sec, archive is kept for 1day back
1min, archive is kept for 7day back
2min, archive is kept for 30day back
5min, archive is kept for 1year back
Which makes these nice graphs;
The graphs where made in PHP with the following code;
<?php
$opts = array(
'--width', '600',
'--height', '100',
'--title', 'Avg Time Delta xxxxxxxxxx (Last 1 Hr)',
'--vertical-label', 'Time Delta',
'--watermark', 'xxxxxxxxxx',
'--start', 'end-1h',
'DEF:out=test.rrd:ds0:AVERAGE',
'DEF:max=test.rrd:ds0:MAX',
'AREA:out#9966FF:Avg Time Delta',
'LINE:max#996600:Max Time Delta',
);
$ret = rrd_graph("graphs/1hr-graph.png", $opts);
if( !is_array($ret) )
{
$err = rrd_error();
echo "rrd_graph() ERROR: $err\n";
}
echo '<img src="http://server/graphs/1hr-graph.png">';
echo '<BR>';
?>
<?php
$opts = array(
'--width', '600',
'--height', '100',
'--title', 'Last Time Delta xxxxxxxxxx (Last 1 Hr)',
'--vertical-label', 'Time Delta',
'--watermark', 'xxxxxxxxxx',
'--start', 'end-1h',
'DEF:avg=test1.rrd:ds0:AVERAGE',
'DEF:last=test1.rrd:ds0:LAST',
'AREA:avg#99AAFF:Avg Time Delta',
'LINE:last#99AA00:Last Time Delta',
);
$ret = rrd_graph("graphs/1hr-last.png", $opts);
if( !is_array($ret) )
{
$err = rrd_error();
echo "rrd_graph() ERROR: $err\n";
}
echo '<img src="http://server/graphs/1hr-last.png">'
?>
From my own sanity checking and watching the data in realtime it looks like both of those graphs are correct, but behave in slightly different ways. When the data feed which this is monitoring is quite and I'm only getting 1 mesg every 10 sec I get a lot of gaps in the LAST graphs whereas the AVERAGE graphs are smoothed out to fill the gaps. I also tried with setting another RRD to ABSOLUTE but the graphs for that looks 'wrong' and the times are all below 1.0.
So it looks like I can feed my RRD at whatever interval I like from my script. It looks like the RRD will sample my data by its defined interval (In my case 1 sec) and then do what it needs to do based on the way I save it (Gauge, Absolute etc) With my heart-beat set to 100 I should always receive some data before that 100 sec times-out - thus avoiding NAN entries in my database.
At the moment I can't tell how well behaved this config will be during times of disruption (e.g delayed messages from the AMQ server) I will try and run some tests when I get some spare time and report back with anything significant.
I have sone problem with size of sample traincascade. I use opencv_traincascade.exe to train rear of car. Size of rear of car which i need to have a size 72x48. Now i use createsample to create 2000 sample size 72x48 with command
D:\Project_Android\Classifier\bin\opencv_createsamples.exe -info positive.txt -bg negative.txt -vec vector.vec -num 2000 -w 72 -h 48
Now, i have a vector file and negative.txt (sure negative.txt have absolute direction to background images). i use opencv_traincascade.exe with command.
D:\Project_Android\Classifier\bin\opencv_traincascade.exe -data HaarTraining -vec vector.vec -bg negative.txt -numPos 250 -numNeg 1500 -numStages 10 -nonsym -minhitrate 0.999 -maxfalsealarm 0.5 -mode ALL -w 72 -h 48 -precalcValBufSize 2046 -precalcIdxBufSize 2046 PAUSE
I get an error.
When i use size 24x24 (-w 24 -h 24) for createsample.exe step and cascadetraining step, everything is ok, traincasecade is run.
it seem to be size 24x24 is a default size of sample and it still remain. and i seem to be cannot config it in file .bat (i write command in file bat and click file bat to run)
And one thing i see in command window in above picture that: -precalcValBufSize and -precalcIdxBufSize still remain default value 1024. No change with -w -h -precalcValBufSize -precalcIdxBufSize.
So what problem with it, i don't understand why don't i choose size of sample for train. I just can use size 24x24, all size other always give error "Assertion failed <_img.row*_img.height == vecSize> in CvcascadeImageReader...pla pla"
Please help me to solve this problem. So many thank to you
I need a script to check time difference between local servers and internet time.
To obtain local time I'd run something like
[root#check ~]# ntpdate -q timeserver
server x.x.x.x, stratum 1, offset -0.034282, delay 0.04228
11 Dec 12:04:21 ntpdate[62714]: adjust time server x.x.x.x offset -0.034282 sec
from above output we could grab,
11 Dec 12:04:21
To obtain internet time I'd like to run something like
[root#check~]# curl http://www.timeapi.org/utc/now
2015-12-11T12:04:43+00:00
from above output we could grab,
2015-12-11T12:04:43
Then echo ("Time drift detected"), if local time > internet time
I'm a newbie to shell scipting and your support is appreciated.
Many thanks
I am trying to use the textcleaner script for cleaning up real life images that I am using with OCR. The issue I am having is that the images sent to me are rather large sometimes. (3.5mb - 5mb 12MP pics) The command I run with textcleaner ( textcleaner -g -e none -f <int # 10 - 100> -o 5 result1.jpg out1.jpg ) takes about 10 seconds at -f 10 and minutes or more on a -f 100.
To get around this I tried using ImageMagick to compress the image so it was much smaller. Using convert -strip -interlace Plane -gaussian-blur 0.05 -quality 50% main.jpg result1.jpg I was able to take a 3.5mb and convert it almost loss-lessly to ~400kb. However when I run textcleaner on this new file it STILL acts like its a 3.5mb file. (Times are almost exactly the same). I have tested these textcleaner settings against a file NOT compressed #400kb and it is almost instant while -f 100 takes about 12 seconds.
I am about out of ideas. I would like to follow the example here as I am in almost exactly the same situation. However, at the current speed of transformation an entire OCR process could take over 10 minutes when I need this to be around 30 seconds.
My RRD file is not updating, what is the reason?
The graph shows the legend with: -nanv
I created the RRD file using this syntax:
rrdtool create ups.rrd --step 300
DS:input:GAUGE:600:0:360
DS:output:GAUGE:600:0:360
DS:temp:GAUGE:600:0:100
DS:load:GAUGE:600:0:100
DS:bcharge:GAUGE:600:0:100
DS:battv:GAUGE:600:0:100
RRA:AVERAGE:0.5:12:24
RRA:AVERAGE:0.5:288:31
Then I updated the file with this syntax:
rrdtool update ups.rrd N:$inputv:$outputv:$temp:$load:$bcharge:$battv
And graphed it with this:
rrdtool graph ups-day.png
-t "ups "
-s -1day
-h 120 -w 616
-a PNG
-cBACK#F9F9F9
-cSHADEA#DDDDDD
-cSHADEB#DDDDDD
-cGRID#D0D0D0
-cMGRID#D0D0D0
-cARROW#0033CC
DEF:input=ups.rrd:input:AVERAGE
DEF:output=ups.rrd:output:AVERAGE
DEF:temp=ups.rrd:temp:AVERAGE
DEF:load=ups.rrd:load:AVERAGE
DEF:bcharge=ups.rrd:bcharge:AVERAGE
DEF:battv=ups.rrd:battv:AVERAGE
LINE:input#336600
AREA:input#32CD3260:"Input Voltage"
GPRINT:input:MAX:" Max %lgv"
GPRINT:input:AVERAGE:" Avg %lgv"
GPRINT:input:LAST:"Current %lgv\n"
LINE:output#4169E1:"Output Voltage"
GPRINT:output:MAX:"Max %lgv"
GPRINT:output:AVERAGE:" Avg %lgv"
GPRINT:output:LAST:"Current %lgv\n"
LINE:load#FD570E:"Load"
GPRINT:load:MAX:" Max %lg%%"
GPRINT:load:AVERAGE:" Avg %lg%%"
GPRINT:load:LAST:" Current %lg%%\n"
LINE:temp#000ACE:"Temperature"
GPRINT:temp:MAX:" Max %lgc"
GPRINT:temp:AVERAGE:" Avg %lgc"
GPRINT:temp:LAST:" Current %lgc"
You will need at least 13 updates, each 5min apart (IE, 12 PDP (primary data points)) before you can get a single CDP (consolidated data point) written to your RRAs, enabling you to get a data point on the graph. This is because your smallest resolution RRA is a Count 12, meaning you need 12 PDP to make one CDP.
Until you have enough data to write a CDP, you have nothing to graph, and your graph will always have unknown data.
Alternatively, add a smaller resolution RRA (maybe Count 1) so that you do not need to collect data for so long before you have a full CDP.
The update script needs to be run at exactly the same interval as defined in your database.
I see it has a step value of 300 so the database should be updated every 5 minutes.
Just place you update script in a cron job (you can also do it for your graph script)
For example,
sudo crontab -e
If run for the first time choose your favorite editor (I usually go with Vim) and add the full path location of your script and run it every 5 minutes. So add this (don't forget to rename the path):
*/5 * * * * /usr/local/update_script > /dev/null && /usr/local/graph_script > /dev/null
Save it, and wait a couple of minutes. I usually redirect the output to /dev/null in case of the output that can be generated by a script. So if a script that will be executed gives an output crontab will fail and send a notification.