i have create a rrd databasew (under php) with this code:
$opts = array( "--step", "60",
"DS:wattmin:GAUGE:300:0:8000",
"RRA:AVERAGE:0.5:1:2160",
"RRA:AVERAGE:0.5:5:2016",
"RRA:AVERAGE:0.5:15:2880",
"RRA:AVERAGE:0.5:60:8760",
"RRA:MIN:0.5:1:2160",
"RRA:MIN:0.5:5:2016",
"RRA:MIN:0.5:15:2880",
"RRA:MAX:0.5:60:8760",
"RRA:MAX:0.5:1:2160",
"RRA:MAX:0.5:5:2016",
"RRA:MAX:0.5:15:2880",
"RRA:MAX:0.5:60:8760");
$ret = rrd_create(RRD_DB_WP, $opts);
The image will be created like that:
$graphs=array("-6h","-12h","-1d","-1w","-1m","-1y");
$opts = array(
"-e now",
"--vertical-label=°C",
"-h 250",
"DEF:inoctets=".RRD_DB_WP.":wattmin:AVERAGE",
"AREA:inoctets#60B5E8:Watt/min",
"GPRINT:inbits:LAST:Las\: %4.0lfW",
"GPRINT:inbits:AVERAGE:Avg\: %4.0lfW",
"GPRINT:inbits:MAX:Max\: %4.0lfW\\n",
"DEF:grad8=".RRD_DB_TEMPS.":grad8:LAST",
"LINE2:grad8#F5A9A9:Wasser ",
"GPRINT:grad8:LAST:Las\: %2.1lf°C",
"GPRINT:grad8:AVERAGE:Avg\: %2.1lf°C",
"GPRINT:grad8:MIN:Min\: %2.1lf°C",
"GPRINT:grad8:MAX:Max\: %1.1lf°C\\n" );
$ret = rrd_graph(RRD_OUT_PATH. "/waermepumpe".$graph.".gif", $opts);
Everything works fine, but all data after 3 days are truncated and the created graphs after that time are always empty.
Bfo
If your graphs are showing no data past 3 days, then they may be being generated using the smaller RRA for some reason, which does not hold data back that far.
Possible reasons for this
You have incorrectly created your RRD file and the larger RRAs are missing. Use rrdtool info to verify the RRD file structure is as you expect.
You have not collected data for that long ago. Use rrdtool dump to view the content of the RRA and confirm there really is data stored.
You have changed your RRA sizes or added new ones ( using rrdtool tune ) and though they now exist they do not have data
Your graphing is using incorrect start and endpoints for the time window
Your graphing is forcing 60s granularity - and hence the smaller RRA - but is trying to graph a time window outside of that RRA
Related
Among the fragments I streamed into KVS via its gstreamer plugin, I see that hundreds of fragments - over 55% - have 0 in the field "FragmentLengthInMilliseconds" (fetched via the LIST_FRAGMENTS api). They all have the same timestamp. Another thing I observed is that in these cases, the value of "FragmentSizeInBytes" is almost always 12265.
I wonder why these fragments even exist? Am I missing something in my gstreamer plugin configuration - to get rid of them (if they are useless)? Here is how I configured it in C++:
g_object_set(kvs_sink,
"stream-name", stream_name.c_str(),
"storage-size", 1024,
"retention-period", 168, // in hours
"aws-region", AWS_REGION, NULL);
This is the plugin I'm using https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp
Your configuration looks OK. Fragment duration is calculated in the backend by adding the frame durations for the entire fragment. Some media pipelines have no way of specifying the frame durations and the fragment duration then evaluates to the timestamp of the last frame minus the timestamp of the first frame. In case of a single frame fragments and no duration specified, it will evaluate to 0. The 0 duration points to an issue with your media pipeline upstream. You might want to double check your configuration. If you are actually streaming video, then ensure your encoder produces a regular cadence I-frames. Take a look at the multiple samples in the repository and look through the docs.
I'm sorry for this poor title but can't formulate correct title at the time.
What is the process:
Get message id aka offset from a sheet.
Get list of updates from telegram based in this offset.
Store last message id which will be used as offset in p.1
Process updates.
Offset allows you to fetch updates only received after this offset.
E.g i get 10 messages 1st time and last message id aka offset is 100500. Before 2nd run bot received additional 10 message so there's 20 in total. To not load all of 20 messages (10 of which i already processed) i need to specify offset from 1st run, so API will return only last 10 messages.
PQ is run in Excel.
let
// Offset is number read from table in Excel. Let's say it's 10.
Offset = 10
// Returns list of messages as JSON objects.
Updates = GetUpdates(Offset),
// This update_id will be used as offset in next query run.
LastMessageId = List.Last(Updates)[update_id],
// Map Process function to each item in the list of update JSON objects.
Map = List.Transform(Updates, each Process(_))
in
Map
The issue is that i need to store/read this offset number each time query is executed AND process updates.
Because of lazy evaluation based on code below i can either output LastMessageId from the query or output result of a Map function.
The question is: how can i do both things store/load LastMessageId from Updates and process those updates.
Thank you.
I am facing problems while using S3 output plugin with fluent-d.
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
using %index at end never resolves to _0,_1 . But i always end up with log file names as
sflow_.log
while i need sflow_0.log
Regards,
Can you paste your fluent.conf?. It's hard to find the problem without full info. File creations are mainly controlled by time slice flag and buffer configuration..
<buffer>
#type file or memory
path /fluentd/buffer/s3
timekey_wait 1m
timekey 1m
chunk_limit_size 64m
</buffer>
time_slice_format %Y%m%d%H%M
With above, you create a file every minute and within 1min if your buffer limit is reached or due to any other factor another file is created with index 1 under same minute.
I'm trying to find out if I can store values captured at irregular intervals into an RRD.
I have a script which connects to an ActiveMQ server subscribes to a queue or topic and looks at the message header time stamp, compares it with Time.now to give me a delta.
The data I get from my script is as below;
000000.681 Time Delta
000000.793 Time Delta
000000.583 Time Delta
000001.994 Time Delta
The issue I face is that messages from the ActiveMQ don't necessarily come in at a 'regular interval' (e.g 1/sec, 1/2sec) They could come in at peak times as 5 a second, and quite times 1 every 10 seconds.
I'd like to be able to capture the output into an RRD so I can graph against it but having a look around on the internet it's not clear is this can be done, or if I'd be better off using a.n.other database/store to capture the data into.
The eventual output I'd like would be a graph showing the time delta for each message.
It looks like I could set the RRD using --step to 1 second, and the hart beat to 2 seconds having had a read of the docs.
I found a couple of posts here and here which talk about being careful with the intervals and the fact my data might be averaged, smoothed or otherwise messed about with when written to the RRD. But nothing I've found online has a similar usage case to mine so its a bit hard to know where I should be looking. I'd like my data to be stored as point for each message received.
I have a couple of RRD's setup for testing; one is taking the AVERAGE the other is taking the LAST to produce some graphs. My heartbeat is set for 100 seconds, but the interval is set to 1. I'm now getting data which looks correct. I'm also guessing that empty spaces in graph from the LAST RRA are due to my data coming in slower that 1 per second?
I'll post my create code & output as an answer.
rrdtool will always store data at regular intervals. As data is handed over to rrdtool, it first gets re-sampled to the --step interval. and then further consolidated to the intervals setup in the RRAs.
The exact arrival time of the data (to the millisecond) is taken into account as the re-sampling takes place ...
If two data points are further apart than specified by mrhb, the data is considered non-continuous and rrdtool will store 'unknown' for the interval affected.
I ended up making two sets of RRD's to experiment with;
rrdtool create test1.rrd \
--step '1' \
'DS:ds0:GAUGE:5:0:U' \
'RRA:AVERAGE:0.5:1:86400' \
'RRA:MAX:0.5:1:86400' \
'RRA:AVERAGE:0.5:60:10080' \
'RRA:MAX:0.5:60:10080' \
'RRA:AVERAGE:0.5:120:21600' \
'RRA:MAX:0.5:120:21600' \
'RRA:AVERAGE:0.5:300:105120' \
'RRA:MAX:0.5:300:105120'
and
rrdtool create test.rrd \
--step '1' \
'DS:ds0:GAUGE:5:0:U' \
'RRA:AVERAGE:0.5:1:86400' \
'RRA:LAST:0.5:1:86400' \
'RRA:AVERAGE:0.5:60:10080' \
'RRA:LAST:0.5:60:10080' \
'RRA:AVERAGE:0.5:120:21600' \
'RRA:LAST:0.5:120:21600' \
'RRA:AVERAGE:0.5:300:105120' \
'RRA:MAX:0.5:300:105120'
Which allows me to store;
1sec, archive is kept for 1day back
1min, archive is kept for 7day back
2min, archive is kept for 30day back
5min, archive is kept for 1year back
Which makes these nice graphs;
The graphs where made in PHP with the following code;
<?php
$opts = array(
'--width', '600',
'--height', '100',
'--title', 'Avg Time Delta xxxxxxxxxx (Last 1 Hr)',
'--vertical-label', 'Time Delta',
'--watermark', 'xxxxxxxxxx',
'--start', 'end-1h',
'DEF:out=test.rrd:ds0:AVERAGE',
'DEF:max=test.rrd:ds0:MAX',
'AREA:out#9966FF:Avg Time Delta',
'LINE:max#996600:Max Time Delta',
);
$ret = rrd_graph("graphs/1hr-graph.png", $opts);
if( !is_array($ret) )
{
$err = rrd_error();
echo "rrd_graph() ERROR: $err\n";
}
echo '<img src="http://server/graphs/1hr-graph.png">';
echo '<BR>';
?>
<?php
$opts = array(
'--width', '600',
'--height', '100',
'--title', 'Last Time Delta xxxxxxxxxx (Last 1 Hr)',
'--vertical-label', 'Time Delta',
'--watermark', 'xxxxxxxxxx',
'--start', 'end-1h',
'DEF:avg=test1.rrd:ds0:AVERAGE',
'DEF:last=test1.rrd:ds0:LAST',
'AREA:avg#99AAFF:Avg Time Delta',
'LINE:last#99AA00:Last Time Delta',
);
$ret = rrd_graph("graphs/1hr-last.png", $opts);
if( !is_array($ret) )
{
$err = rrd_error();
echo "rrd_graph() ERROR: $err\n";
}
echo '<img src="http://server/graphs/1hr-last.png">'
?>
From my own sanity checking and watching the data in realtime it looks like both of those graphs are correct, but behave in slightly different ways. When the data feed which this is monitoring is quite and I'm only getting 1 mesg every 10 sec I get a lot of gaps in the LAST graphs whereas the AVERAGE graphs are smoothed out to fill the gaps. I also tried with setting another RRD to ABSOLUTE but the graphs for that looks 'wrong' and the times are all below 1.0.
So it looks like I can feed my RRD at whatever interval I like from my script. It looks like the RRD will sample my data by its defined interval (In my case 1 sec) and then do what it needs to do based on the way I save it (Gauge, Absolute etc) With my heart-beat set to 100 I should always receive some data before that 100 sec times-out - thus avoiding NAN entries in my database.
At the moment I can't tell how well behaved this config will be during times of disruption (e.g delayed messages from the AMQ server) I will try and run some tests when I get some spare time and report back with anything significant.
My RRD file is not updating, what is the reason?
The graph shows the legend with: -nanv
I created the RRD file using this syntax:
rrdtool create ups.rrd --step 300
DS:input:GAUGE:600:0:360
DS:output:GAUGE:600:0:360
DS:temp:GAUGE:600:0:100
DS:load:GAUGE:600:0:100
DS:bcharge:GAUGE:600:0:100
DS:battv:GAUGE:600:0:100
RRA:AVERAGE:0.5:12:24
RRA:AVERAGE:0.5:288:31
Then I updated the file with this syntax:
rrdtool update ups.rrd N:$inputv:$outputv:$temp:$load:$bcharge:$battv
And graphed it with this:
rrdtool graph ups-day.png
-t "ups "
-s -1day
-h 120 -w 616
-a PNG
-cBACK#F9F9F9
-cSHADEA#DDDDDD
-cSHADEB#DDDDDD
-cGRID#D0D0D0
-cMGRID#D0D0D0
-cARROW#0033CC
DEF:input=ups.rrd:input:AVERAGE
DEF:output=ups.rrd:output:AVERAGE
DEF:temp=ups.rrd:temp:AVERAGE
DEF:load=ups.rrd:load:AVERAGE
DEF:bcharge=ups.rrd:bcharge:AVERAGE
DEF:battv=ups.rrd:battv:AVERAGE
LINE:input#336600
AREA:input#32CD3260:"Input Voltage"
GPRINT:input:MAX:" Max %lgv"
GPRINT:input:AVERAGE:" Avg %lgv"
GPRINT:input:LAST:"Current %lgv\n"
LINE:output#4169E1:"Output Voltage"
GPRINT:output:MAX:"Max %lgv"
GPRINT:output:AVERAGE:" Avg %lgv"
GPRINT:output:LAST:"Current %lgv\n"
LINE:load#FD570E:"Load"
GPRINT:load:MAX:" Max %lg%%"
GPRINT:load:AVERAGE:" Avg %lg%%"
GPRINT:load:LAST:" Current %lg%%\n"
LINE:temp#000ACE:"Temperature"
GPRINT:temp:MAX:" Max %lgc"
GPRINT:temp:AVERAGE:" Avg %lgc"
GPRINT:temp:LAST:" Current %lgc"
You will need at least 13 updates, each 5min apart (IE, 12 PDP (primary data points)) before you can get a single CDP (consolidated data point) written to your RRAs, enabling you to get a data point on the graph. This is because your smallest resolution RRA is a Count 12, meaning you need 12 PDP to make one CDP.
Until you have enough data to write a CDP, you have nothing to graph, and your graph will always have unknown data.
Alternatively, add a smaller resolution RRA (maybe Count 1) so that you do not need to collect data for so long before you have a full CDP.
The update script needs to be run at exactly the same interval as defined in your database.
I see it has a step value of 300 so the database should be updated every 5 minutes.
Just place you update script in a cron job (you can also do it for your graph script)
For example,
sudo crontab -e
If run for the first time choose your favorite editor (I usually go with Vim) and add the full path location of your script and run it every 5 minutes. So add this (don't forget to rename the path):
*/5 * * * * /usr/local/update_script > /dev/null && /usr/local/graph_script > /dev/null
Save it, and wait a couple of minutes. I usually redirect the output to /dev/null in case of the output that can be generated by a script. So if a script that will be executed gives an output crontab will fail and send a notification.