I am using rrdtool to record data off of a Morningstar Solar Charge controller. The data is obtained through SNMP. One of the datapoints being recorded is the "Charge Current" produced by the Solar array. I'm using PHP's "rrd_graph" to generate the graph. I have no problem generating a graph to show the power generated by the solar array over time, but I also want a summary of the AmpHours generated for the past "x" time displayed by the graph. The data is recorded into the rrd database every 60 seconds. This is the PHP code to display the desired graph:
// Solar Array
$opts= array("--start", $start, "--end", $timestamp,
"-h", "250",
"-w", "800",
"-E",
"-v", "Watts",
"--title", "Array Power for $node",
"DEF:arraymaxpower=/home/anr/data/solar/$node.rrd:arraymaxpower:AVERAGE",
"DEF:arraypower=/home/anr/data/solar/$node.rrd:arraypower:AVERAGE",
"DEF:chargecurrent=/home/anr/data/solar/$node.rrd:chargecurrent:AVERAGE",
"CDEF:amphours=chargecurrent,60,/",
"VDEF:amphourstot=amphours,TOTAL",
"AREA:arraypower#ffaf8f:Array Power",
"LINE1:arraypower#852600",
"LINE2:arraymaxpower#336600:Array Max Power",
"GPRINT:amphourstot:Amp Hours\: %5.2lf "
);
$ret = rrd_graph("/var/www/html/admin/solar/graphs/$node-arraypower.png", $opts);
if (! $ret) {
echo "<b>Graph error: </b>" . rrd_error() . "\n";
}
echo "<img src='/admin/solar/graphs/$node-arraypower.png' alt='Generated RRD image'><br />";
While I'm not displaying the charge. current in the graph, the datapoint generated by the DEF statement is needed to calculate the AmpHours value that I need. Since the data is stored every 60 seconds, I assumed that if I simply divided the data by 60 in a code, it would change the stored value from "AmpMinutes" to "AmpHours" and then I could use a VDEF to Total that value over the displayed range in the graph. However, I am ending up with numbers that are way too high. Any idea on what I am doing wrong?
Related
I have a water flowmeter connected to a RPi which is writing data to a simple RRD
RRDs::create ($rrdfile, "--start", 1572829200,
"--step", 60,
"DS:FLOW1:GAUGE:90:U:U",
"RRA:MAX:0.5:1:10512000",);
From this I generate a graph for the last 24 hours and some statistics for the last few days. A simplified version follows
RRDs::graph "temps.png",
"--start=now-1d",
"--end=now",
"--width=1000",
"--base=1000",
"--height=240",
"--title=Flow Data - ",
"--slope-mode",
"--vertical-label=Volume of Water",
"DEF:flow-now=flow.rrd:FLOW1:AVERAGE", #Used to generate the graph
"DEF:flow-1d=flow.rrd:FLOW1:AVERAGE:end=midnight:start=end-1d", #Data for yesterday
"CDEF:flow-1d-1=flow-1d,25440,/", #Convert raw data to litres
"VDEF:flow-1dtotal=flow-1d-1,TOTAL", #Get total litres
"GPRINT:flow-1dtotal:Total Volume last 1 day = %.2lf L", #Print total for yesterday
I would like to add an arbitrary value to flow-1dtotal but can't work out how. Something along the lines of the psuedo code below is what I need
flow-1dtotal = flow-1dtotal + 1000
Thanks for reading and for any suggestions
Since I didn't find anywhere else this topic I will ask it here. I am getting data from CSV file, I have written datetime format in one of columns. I get that column with pandas module and then I need to count occurrences in specific time slots and plot that with matplotlib. Bellow you can see example of column.
Time and Date
0 2015-08-21 10:51:06.398000
1 2015-08-21 10:51:00.017000
2 2015-08-21 10:52:06.402000
3 2015-08-21 10:54:06.407000
...
I know how I can split time like so:
pd.date_range("10:50", "12:30", freq="1min").time
But how can I assign occurrences of my read values from CSV and then plot it? Any advice or direction would help.
It's hard to tell what you want as you haven't posted desired output but if I understand you correctly you want to count the number of rows in time intervals of certain length. You can do this by combining resample and len. To use resample, first set the index to 'Time and Date:
df.set_index('Date and Time', drop=False)
Note that drop=False is only necessary if the data frame has no other columns.
Then to get the number of rows in each 1-minute interval do
counts = df.resample('1min', len).astype(int)
If there are multiple dates and you want to sum the counts for each time interval over dates do
counts.groupby(lambda ts: ts.time()).sum()
I want to generate fake values in RRD DB for a period of 1 month and with 5 seconds as a frequency for data collection. Is there any tool which would fill RRD DB with fake data for given time duration.
I Googled a lot but did not find any such tool.
Please help.
I would recommend the following one liner:
perl -e 'my $start = time - 30 * 24 * 3600; print join " ","update","my.rrd",(map { ($start+$_*5).":".rand} 0..(30*24*3600/5))' | rrdtool -
this assumes you have an rrd file called my.rrd and that is contains just one data source expecting GAUGE type data.
I'm trying to use the Neo4j 2.1.5 regex matching in Cypher and running into problems.
I need to implement a full text search on specific fields that a user has access to. The access requirement is key and is what prevents me from just dumping everything into a Lucene instance and querying that way. The access system is dynamic and so I need to query for the set of nodes that a particular user has access to and then within those nodes perform the search. I would really like to match the set of nodes against a Lucene query, but I can't figure out how to do that so I'm just using basic regex matching for now. My problem is that Neo4j doesn't always return the expected results.
For example, I have about 200 nodes with one of them being the following:
( i:node {name: "Linear Glass Mosaic Tiles", description: "Introducing our new Rip Curl linear glass mosaic tiles. This Caribbean color combination of greens and blues brings a warm inviting feeling to a kitchen backsplash or bathroom. The colors work very well with white cabinetry or larger tiles. We also carry this product in a small subway mosaic to give you some options! SOLD OUT: Back in stock end of August. Call us to pre-order and save 10%!"})
This query produces one result:
MATCH (p)-->(:group)-->(i:node)
WHERE (i.name =~ "(?i).*mosaic.*")
RETURN i
> Returned 1 row in 569 ms
But this query produces zero results even though the description property matches the expression:
MATCH (p)-->(:group)-->(i:node)
WHERE (i.description=~ "(?i).*mosaic.*")
RETURN i
> Returned 0 rows in 601 ms
And this query also produces zero results even though it includes the name property which returned results previously:
MATCH (p)-->(:group)-->(i:node)
WITH i, (p.name + i.name + COALESCE(i.description, "")) AS searchText
WHERE (searchText =~ "(?i).*mosaic.*")
RETURN i
> Returned 0 rows in 487 ms
MATCH (p)-->(:group)-->(i:node)
WITH i, (p.name + i.name + COALESCE(i.description, "")) AS searchText
RETURN searchText
>
...
SotoLinear Glass Mosaic Tiles Introducing our new Rip Curl linear glass mosaic tiles. This Caribbean color combination of greens and blues brings a warm inviting feeling to a kitchen backsplash or bathroom. The colors work very well with white cabinetry or larger tiles. We also carry this product in a small subway mosaic to give you some options! SOLD OUT: Back in stock end of August. Call us to pre-order and save 10%!
...
Even more odd, if I search for a different term, it returns all of the expected results without a problem.
MATCH (p)-->(:group)-->(i:node)
WITH i, (p.name + i.name + COALESCE(i.description, "")) AS searchText
WHERE (searchText =~ "(?i).*plumbing.*")
RETURN i
> Returned 8 rows in 522 ms
I then tried to cache the search text on the nodes and I added an index to see if that would change anything, but it still didn't produce any results.
CREATE INDEX ON :node(searchText)
MATCH (p)-->(:group)-->(i:node)
WHERE (i.searchText =~ "(?i).*mosaic.*")
RETURN i
> Returned 0 rows in 3182 ms
I then tried to simplify the data to reproduce the problem, but in this simple case it works as expected:
MERGE (i:node {name: "Linear Glass Mosaic Tiles", description: "Introducing our new Rip Curl linear glass mosaic tiles. This Caribbean color combination of greens and blues brings a warm inviting feeling to a kitchen backsplash or bathroom. The colors work very well with white cabinetry or larger tiles. We also carry this product in a small subway mosaic to give you some options! SOLD OUT: Back in stock end of August. Call us to pre-order and save 10%!"})
WITH i, (
i.name + " " + COALESCE(i.description, "")
) AS searchText
WHERE searchText =~ "(?i).*mosaic.*"
RETURN i
> Returned 1 rows in 630 ms
I tried using the CYPHER 2.1.EXPERIMENTAL tag as well but that didn't change any of the results. Am I making incorrect assumptions on how the regex support works? Is there something else I should try or some other way to debug the problem?
Additional information
Here is a sample call that I make to the Cypher Transactional Rest API when creating my nodes. This is the actual plain text that is sent (other than some formatting for easier reading) when adding nodes to the database. Any string encoding is just standard URL encoding that is performed by Go when creating a new HTTP request.
{"statements":[
{
"parameters":
{
"p01":"lsF30nP7TsyFh",
"p02":
{
"description":"Introducing our new Rip Curl linear glass mosaic tiles. This Caribbean color combination of greens and blues brings a warm inviting feeling to a kitchen backsplash or bathroom. The colors work very well with white cabinetry or larger tiles. We also carry this product in a small subway mosaic to give you some options! SOLD OUT: Back in stock end of August. Call us to pre-order and save 10%!",
"id":"lsF3BxzFdn0kj",
"name":"Linear Glass Mosaic Tiles",
"object":"material"
}
},
"resultDataContents":["row"],
"statement":
"MATCH (p:project { id: { p01 } })
WITH p
CREATE UNIQUE (p)-[:MATERIAL]->(:materials:group {name: \"Materials\"})-[:MATERIAL]->(m:material { p02 })"
}
]}
If it is an encoding issue, why does a search on name work, description not work, and name + description not work? Is there any way to examine the database to see if/how the data was encoded. When I perform searches, the text returned appears correct.
just a few notes:
probably replace create unique with merge (which works a bit differently)
for your fulltext search I would go with the lucene legacy index for performance, if your group restriction is not limiting enough to keep the response below a few ms
I just tried your exact json statement, and it works perfectly.
inserted with
curl -H accept:application/json -H content-type:application/json -d #insert.json \
-XPOST http://localhost:7474/db/data/transaction/commit
json:
{"statements":[
{
"parameters":
{
"p01":"lsF30nP7TsyFh",
"p02":
{
"description":"Introducing our new Rip Curl linear glass mosaic tiles. This Caribbean color combination of greens and blues brings a warm inviting feeling to a kitchen backsplash or bathroom. The colors work very well with white cabinetry or larger tiles. We also carry this product in a small subway mosaic to give you some options! SOLD OUT: Back in stock end of August. Call us to pre-order and save 10%!",
"id":"lsF3BxzFdn0kj",
"name":"Linear Glass Mosaic Tiles",
"object":"material"
}
},
"resultDataContents":["row"],
"statement":
"MERGE (p:project { id: { p01 } })
WITH p
CREATE UNIQUE (p)-[:MATERIAL]->(:materials:group {name: \"Materials\"})-[:MATERIAL]->(m:material { p02 }) RETURN m"
}
]}
queried:
MATCH (p)-->(:group)-->(i:material)
WHERE (i.description=~ "(?i).*mosaic.*")
RETURN i
returns:
name: Linear Glass Mosaic Tiles
id: lsF3BxzFdn0kj
description: Introducing our new Rip Curl linear glass mosaic tiles. This Caribbean color combination of greens and blues brings a warm inviting feeling to a kitchen backsplash or bathroom. The colors work very well with white cabinetry or larger tiles. We also carry this product in a small subway mosaic to give you some options! SOLD OUT: Back in stock end of August. Call us to pre-order and save 10%!
object: material
What you can try to check your data is to look at the json or csv dumps that the browser offers (little download icons on the result and table-result)
Or you use neo4j-shell with my shell-import-tools to actually output csv or graphml and check those files.
Or use a bit of java (or groovy) code to check your data.
There is also the consistency-checker that comes with the neo4j-enterprise download. Here is a blog post on how to run it.
java -cp 'lib/*:system/lib/*' org.neo4j.consistency.ConsistencyCheckTool /tmp/foo
I added a groovy test script here: https://gist.github.com/jexp/5a183c3501869ee63d30
One more idea: regexp flags
Sometimes there is a multiline thing going on, there are two more flags:
multiline (?m) which also matches across multiple lines and
dotall (?s) which allows the dot also to match special chars like newlines
So could you try (?ism).*mosaic.*
I want to make a chart of city council members in my city over time. I envision this as kind of being like a line chart. The x axis would be years. There are nine city council seats, so there would be nine straight lines, and each would show who was city council member over time (perhaps through different colored line segments or by showing their names onMouseOver). Perhaps this is like a time line.
When I graph the city's budget, since both the years and city budget are type "number," this classic line graph works out nicely.
For this new graph, I am passing all of the data types "string" since they are peoples' names, and Google Charts API is giving the error: "Data column(s) for axis #0 cannot be of type string"
How can I make this chart? (I not only want to graph numeric data like budget surplus or deficit or number of robberies, but relate [in another chart] who was in charge at that time.)
In PHP, I queried my mySQL database and produced a JSON object in the format Google Chart API needs to receive to make horizontal lines over time that show names onHover like this:
$conn = mysql_connect("x","y","z");
mysql_select_db("a",$conn);
$sql = "SELECT year,d_mayor,d_council1,d_council2,d_council3
FROM metrics WHERE year
IN ('1998','1999','2000','2001','2002','2003','2004','2005','2006','2007','2008','2009','2010','2011','2012')";
$sth = mysql_query($sql, $conn) or die(mysql_error());
//start the json data in the format Google Chart js/API expects to receive it change
$JSONdata = "{
\"cols\": [
{\"label\":\"Year\",\"type\":\"string\"},
{\"label\":\"City Council 1\",\"type\":\"number\"},
{\"label\":\"City Council 2\",\"type\":\"number\"},
{\"label\":\"City Council 3\",\"type\":\"number\"},
{\"label\":\"Mayor\",\"type\":\"number\"}
],
\"rows\": [";
//loop through the db query result set and put into the chart cell values (note last ojbect in array has "," behind it but its working)
while($r = mysql_fetch_assoc($sth)) {
$JSONdata .= "{\"c\":[{\"v\": " . $r['year'] . "}, {\"v\": 1, \"f\": \"" . $r['d_council1'] . "\"}, {\"v\": 2, \"f\": \"" . $r['d_council2'] . "\"}, {\"v\": 3, \"f\": \"" . $r['d_council3'] . "\"}, {\"v\": 10, \"f\": \"" . $r['d_mayor'] . "\"},]},";
}
//end the json data/object literal with the correct syntax
$JSONdata .= "]}";
echo $JSONdata;
mysql_close($conn);