Calculate Device seconds On using Kinesis Analytics - amazon-web-services

Question

I'm experimenting with Kinesis analytics and have solved many problems with it but actually stuck with the following:
I actually have a stream with records that reflects when a Device is turned on an off like:
device_id | timestamp | reading
1 | 2011/09/01 22:30 | 1
1 | 2011/09/01 23:00 | 0
1 | 2011/09/02 03:30 | 1
1 | 2011/09/02 03:31 | 0
I'm using 1 for On and 0 for Off in the reading field.
What I'm trying to accomplish is create a PUMP that redirects the number of seconds a Device has been on every 5 minutes window to another stream looking like:
device_id | timestamp | reading
1 | 2011/09/01 22:35 | 300
1 | 2011/09/01 22:40 | 300
1 | 2011/09/01 22:45 | 300
1 | 2011/09/01 22:50 | 300
1 | 2011/09/01 22:55 | 300
1 | 2011/09/01 23:00 | 300
1 | 2011/09/01 23:05 | 0
1 | 2011/09/01 23:10 | 0
...
Not sure if this is something that can be accomplished with Kinesis Analytics, I can actually do it querying a SQL table but I'm stuck with the fact that is streaming data.

Answer 1

This is possible with Drools Kinesis Analytics (managed service on Amazon):
Types:
package com.text;
import java.util.Deque;
declare EventA
#role( event )
id: int;
timestamp: long;
on: boolean;
//not part of the message
seen: boolean;
end
declare Session
id: int #key;
events: Deque;
end
declare Report
id: int #key;
timestamp: long #key;
onInLast5Mins: int;
end
Rules:
package com.text;
import java.util.Deque;
import java.util.ArrayDeque;
declare enum Constants
// 20 seconds - faster to test
WINDOW_SIZE(20*1000);
value: int;
end
rule "Reporter"
// 20 seconds - faster to test
timer(cron:0/20 * * ? * * *)
when
$s: Session()
then
long now = System.currentTimeMillis();
int on = 0; //how long was on
int off = 0; //how long was off
int toPersist = 0; //last interesting event
for (EventA a : (Deque<EventA>)$s.getEvents()) {
toPersist ++;
boolean stop = false;
// time elapsed since the reading till now
int delta = (int)(now - a.getTimestamp());
if (delta >= Constants.WINDOW_SIZE.getValue()) {
delta = Constants.WINDOW_SIZE.getValue();
stop = true;
}
// remove time already counted
delta -= (on+off);
if (a.isOn())
on += delta;
else
off += delta;
if (stop)
break;
}
int toRemove = $s.getEvents().size() - toPersist;
while (toRemove > 0) {
// this event is out of window of interest - delete
delete($s.getEvents().removeLast());
toRemove --;
}
insertLogical(new Report($s.getId(), now, on));
end
rule "SessionCreate"
when
// for every new EventA
EventA(!seen, $id: id) from entry-point events
// check there is no session
not (exists(Session(id == $id)))
then
insert(new Session($id, new ArrayDeque()));
end
rule "SessionJoin"
when
// for every new EventA
$a : EventA(!seen) from entry-point events
// get event's session
$g: Session(id == $a.id)
then
$g.getEvents().push($a);
modify($a) {
setSeen(true),
setTimestamp(System.currentTimeMillis())
};
end

Answer 2

You can do this using SQL with the Stride HTTP API. You can chain together networks of continuous SQL queries and subscribe to streams of changes, as well as fire realtime webhooks if you want to take some kind of arbitrary action when this happens. See the Stride API docs for more info on this.