How to implement execution plans using siddhi? - wso2

First of all let me description my requirement:
data.csv:
column one is id,column two is vlaue.
1,0
2,0
3,0
4,86
5,87
6,88
7,89
8,86
9,0
10,0
11,0
12,0
13,0
14,86
15,87
16,88
17,89
18,0
19,0
20,0
here is my InputStream and my OutPutStream:
id int,value int
data.csv will insert into InputStream by using Event Stream Simulator.
If there are five consecutive value>=85, I would record the first id,value into OutPutStream.
For example,I will record id=4,value=86 ,but id=14 to id=17 i will ignore it.
So how can i write siddhi script in execution plans to implement it?
==========================================================================
data2.csv:
1,0
2,0
3,0
4,86
5,87
6,88
7,89
8,86
9,87
10,88
11,89
12,90
13,91
14,86
15,87
16,88
17,89
18,90
19,90
20,90
21,0
22,0,
23,87
24,85
25,86
26,0
27,17
...
200,91
201,0

For exactly five consecutive values >= 85
from every a1=InputStream[value>=85], a2=InputStream[value>=85]+, a3=InputStream[value<85]
select a1.id, a1.value
having (not (a2[3] is null)) and (a2[4] is null)
insert into OutPutStream;
For more than five consecutive values >= 85
from every a1=InputStream[value>=85], a2=InputStream[value>=85]+, a3=InputStream[value<85]
select a1.id, a1.value
having (not (a2[3] is null))
insert into OutPutStream;

from every a1=InputStream[value>=85], a1=InputStream[value>=85]<4>
select a1.id, a1.value
insert into OutPutStream;
Should work!

Related

How can I make a MicroBit MakeCode mass code creator?

I am working on a project for my town's Maker Faire. What I'm trying to do is have a Micro:Bit send a message through radio, where another one would pick it up and send it through another channel. Then another Micro:Bit would pick that up and so on and so forth. I have the code for the starting micro:bit that sends the first message, and the second micro:bit that receives the first one's message and sends it out again. Each new Micro:Bit bumps up the radio channels by one. Is there any way to do this automatically without having to manually bump it up for each new Micro:bit?
This is my code for the second Micro:Bit:
radio.onReceivedString(function (receivedString) {
radio.setGroup(1)
basic.showString(receivedString)
radio.setGroup(2)
radio.sendString(receivedString)
})
Thanks!
The challenge here is coming up with a way so that each micro:bit knows what its sequence number is on startup. If you're able to initialise each micro:bit with a unique sequence number (eg: 0, 1, 2, 3, 4, 5), then you can flash the same code on each micro:bit and just use the sequence number as an offset. ie: setGroup(sequenceNumber)... setGroup (sequenceNumber + 1).
In the case of the first micro:bit that will be groups 0 and 1 respectively, in the case of the second micro:bit that will be groups 1 and 2 respectively, and so on.
I can think of a few ways of having each micro:bit have its own unique sequence number on startup. One way you can do that is have them all set to 0 on startup, and then use the buttons on each micro:bit to change the sequence number. Something like this:
let sequenceNumber = 0;
input.onButtonPressed(Button.A, function () {
if (sequenceNumber > 0) sequenceNumber--;
})
input.onButtonPressed(Button.B, function () {
sequenceNumber++;
})
radio.onReceivedString(function (receivedString) {
radio.setGroup(sequenceNumber)
basic.showString(receivedString)
radio.setGroup(sequenceNumber + 1)
radio.sendString(receivedString)
})
The above strategy would require you to go around each micro:bit and manually set their sequence number after flashing it. If you flash it again, you'll have to repeat the process..
Another way to approach this is to have all micro:bits running the same program, except for one, which we'll refer to as the master. This master micro:bit will keep a list of all devices its seen (over radio on a preset group, eg: 0) and for every new micro:bit that requests a sequence number, it'll assign it a unique number and send it back. Each of the other micro:bits will startup in an initialization phase where it continuously requests a sequence number and polls until it's been assigned one by the master micro:bit.
Something like the following:
Master:
let masterGroupNumber = 0; // predetermined group number for master micro:bit
let currentSequenceNumber = 1;
let devices: { [key: string]: number } = {};
radio.setGroup(masterGroupNumber);
radio.onReceivedValue(function (name: string, value: number) {
if (name === "uid") {
// We're received a unique id. Assign it a sequence number
// if it has not already been assigned
let uid = value.toString();
if (!devices[uid])
devices[uid] = currentSequenceNumber++;
radio.sendValue(uid, devices[uid]);
}
})
All other micro:bits:
// Begin the program in the initialization phase,
// we're waiting to be assigned a sequence number
let masterGroupNumber = 0; // predetermined group number for master micro:bit
let sequenceNumber = 0;
let hasSequenceNumber = false;
radio.setGroup(masterGroupNumber);
let uniqueDeviceId = control.deviceSerialNumber();
radio.onReceivedValue(function (name: string, value: number) {
if (name === uniqueDeviceId.toString()) {
sequenceNumber = value;
hasSequenceNumber = true;
}
})
// Poll till we've received a sequence number
while (!hasSequenceNumber) {
// Broadcast our unique device id.
radio.sendValue("uid", uniqueDeviceId);
// Wait a litte
basic.pause(500);
}
// We've established communication with the master micro:bit
// and have received a sequence number
radio.onReceivedString(function (receivedString) {
radio.setGroup(sequenceNumber)
basic.showString(receivedString)
radio.setGroup(sequenceNumber + 1)
radio.sendString(receivedString)
})
There's probably a few other ways you could go about doing this, but I hope this gives you some ideas.
ps: I did not have a chance to test if this code works. Let me know how it goes :)

sorted() list not working as expected. Python 2.7

Trying to sort this list from lowest (399||1) to highest value (11064||2) while conserving the provided data format to be reused in an API loop request.
As you can see below, sorted() is not working as (I) expected. This is Python 2.7.
It looks like it sorts in pieces. Why would 1000-1100 come before 300-700, and then 8000? I cannot find this same issue posted anywhere.
sorted_d = sorted(d)
print sorted_d
Run:
[u'1053||1', u'1092||2', u'1093||1', u'1094||1', u'1094||2', u'1095||1',
u'1095||2', u'1096||7', u'1096||8', u'1097||7', u'1097||8', u'11064||1',
u'11064||2', u'399||1', u'412||1', u'412||2', u'413||1', u'414||1',
u'434||2', u'616||1', u'617||1', u'618||1', u'619||1', u'620||1', u'621||1',
u'622||1', u'727||1', u'8096||1', u'8097||1', u'8099||1', u'8101||1',
u'8105||1', u'8112||1', u'8113||1', u'8140||1', u'8141||1', u'8142||1',
u'8143||1', u'8144||1', u'8146||2', u'8150||1', u'8152||1', u'8153||1',
u'8154||1', u'8157||1', u'8158||1', u'8159||1', u'8160||1', u'8161||1',
u'8162||1', u'8163||1', u'8164||1', u'8165||1', u'8166||1', u'8167||1',
u'8168||1', u'8169||1', u'8170||1', u'8171||1', u'8172||1', u'8173||1',
u'8174||1', u'8175||1', u'8184||2', u'8184||3', u'8185||2', u'8185||3',
u'8186||5', u'8186||6', u'8187||1', u'8188||2', u'8190||2', u'8191||1']
Assistance greatly appreciated.
Thx
You could also split the strings on the || and specify the first part as the key parameter
sorted_d = sorted(d, key = lambda x: int(x.split('||')[0]))
print sorted_d
[u'399||1', u'412||1', u'412||2', u'413||1', u'414||1', u'434||2', u'616||1', u'617||1', u'618||1', u'619||1', u'620||1', u'621||1', u'622||1', u'727||1', u'1053||1', u'1092||2', u'1093||1', u'1094||1', u'1094||2', u'1095||1', u'1095||2', u'1096||7', u'1096||8', u'1097||7', u'1097||8', u'8096||1', u'8097||1', u'8099||1', u'8101||1', u'8105||1', u'8112||1', u'8113||1', u'8140||1', u'8141||1', u'8142||1', u'8143||1', u'8144||1', u'8146||2', u'8150||1', u'8152||1', u'8153||1', u'8154||1', u'8157||1', u'8158||1', u'8159||1', u'8160||1', u'8161||1', u'8162||1', u'8163||1', u'8164||1', u'8165||1', u'8166||1', u'8167||1', u'8168||1', u'8169||1', u'8170||1', u'8171||1', u'8172||1', u'8173||1', u'8174||1', u'8175||1', u'8184||2', u'8184||3', u'8185||2', u'8185||3', u'8186||5', u'8186||6', u'8187||1', u'8188||2', u'8190||2', u'8191||1', u'11064||1', u'11064||2']
Because it's treating your '1053||1' data as strings and sorting as a string type instead of as a numeric value. So it effectively is sorting in this type of manner, ascending:
1
10
100
1000
2
20
200
2000

Python - get parent index from child index, child level and parent level

I need help to figure out how to get parent index from child index, child-level and parent-level using Python.
I have dataset with three columns: index, child-level and parent-level.
The records are in order of hierarchy.
Index is just the line number of record.
Child-level is number indicating level in hierarchy of nested parent child records.
Parent-level = child-level - 1
My challenge is, for each record, I want to use Python to get each record's parent index.
I suspect a list comprehension might be used to get the max index value where the self join index < child.index and the self join level = child.level
This is a visual representation of the data set.
This is sample data and expected result. Goal is to get parent index.
Index, Child-Level,Parent-Level,Parent-Index
1,1,1,1
2,2,1,1
4,4,3,3
9,9,8,8
3,3,2,2
5,5,4,4
8,8,7,7
6,6,5,5
7,7,6,6
10,10,9,9
11,11,10,10
12,12,11,11
13,13,12,12
14,14,13,13
15,14,13,13
16,14,13,13
17,14,13,13
18,14,13,13
19,14,13,13
20,14,13,13
21,13,12,12
22,13,12,12
23,13,12,12
24,14,13,23
25,14,13,23
26,14,13,23
27,11,10,10
28,9,8,8
29,9,8,8
30,9,8,8
31,9,8,8
32,9,8,8
33,9,8,8
34,9,8,8
35,8,7,7
36,9,8,35
37,10,9,36
38,11,10,37
39,11,10,37
40,12,11,39
41,12,11,39
42,13,12,41
43,13,12,41
44,13,12,41
45,11,10,37
46,12,11,45
47,13,12,46
48,14,13,47
49,14,13,47
50,14,13,47
51,14,13,47
52,14,13,47
53,14,13,47
54,14,13,47
55,13,12,46
56,13,12,46
57,13,12,46
58,9,8,35
59,9,8,35
60,9,8,35
61,9,8,35
62,8,7,7
63,8,7,7
64,8,7,7
65,8,7,7
66,8,7,7
67,8,7,7
68,8,7,7

How to remove all lines from one list that don't start with the lines from another with regex?

Sorry if the title is a bit convoluted, I found this particular question hard to phrase. Basically, I have two lists, and I'm attempting to modify the first one with information with the second, using Notepad++.
LIST 1 (Not the entire list):
2000,4031161,1,1,1008,1000000
2000,4031162,1,1,1008,1000000
100100,4000019,1,1,0,600000
100100,2000000,1,1,0,20000
100100,2040002,1,1,0,300
100100,2041001,1,1,0,300
100100,2060000,1,1,0,30000
100100,4010000,1,1,0,9000
100100,4020000,1,1,0,9000
100100,2061000,1,1,0,30000
100100,1002067,1,1,0,1500
100100,2010009,1,1,0,20000
100100,2380000,1,1,0,1000
100100,0,4,6,0,400000
100101,4000000,1,1,0,600000
100101,2041006,1,1,0,300
100101,2000000,1,1,0,20000
100101,4020001,1,1,0,9000
100101,2060000,1,1,0,30000
100101,4010001,1,1,0,9000
100101,2061000,1,1,0,30000
100101,1040013,1,1,0,800
100101,1041012,1,1,0,800
100101,1060004,1,1,0,800
100101,1040017,1,1,0,800
100101,1060013,1,1,0,800
100101,2010009,1,1,0,20000
100101,2380001,1,1,0,1000
100101,0,8,12,0,400000
100120,0,1,5,0,400000
100121,0,10,14,0,400000
100121,4000483,1,1,0,400000
100130,4000493,1,1,0,600000
100130,2010000,1,1,0,20000
100130,2010009,1,1,0,20000
100130,4010005,1,1,0,9000
100130,4020005,1,1,0,9000
100130,2040003,1,1,0,300
100130,1002008,1,1,0,1500
100130,1040010,1,1,0,800
100130,1041004,1,1,0,800
100130,1060007,1,1,0,800
100130,2380015,1,1,0,1000
100131,4000494,1,1,0,600000
100131,2000000,1,1,0,20000
100131,2010009,1,1,0,20000
100131,4010006,1,1,0,9000
100131,4020006,1,1,0,9000
100131,2040400,1,1,0,300
100131,2040618,1,1,0,300
100131,1002019,1,1,0,1500
100131,1002002,1,1,0,1500
100131,1040013,1,1,0,800
100131,1041012,1,1,0,800
100131,1060004,1,1,0,800
100131,1072005,1,1,0,800
100131,2380016,1,1,0,1000
100132,4000495,1,1,0,600000
100132,2000000,1,1,0,20000
100132,2010009,1,1,0,20000
100132,4010000,1,1,0,9000
100132,4020007,1,1,0,9000
100132,2040823,1,1,0,300
100132,2041018,1,1,0,300
100132,1002001,1,1,0,1500
100132,1002003,1,1,0,1500
100132,1040014,1,1,0,800
100132,1040015,1,1,0,800
100132,1060008,1,1,0,800
100132,1041014,1,1,0,800
100132,1061014,1,1,0,800
100132,1072004,1,1,0,800
100132,1082003,1,1,0,1000
100132,1442000,1,1,0,700
100132,2380017,1,1,0,1000
100133,4000496,1,1,0,600000
100133,2000000,1,1,0,20000
100133,2010009,1,1,0,20000
100133,4010001,1,1,0,9000
100133,4020003,1,1,0,9000
100133,2048000,1,1,0,300
100133,2041004,1,1,0,300
100133,1002041,1,1,0,1500
100133,1002007,1,1,0,1500
100133,1032001,1,1,0,1000
100133,1040038,1,1,0,800
100133,1060028,1,1,0,800
100133,1041064,1,1,0,800
LIST 2 (Not the entire list):
2000,4031161
2000,4031162
100130,2040003
100131,2040400
100133,2048000
100134,2040500
100134,2044400
130101,4031846
210100,4031273
851000,2290132
1110100,4020002
1110100,4031146
1110130,4000012
1110130,2043102
1110130,1092008
1110130,2048000
1110130,1002033
1110130,1302007
1110130,1032001
1110130,1412012
1110130,4032316
1130100,4031147
1140130,2048001
1140130,1412012
1140130,2044802
1210100,4031846
1210100,4032340
1210102,4032314
2100100,4020006
2100100,4010001
2100100,4010007
2100100,2040420
2100100,2049000
2100101,4010006
2100101,4020001
2100101,4010007
2100101,2044210
2100102,2043212
2100103,2044314
2100104,1452022
2100105,2040316
2100105,2040319
2100105,2044412
2100106,2040926
2100107,1382009
2100108,4010002
2100108,4010001
2100108,4010007
2100108,2044014
2100108,2044214
2110200,2043214
2110200,1452016
2110200,4032390
2110300,2043214
2110301,4010002
2110301,2043114
2130100,2044012
2130100,2044210
2130103,2040617
2220000,4010000
2220000,4020000
2220100,4020006
2230100,4020007
2230100,2040823
2230100,2044010
2230101,4010003
2230102,4031155
2230102,4007001
2230102,1462014
2230103,2040319
2230103,2044114
2230103,1382009
2230104,2040929
2230104,2043112
2230104,1452016
2230105,2040617
2230105,2043015
2230105,4031259
2230106,2040417
2230106,4031268
2230106,4031260
2230106,4031269
2230107,1092030
2230108,2040623
2230108,4031261
2230109,4010004
2230109,4031264
2230110,2044312
2230110,2044805
2230110,1472030
2230111,2049000
2230131,4000008
2230131,1050031
2230200,4031262
2300100,2043112
3000000,2040316
3000000,2040620
3000001,4000068
3000001,2000001
3000001,2000003
3000001,4020004
3000001,4010002
3000001,2050000
3000001,2050001
3000001,2050002
3000001,2050003
3000001,2050004
3100101,4010005
3100101,4020000
3100101,4010007
3100101,4130005
3100101,4130009
3100101,1332025
3110100,4130002
3110100,4130008
3110100,4130010
3110100,4007005
3110101,2044012
3110101,4130002
3110102,4131002
3110102,2044210
3110102,4130003
3110102,4130004
3110102,4130011
3110102,4031129
3110102,4007007
3110102,4007001
3110102,1302030
3110300,2040530
3110300,2044410
3110300,4130002
3110300,4130009
3110300,4130013
3110300,4007000
3110300,4007004
3110301,4010005
3110301,4020000
3110301,4010007
3110301,2040420
3110301,4130001
3110301,4130006
3110302,2040324
3110302,2044210
3110302,4130010
3110302,4130015
3110302,4031694
3110302,4007003
3110303,2040417
3110303,2044112
3110303,2044310
3110303,2044809
3110303,4130001
3110303,4130002
3110303,4130016
3110303,4031694
3110303,1472030
3210100,4010002
3210100,4130011
3210100,4130016
3210100,4130017
3210100,4007003
3210100,4007001
3210100,1382009
3210200,4130007
3210200,4130016
3210200,4007000
3210200,4007006
3210200,4007001
3210201,2043114
3210201,4130003
3210201,4130004
3210201,4130012
3210202,2043110
3210202,2044807
3210202,4130006
3210202,4130012
3210203,2040923
3210203,2043212
3210204,2040617
3210204,4130015
3210204,4130017
3210205,4130001
3210205,4130004
3210205,4130014
3210205,4031093
3210205,4007007
3210205,4007005
3210205,1412011
3210206,4130015
3210206,4130016
3210207,2049000
3210207,4130007
3210207,4130008
3210208,4130006
3210208,4130008
3210208,2382028
3210208,4031279
3210208,4007002
3210208,4007004
3210208,1452022
3210450,4130000
3210450,4130014
3210450,4130017
3210450,4007004
3210800,4020004
3210800,2044414
3210800,4130008
3210800,4130010
3210800,1452022
3220000,1322027
3220000,2044112
3220000,2044412
3230100,4130006
3230100,4130012
3230100,4130017
3230100,4031239
3230101,4130007
3230101,4130014
3230101,4007000
3230101,4007003
3230102,2040024
3230102,2040423
3230102,4130011
3230102,4130015
3230103,2044112
3230103,4130001
3230103,4130011
3230104,2044212
3230104,4130000
3230104,4130003
3230104,4130005
3230104,4031263
3230200,2044807
3230200,4130009
3230200,4130014
3230200,4031309
3230200,4007000
3230200,1432012
3230300,4000067
3230300,2000002
3230300,2000003
3230300,4020000
3230300,4010001
3230300,4004000
3230300,4004001
3230300,4004002
3230300,4004003
3230302,4130005
3230302,4130012
3230302,4130013
3230302,4031089
3230302,1422014
3230303,2044312
3230303,4130009
3230303,4130010
3230303,4130012
3230304,4010001
3230304,2040316
3230304,2049000
3230304,4130002
3230304,4130017
3230305,2040926
3230305,4130003
3230305,4130004
3230305,4130014
3230306,4130000
3230306,4130010
3230306,4007000
3230306,4007005
3230306,1472032
3230307,4010001
3230307,2040929
3230307,2044110
3230307,4130010
3230307,4130013
3230308,2043210
3230308,4130004
3230308,4130006
3230308,4130015
3230400,4130001
3230400,4130008
3230400,4031140
3230400,4031135
3230400,4007002
3230400,4007004
3230405,4131005
3230405,2044410
3230405,4130009
3230405,4130013
3300001,4130005
3300001,4130009
3300005,2043801
3300005,2044801
3300006,2040602
3300006,1041076
3300006,1072126
3300007,2040001
3300007,2040301
3300007,2043701
3300007,2043801
3300007,2040601
3300007,1041033
3300007,2040302
3300007,2044801
3300008,2040301
3300008,2043801
3300008,2044802
4110300,4130002
4110300,4130013
4110300,2382057
4110300,4007004
4110301,4130007
4110301,4130012
4110301,2382072
4110301,4007000
4110301,4007005
4110301,4007006
4110302,2000002
4110302,2000003
4110302,4020000
4110302,4020006
4110302,4130012
4110302,2044102
4110302,1372007
4110302,4006001
4110302,1040089
4110302,1050045
4110302,4004002
4110302,2040001
4110302,4000359
4110302,1082198
4110302,4007006
4110302,4007001
4130100,2040025
4130100,2040621
4130100,2044014
So, basically, I need to remove every line in LIST 1 that doesn't start with one of the lines from LIST 2. So, for example, the first line of LIST 2 is "2000,4031161", so I DO NOT want to remove "2000,4031161,1,1,1008,1000000". One line in LIST 1 is "100100,4000019,1,1,0,600000", and since there is no line in LIST 2 that says "100100,4000019", I want that line removed. The real list is a couple tens of thousands of lines long. I made a really REALLY long regex command that should have sorted it all for me using search and replace, but then I found out there was a 2048 character limit, and I'm interested in finding a better way to do this anyway.
I've no clue using notepad++ but using gawk for windows you can do this:
gawk -F"," 'NR==FNR{l[$0]++; next} {if ($1","$2 in l) print $0 }' file2 file1
The first block make a list l of entries in file2, the first file given in arguments (Number Record == File Number Record) and skip to the next record.
Once file2 has been processed, the second block is executed for each line, as we use , as field separator we search the 2 first fieds as keys in l and print the line only if they're present in the list.
List in awk are c hashes under the hood, so the RAM should not be a problem even for very large number of lines in file2.

Regex in config for dynamic columns in logstash

I have the log file of which i have pasted two rows below:
Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=traffic subtype=forward level=notice vd=root srcip=172.16.1.251 srcport=62032 srcintf="Combo_LAN" dstip=X.X.X.X dstport=X dstintf="wan2" sessionid=16172588 status=close user="X.X" group="Open Group" policyid=2 dstcountry="United States" srccountry="Reserved" trandisp=snat transip=X.X.X.X transport=X service=HTTP proto=6 applist="Block_Applications" duration=11 sentbyte=2377 rcvdbyte=784 sentpkt=6 rcvdpkt=7 identidx=5 utmaction=passthrough utmevent=webfilter utmsubtype=ftgd-cat urlcnt=1 hostname="tacoda.at.atwola.com" catdesc="Advertising"
Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=utm subtype=webfilter eventtype=ftgd_allow level=notice vd="root" policyid=2 identidx=5 sessionid=15536743 user="X.X" srcip=X.X.X.X srcport=X srcintf="Combo_LAN" dstip=X.X.X.X dstport=80 dstintf="wan2" service="http" hostname="streaming.sbismart.com" profiletype="Webfilter_Profile" profile="Open Group_Policy" status="passthrough" reqtype="direct" url="/diffusion/" sentbyte=984 rcvdbyte=202 msg="URL belongs to an allowed category in policy" method=domain class=0 cat=18 catdesc="Brokerage and Trading"
My question is i can parse the data if number of columns and order is fixed.
But, how do i parse the dynamic columns in the config file so that i don't get the _grokparsefailure?
Ruby Plugin can help you.
Here is the configuration:
input {
stdin{
}
}
filter {
ruby {
code => '
msg = event["message"]
msgIndex = msg.index("date=")
msgInsert = msg[msgIndex..-1]
msgMap = msgInsert.scan(/(\w+)=("(.*?)"|([^ ]+))/).map { |(first, second)| [first, second] }
for x in msgMap
key = x[0]
value = x[1]
event[key] = value
end
'
}
}
output {
stdout{
codec => rubydebug
}
}
First, get all the key=value pair by index the start value date=
Then map all the key,value to string array.
Use For loop to insert all the value.
I have try your logs and I can create all the correspond field with the value.
Hope this can help you
The simple answer to avoiding grokparsefailure is to provide a valid pattern that matches your input. That said, your question seems to imply that the fields are not always specified in this order. Given the examples, you should be using the "kv" filter to split these key/value pairs into fields.