Environment: WSO2 Stream Processor 4.3.0
Let's say I have two very simple streams:
Stream where newly created requests (unfulfilled) are being delivered in real time (t1)
RequestStream(requestId)
Stream where requestsIds appear when the request has been fulfilled in real time (t2)
FulfilmentStream(requestId)
It's guaranteed that t2 is always > t1
How can I implement a SiddhiQL statement to identify requestIds that appear at RequestStream (event1) and haven't appeard in FulfilmentStream (event2) after 5 minutes have been elapsed since event1?
Working Siddhi App based on Tishan answer:
#App:name('FailedToFulfillInAmountOfTime')
#source(
type="kafka",
topic.list="some_topic",
threading.option="single.thread",
group.id="some_group",
bootstrap.servers="xxx.xxx.xxx.xxx:6667",
#Map(type="json", #attributes(request_id = '$.alarm_id', severity = '$.severity', managed_object = '$.ManagedObject')))
define stream OrigAlarmStream (request_id int, severity string, managed_object string);
#sink(type='log', prefix='Got this execution request')
define stream RequestStream (request_id int, severity string, managed_object string);
#sink(type='log', prefix='Got this fulfillment confirmation:')
define stream FulfillmentStream (request_id int, severity string, managed_object string);
#sink(type='log', prefix='This fulfillment was not done within 1 min:')
define stream AlertStream(request_id int);
#info(name='getExpiredRequests')
from every e1=RequestStream -> not FulfillmentStream[e1.request_id == request_id] for 1 min
select e1.request_id
insert into AlertStream;
#info(name='CopyFulfillments')
from OrigAlarmStream[severity == 'Clear']
select request_id, severity, managed_object
insert into FulfillmentStream;
#info(name='CopyRequests')
from OrigAlarmStream[severity != 'Clear']
select request_id, severity, managed_object
insert into RequestStream;
You can use logical patterns to achieve your requirement. Please refer below query.
from e1=RequestStream -> not e2=FulfilmentStream[e1.requestId == e2.requestId] for '5 min'
select e1.requestId as requestId
insert into AlertStream;
Here we have defined a pattern with not condition. This will be triggered when an event in RequestStream comes and within 5 minutes no event comes into FulfilmentStream within 5 minutes. Please refer logical patterns for more information.
Related
I am using SiddhiQL for complex event processing. My use case is to check whether two events of a particular type with certain filters occur within 15 mins.
Like for example -
If event1[filter 1] and event2[filter 2] within 15 mins
Insert into output stream
In my case any of the two events can occur first and i need to check whether after receiving the first event, do i receive the second event within 15 mins.
Is this possible in SiddhiQL?
EDIT #1 - I have defined my streams and events below (The below code does not work)
define stream RegulatorStream(deviceID long, roomNo int, tempSet double);
#sink(type = 'log', prefix = "LOGGER")
define stream outputStream (roomNo int,rooomNo int);
from e1 = RegulatorStream[roomNo==23] and e2 = RegulatorStream[e1.deviceID == deviceID AND roomNo ==24] within 5 minutes
select e2.roomNo,123 as rooomNo
insert into outputStream;
In above case, I need to alert when I receive events in my RegulatorStream having roomNo = 23 AND roomNo = 24 within 5 mins in any order with same deviceID.
How can this be achieved in SiddhiQL?
Yes this can achieved with Siddhi Patterns. Please refer the documentation on Siddhi Patterns in https://siddhi.io/en/v5.1/docs/query-guide/#pattern and examples in https://siddhi.io/en/v5.1/docs/examples/logical-pattern/.
You can use OR operation within the pattern to bypass the event occurrence order.
I am trying to create a siddhi application where it adds output when a person is in proximity to certain preset locations. These locations are stored in the database. The input is sent from postman as of now.
But I keep getting the error saying the datatype is not 'double'. I have even checked the table details and the datatype in mysql table is set to double.
Following are the details for siddhi code and the error.
Can someone please guide me.
location for the siddhi extension:
https://wso2-extensions.github.io/siddhi-gpl-execution-geo/api/latest/
Siddhi code:
#App:name('ShipmentHistoryApp')
#source(type = 'http', receiver.url='http://localhost:5008/RawMaterials', #map(type = 'json'))
define stream WalkingStream(latitude DOUBLE, longitude DOUBLE, device_id string);
#store(type='rdbms', jdbc.url="jdbc:mysql://127.0.0.1:3306/SweetFactoryDB", username="root", password="root" , jdbc.driver.name="com.mysql.jdbc.Driver")
define table Offers(c string, offer string, latitude DOUBLE, longitude DOUBLE);
#sink(type='log')
define stream SetLocation(a string, b string,one bool, two bool, dis double);
#sink(type='log', prefix='Only log')
define stream info(one bool, two bool);
from WalkingStream as w
join SetLocation as o
select o.a, o.b, instanceOfDouble(o.latitude) as one, instanceOfDouble(o.longitude) as two, geo:distance(w.latitude,w.longitude,o.latitude,o.longitude) as dis insert into Output;
I'm getting this error when trying to find distance between two locations.
org.wso2.siddhi.core.exception.SiddhiAppRuntimeException: Invalid input given to geo:distance() function. Third argument should be double
at org.wso2.extension.siddhi.gpl.execution.geo.function.GeoDistanceFunctionExecutor.execute(GeoDistanceFunctionExecutor.java:123)
at org.wso2.siddhi.core.executor.function.FunctionExecutor.execute(FunctionExecutor.java:109)
at org.wso2.siddhi.core.query.selector.attribute.processor.AttributeProcessor.process(AttributeProcessor.java:41)
at org.wso2.siddhi.core.query.selector.QuerySelector.processNoGroupBy(QuerySelector.java:145)
at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:87)
at org.wso2.siddhi.core.query.input.stream.join.JoinProcessor.process(JoinProcessor.java:110)
at org.wso2.siddhi.core.query.processor.stream.window.LengthWindowProcessor.process(LengthWindowProcessor.java:135)
at org.wso2.siddhi.core.query.processor.stream.window.WindowProcessor.processEventChunk(WindowProcessor.java:66)
at org.wso2.siddhi.core.query.processor.stream.AbstractStreamProcessor.process(AbstractStreamProcessor.java:123)
at org.wso2.siddhi.core.query.input.stream.join.JoinProcessor.process(JoinProcessor.java:118)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.processAndClear(ProcessStreamReceiver.java:187)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.process(ProcessStreamReceiver.java:97)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.receive(ProcessStreamReceiver.java:133)
at org.wso2.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:151)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:358)
at org.wso2.siddhi.core.stream.input.InputDistributor.send(InputDistributor.java:34)
at org.wso2.siddhi.core.stream.input.InputEntryValve.send(InputEntryValve.java:44)
at org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:61)
at org.wso2.siddhi.core.stream.input.source.PassThroughSourceHandler.sendEvent(PassThroughSourceHandler.java:35)
at org.wso2.siddhi.core.stream.input.source.InputEventHandler.sendEvent(InputEventHandler.java:76)
at org.wso2.extension.siddhi.map.json.sourcemapper.JsonSourceMapper.mapAndProcess(JsonSourceMapper.java:211)
at org.wso2.siddhi.core.stream.input.source.SourceMapper.onEvent(SourceMapper.java:132)
at org.wso2.extension.siddhi.io.http.source.HttpWorkerThread.run(HttpWorkerThread.java:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Alternatively I've also tried to cast the latitude from table into double in the query.
This can be occurred when the 3rd argument of geo:distance function becomes null.
Can you verify whether o.latitude can become null?
I am using the GEO extension within function into an execution plan.
I have multiple event streams, which contain sensor information, including the location of each sensor (in geographical coordinates).
Furthermore, I have a Polygon (example below, which contains the coordinates of each point). I would like to check if it's possible to determine whether the sensors are within the boundaries of this polygon.
My execution plan is the following:
#Plan:name('TestExecutionPlan')
define stream sensorStream (id string, lat double, longi double);
define stream outputStream (id string);
from sensorStream [geo:within(lat,longi,{"type": "Polygon", "coordinates": [[[37.9807986, 23.7262081],[37.9807986, 23.7262081],[37.9792256, 23.7302850],[37.9789888, 23.7268089],[37.9807986, 23.7262081]]]})]
select id
insert into outputStream;
When I'm running my execution plan in Siddhi Try It Tool of the WSO2CEP Management Console the following error is occurring:
You have an error in your SiddhiQL at line 16:108, no viable
alternative at input 'geo:within(sensorStream.lat,
sensorStream.longi,{'type':'Polygon','coordinates':[[[37.9807986,
23.7262081],[37.9807986, 23.7262081],[37.9792256, 23.7302850],[37.9789888, 23.7268089],[37.9807986, 23.7262081]]]}'
I do not know why that error occurs.
I would be very grateful if somebody could help me on this matter.
Thanks!
I solved the error.
The error was existing because it needed to include the {"type": "Polygon", "coordinates": [[[-104.05,48.99],[-97.22,48.98],[-96.58,45.94],[-104.03,45.94],[-104.05,48.99]]]} with question marks i.e to be "{"type": "Polygon", "coordinates": [[[-104.05,48.99],[-97.22,48.98],[-96.58,45.94],[-104.03,45.94],[-104.05,48.99]]]}", because it is a string value as it explained in the syntax of the geo:within function in the Siddhi Geo Extension (https://docs.wso2.com/display/CEP420/Geo+Extension).
So, the execution plan which worked is the following:
#Plan:name('TestExecutionPlan')
define stream sensorStream (id string, lat double, longi double);
define stream outputStream (id string);
from sensorStream [geo:within(lat, longi, " { 'type': 'Polygon', 'coordinates': [[[37.9807986, 23.7262081],[37.9807986, 23.7262081],[37.9792256, 23.7302850],[37.9789888, 23.7268089],[37.9807986, 23.7262081]]] } " )]
select id
insert into outputStream;
From the below stream, I want to alert the event which has occurred twice when the temperature goes beyond 90 (like every 2 event with temp > 90 needs to be alerted).
InputStream=[1001,91]
InputStream=[1001,86]
InputStream=[1002,70]
InputStream=[1001,85]
InputStream=[1003,70]
InputStream=[1003,85]
InputStream=[1002,70]
InputStream=[1003,70]
InputStream=[1003,87]
InputStream=[1002,70]
InputStream=[1001,95]
InputStream=[1001,96]
InputStream=[1001,97]
InputStream=[1001,98]
InputStream=[1001,98]
I have written something like this:
#Plan:name('TestExecutionPlan')
define stream InputStream (id string, temp int);
partition with (id of InputStream)
begin
from InputStream
select id, temp
having temp > 90
insert into CriticalStream
end;
from CriticalStream[count(id) == 2]
select id, temp
group by id
--having count(id) == 2
insert into EventReporter;
However its alerting only 1 event in the EventReporter stream.
Below is the screen shot from Try It
I am expecting the EventReporter stream to have [1001,97] and [1001,98] as well, right now it has only the record for [1001,95]. Could someone please point out what I am doing wrong here. How I can loop through the events after grouping it? I tried adding window.time and window.length, but not getting the desired output. Any help / guidance would be really appreciated. Thank you.
You won't be requiring a partition there. You can simply use a filter and a lengthBatch window to get your desired output. Try below execution plan;
#Plan:name('ExecutionPlan')
#Import('InputStream:1.0.0')
define stream InputStream (id string, temp int);
/* Filter events with temp > 90 */
from InputStream[temp > 90]
insert into CriticalStream;
/* Aggregate within a lengthBatch window, while group by id*/
from CriticalStream#window.lengthBatch(2)
select id, temp, count() as count
group by id
insert into EventReporter;
/* Just for logging the result in the cosole */
from EventReporter#log("Logging EventReporter : ")
insert into #temp;
I keep trying to make complex correlations with Siddhi, on this occasion I have two input streams, web client consult and notices sent to clients visits, I want to generate an alert if the first stream for each client is repeated more than once as long as the second stream not It has occurred under two windows and depends of the status of this events.
define stream consults (idClient string,dniClient string,codProduct string,codSubProduct string,chanel string,time string )
define stream comercialActions(idClient string, idAccionComercial string,codProduct string,codSubProduct string,chanel string,time string,status string)
from consults[codProduct=='Fondos']#window.time(50 seconds) select idClient,codProduct, codSubProduct, chanel, time, count(idClient) as visitCount group by idClient insert into consultsAvg for current-events
from consultsAvg[visitCount==1] select idClient, '' as idAccionComercial,codProduct, codSubProduct ,chanel, time, 'temp' as status insert into comercialActions for all-events
from comercialActions[status=='temp' or status == 'Lanzada' ]#window.time(5 seconds) select idClient as idClient, codProduct, codSubProduct, chanel, status, count(idClient) as num_status group by idClient insert into acciones_generadas for all-events
from comercialActions[status=='temp' or status=='Aceptada' or status =='Rechazada'or status=='Caduca']#window.time(3 seconds) select idClient as idClient, codProduct, codSubProduct, chanel, status, count(idClient) as num_status group by idClient insert into acciones_realizadas for all-events
from consultsAvg[visitCount>=2]#window.time(50 seconds) as c join acciones_realizadas[num_status>=1]#window.time(5 seconds) as ag on c.idClient == ag.idClient and c.codProduct==ag.codProduct select c.idClient,c.codProduct,c.codSubProduct,c.chanel, c.time, count(c.idClient) as conteo insert into posible_ac for all-events
from posible_ac#window.time(5 seconds) as pac join acciones_generadas[num_status>=1]#window.time(1 seconds) as ar on pac.idClient == ar.idClient select pac.idClient,pac.codProduct,pac.codSubProduct,pac.chanel,pac.time,conteo, count(ar.idClient) as conteo2 insert into enviar_Ac
from enviar_Ac[conteo==1 and conteo2==1] select idClient, codProduct,codSubProduct, chanel, time insert into generar_accion_comercial
What I try to do is use intermediate streams to count the number of website hits when this is greater than or equal to 2 , I see if it has already made a commercial action for that customer through various joins...
I think I 've become very complicated and do not know if there would be a simpler solution ??? , considering it does not have the function Siddhi NOT Happened nor other join ( left join )
You can accomplish this with a pattern. In this case i assume that we have to wait for 1 minute for an event from the second stream and if there's none, and more than 1 event from the first, we are going to emit an output.
from consults#window.time(1 minute)
select idClient, count(idClient) as idCount, <select more attributes here>
insert into expiredConsultsStream for expired-events;
from expiredConsultsStream[idCount > 1]
select *
insert into filteredConsultsStream;
from firstEvent = consults ->
nonOccurringEvent = commercialActions[firstEvent.idClient == idClient]
or
triggerEvent = filteredConsultsStream[firstEvent.idClient == idClient]
select firstEvent.idClient as id, triggerEvent.idCount as idCount, nonOccurringEvent.idClient as nid
having( not (nid instanceof string))
insert into alertStream;
These are draft queries, so may require some modifications to get them working. The filteredConsultsStream contains consult events with more than 1 occurrence within the last minute.
In the last query we get the or of the conditions as:
nonOccurringEvent = commercialActions[firstEvent.idClient == idClient]
or
triggerEvent = filteredConsultsStream[firstEvent.idClient == idClient]
So the query will be triggered by one of those above occurrences. But, then we need to find whether the condition is triggered by commercialActions. For that we use the 'having' clause and check whether the id is null (id is null implies that the event is null, the non-occurrence). Finally we emit the output.
You can find a better description for a somewhat similar query here (that is new 4.0.0 version btw and there are small syntax changes)