I have following Source in WSO2 Siddhi CEP:
#source(type='kafka',
topic.list='userInfo',
partition.no.list='0',
threading.option='single.thread',
group.id="threadAccessor",
bootstrap.servers='localhost:9092',
#map(type='json', #attributes(USERID = '$.USERID', tst = '$.TST', ACTIVITY = '$.ACTIVITY', AVG_HBR = '$.AVG_HBR')))
define stream SweetProductionStream (USERID string, tst long, ACTIVITY string, AVG_HBR int);
Is there a way to override value returned by function eventTimestamp() using mapping ? Is is possible at all ?
I know that it is for externalTimeWindow, but I would like to use my own timestamps for pattern finding.
Thanks.
Event's timestamp is immutable, you can only access its timestamp with eventTimestamp(). If you need to use your own timestamp then have it as an attribute in the stream and use that.
I'm not sure what you are trying to achieve in the source. You can pass the timestamp in the message and extract that, and use that as an attribute. You can pass this attribute when processing externalTimeWindow, and also when using patterns you can use the attribute and define time base conditions along with others.
Related
we all know that if we need to retrieve data from the database the data will back as a queryset but the question is How can I retrieve the data from database which is the name of it is queryset but remove that name from it.
maybe I can't be clarified enough in explanation so you can look at the next example to understand what I mean:
AnyObjects.objects.all().values()
this line will back the data like so:
<QuerySet [{'key': 'value'}]
now you can see the first name that is on the left side of retrieving data which is: "QuerySet" so, I need to remove that name to make the data as follows:
[{'key': 'value'}]
if you wonder about why so, the abbreviation of answer is I want to use Dataframe by pandas so, to put the data in Dataframe method I should use that layout.
any help please!!
You don't have to change it from a Queryset to anything else; pandas.DataFrame can take any Iterable as data. So
df = pandas.DataFrame(djangoapp.models.Model.objects.all().values())
Gives you the DataFrame you expect. (though you may want to double check df.dtypes. If there are Nones in your data, the column may end up to be of object type.)
You can use list(…) to convert it to a list of dictionaries:
list(AnyObjects.objects.values())
You will need to serialize it with the json package to obtain a JSON blob, since strings with single quotes are not valid JSON, in order to make it a JSON blob, you can work with:
import json
json.dumps(list(AnyObjects.object.values()))
I have a rather complex query that's generating a Django RawQuerySet. This specific query returns some fields that aren't part of the model that the RawQuerySet is based on, so I'm using .annotate(field_name=models.Value('field_name')) to attach it as an attribute to individual records in the RawQuerySet. The most important custom field is actually a uuid, which I use to compose URLs using Django's {% url %} functionality.
Here's the problem: I'm not using standard uuids inside my app, I'm using SmallUUIDs (compressed UUIDs.) These are stored in the database as native uuidfields then converted to shorter strings in python. So I need to somehow convert the uuid returned as part of the RawQuerySet to a SmallUUID for use inside a template to generate a URL.
My code looks somewhat like this:
query = "SELECT othertable.uuid_field as my_uuid FROM myapp_mymodel
JOIN othertable o ON myapp_mymodel.x = othertable.x"
MyModel.objects.annotate(
my_uuid=models.Value('my_uuid'),
).raw(query)
Now there is a logical solution here, there's an optional kwarg for models.Value called output_field, making the code look like this:
MyModel.objects.annotate(
my_uuid=models.Value('my_uuid', output_field=SmallUUIDField()),
).raw(query)
But it doesn't work! That kwarg is completely ignored and the type of the attribute is based on the type returned from the database and not what's in output_field. In my case, I'm getting a uuid output because Postgres is returning a UUID type, but if I were to change the query to SELECT cast othertable.uuid_field as text) as my_uuid I'd get the attribute in the format of a string. It appears that Django (at least version 1.11.12) doesn't actually care what is in that kwarg in this instance.
So here's what I'm thinking are my potential solutions, in no particular order:
Change the way the query is formatted somehow (either in Django or in the SQL)
Change the resulting RawQuerySet in some way before it's passed to the view
Change something inside the templates to convert the UUID to a smalluuid for use in the URL reverse process.
What's my best next steps here?
A couple of issues with your current approach:
Value() isn't doing what you think it is - your annotation is literally just annotating each row with the value "my_uuid" because that is what you have passed to it. It isn't looking up the field of that name (to do that you need to use F expressions).
Point 1 above doesn't matter anyway because as soon as you use raw() then the annotation is ignored - which is why you see no effect coming from it.
Bottom line is that trying to annotate a RawQuerySet isn't going to be easy. There is a translations argument that it accepts, but I can't think of a way to get that to work with the type of join you are using.
The next best suggestion that I can think of is that you just manually convert the field into a SmallUUID object when you need it - something like this:
from smalluuid import SmallUUID
objects = MyModel.objects.raw(query)
for o in objects:
# Take the hex string obtained from the database and convert it to a SmallUUID object.
# If your database has a built-in UUID type you will need to do
# SmallUUID(small=o.my_uuid) instead.
my_uuid = SmallUUID(hex=o.my_uuid)
(I'm doing this in a loop just to illustrate - depending on where you need this you can do it in a template tag or view).
Is there a way to query an object, 'extract' a nested piece of data from a JSONField field and then make it available as a custom, temporary field on each instance of the Queryset?
In my use case, I'm storing overflow metadata from Twitter's API in a data field for later use. I'd like to be able to access the nested field followers_count within TwitterPost.data.
I've read the docs about how to filter based on nested values but not how to extract it as a temporary field when generating a queryset.
Similarly, I've read the annotate docs for ways to create a custom temporary field but the examples all use aggregation functions on simple fields, so not JSONFields.
Thanks in advance for any suggestions.
Example model:
from django.contrib.postgres.fields import JSONField
class TwitterPost(models.Model):
id = models.IntegerField()
data = JSONField()
Example JSON value for the data field:
{
'followers_count': 7172,
"default_profile_image": false,
"profile_text_color": "000000"
}
Pseudocode for what I'd like to be able to do:
TwitterPost.objects.annotate(followers_count=instance.data.followers_count)
This is probably a late answer, but there is a way to do it
from django.contrib.postgres.fields.jsonb import KeyTransform
TwitterPost.objects.annotate(followers_count=KeyTransform('followers_count', 'data'))
OR KeyTextTransform could be used instead of KeyTransform (for converting to string)
If you want to access the data inside a JSONField, you've to use __. In your example it will be something like this
TwitterPost.objects.annotate(followers_count=instance.data__followers_count)
Take a look to the documentation here
I'm using Poco::Net:HTMLForm to POST a request.
Now I want to put in the form:
form[key] = value1
form[key] = value2
And the server side can transform key to a list [value1, value2].
I can do this while test with Postman, but in POCO HTMLForm, value2 will overwrite value1.
Is there any way to implement this in Poco::Net::HTMLForm?
You can use
form.add(name, value);
to add multiple fields with the same name.
No, as you noticed - same keys will be overwritten. Assembling data yourself is not that complicated, though:
?key=value1&key=value2
Issue filed on github.
UPDATE: use form.add(name, value), it will allow multiple values.
I want to build index from mysql table via DIH. One column is an integer type called "tags", which is used as a bitwise process query in mysql.
select * from mytable where (tags & 1) > 0
So, I intend to convert the "tags" into multiple values in data import handler, so that I can use BooleanQuery for better performance.
One solution might create a customized DIH template, could you please give me some advice?
Thanks.
Not sure I got your use case, However you can use the tags field and Use ScriptTransformer to convert it into multivalued fields
Example -
Data Config - Add custom field -
<script><![CDATA[
function addfield(row){
var fieldName = row.get('tags');
// Remove tags and Split/Convert tags as Array for Multivalued field
row.put(fieldName, tagsarray);
return row;
}
]]></script>