I'm using django-nonrel and mongodb to develop app. I know that object id is start with a timestamp of the insertion time of object creation. So it's possible to do time range query based on _id field.
How can I generate a minimal object_id based on a given time in python or django?
Here is a much more pythonic version of the other answer here provided by OP, along with documentation:
from bson.objectid import ObjectId
import datetime
def datetime_to_objectid(dt):
# ObjectId is a 12-byte BSON type, constructed using:
# a 4-byte value representing the seconds since the Unix epoch,
# a 3-byte machine identifier,
# a 2-byte process id, and
# a 3-byte counter, starting with a random value.
timestamp = int((dt - datetime.datetime(1970,1,1)).total_seconds())
time_bytes = format(timestamp, 'x') #4 bytes
return ObjectId(time_bytes+'00'*8) #+8 bytes
However, starting with version 1.6 of pymongo, it would be much more elegant to do the following:
from bson.objectid import ObjectId
ObjectId.from_datetime(dt)
from bson.objectid import ObjectId
import time
def get_minimal_object_id_for_int_timestamp(int_timestamp=None):
if not int_timestamp:
int_timestamp=int(time.time())
return ObjectId(hex(int(int_timestamp))[2:]+'0000000000000000')
def get_int_timestamp_from_time_string(time_string=None):
# format "YYYY-MM-DD hh:mm:ss" like '2012-01-05 13:01:51'
if not time_string:
return int(time.time())
return int(time.mktime(time.strptime(time_string, '%Y-%m-%d %H:%M:%S')))
def get_minimal_object_id_for_time_string(time_string=None):
return get_minimal_object_id_for_int_timestamp(get_int_timestamp_from_time_string(time_string=time_string))
I find the solution finally. hope it helps to others.
Related
I have a one to many relation between session and camp. Now I have to get the max and min dates of all camps combined for a particular session.
I am able to do it like this:
sess = Session.objects.last()
max_min_dates = sess.camp.aggregate(Min('start_date'), Max('end_date'))
But if I try to send this from HttpResponse then I am getting this error:
TypeError: Object of type 'date' is not JSON serializable
So I need to send the formatted date values in that. How can I modify the above code to get the same?
The default encoder for json.dumps() does not support date encoding (ie. can't convert date into str). You can use django encoder instead, it supports a few more data types see the link for more info.
Django Ref
import json
from django.core.serializers.json import DjangoJSONEncoder
json_str = json.dumps(max_min_dates, cls=DjangoJSONEncoder)
I'm trying to create a TimeRangeField but am having some problems saving to the database (PostgreSQL version 9.5.23).
from psycopg2.extras import DateTimeRange
from django import forms
from django.contrib.postgres.forms import BaseRangeField
from django.contrib.postgres.fields.ranges import RangeField
from django.db import models
class TimeRangeFormField(BaseRangeField):
default_error_messages = {'invalid': 'Enter two valid times.'}
base_field = forms.TimeField
range_type = DateTimeRange
class TimeRangeField(RangeField):
base_field = models.TimeField
range_type = DateTimeRange
form_field = TimeRangeFormField
def db_type(self, connection):
return 'tsrange'
The error when saving seems pretty self explanatory - pretty sure I need to cast the time object to a string but I have no idea how to do that.
function tsrange(time without time zone, time without time zone, unknown) does not exist
LINE 1: ...('b9925dd3-d4a8-4914-8e85-7380d9a33de5'::uuid, 1, tsrange('1...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
Here's the SQL:
INSERT INTO "fulfillment_deliveryslot"
(
"uuid",
"day_id",
"window",
"max_orders",
"is_active"
)
VALUES
(
'b9925dd3-d4a8-4914-8e85-7380d9a33de5'::uuid,
1,
tsrange('12:38:31'::time, '13:00:00'::time, '[)'),
5,
true
)
returning "fulfillment_deliveryslot"."id"
The issue is here: time without time zone. You are using tsrange and that needs a timestamp:
https://www.postgresql.org/docs/9.5/rangetypes.html
tsrange — Range of timestamp without time zone
In other words you are passing a time into something that needs a timestamp.
My mobile client is sending up inaccurate datetime offset information. For example:
2019-05-03T17:55:12-0700
The time is actually the correct UTC time however, the offset should read -0000.
I can not currently modify the client to correct the issue causing this. So I need to throw out the offset or change it to -0000.
In the above example, for this user who has their account timezone settings set to PST, it stores the date in validated_data as datetime.datetime(2019, 5, 4, 0, 55, 12, tzinfo=<UTC>)
If client-based time and offset information were synced up, this conversion by DRF would be correct, as it is seven hours off or PST + the current DST. (west coast us is currently -7:00 UTC)
The problem is that by the time I reach my ModelSerializer class, the validated_data already contains what DRF believes is now the correct UTC time.
Where is the appropriate place to mutate this field on the POST body so that by the time DRF attempts to create the DateTime it will build the correct timestamp?
The easiest way to handle this was to modify the DateTime object before it was saved in the model field. So in the create() get the original ISO8601 string from the POST body using self.context['request'].data['created']. Then use dateutil.parser to parse the string into a DateTime, and replace the timezone with a pytz.UTC.
import dateutil.parser
Class MySerializer(serializers.ModelSerializer):
...
def create(self, validated_data):
scan.created = dateutil.parser.parse(self.context['request'].data['created']).replace(tzinfo=pytz.UTC)
It is not a glorious fix, but will work until I can release an updated mobile client.
My goal is to make an alert in ElastAlert for this scenario: no events has occured between midnight and 2 am. (for any date). The problem is how to make a query to Elasticsearch that matches any date but a specific time, because you cannot use regexp or wildcard on timestamp of type 'date'. Any suggestions?
This code returns "Parse failure":
"range": {
"timestamp": {
"gte": "20[0-9]{2}-[0-9]{2}-[0-9]{2}T00:00:00.000Z",
"lt": "20[0-9]{2}-[0-9]{2}-[0-9]{2}T02:00:00.000Z"
}
}
Handling it in a custom rule is ideal.
I wrote the following to do the same kind of filtering:
Note, the dependencies used (dateutil, elastalert.utils) are already bundled with the elastalert framework.
import dateutil.parser
from ruletypes import RuleType
# elastalert.util includes useful utility functions
# such as converting from timestamp to datetime obj
from util import ts_to_dt
# Modified version of http://elastalert.readthedocs.io/en/latest/recipes/adding_rules.html#tutorial
# to catch events happening outside a certain time range
class OutOfTimeRangeRule(RuleType):
""" Match if input time is outside the given range """
# Time range specified by including the following properties in the rule:
required_options = set(['time_start', 'time_end'])
# add_data will be called each time Elasticsearch is queried.
# data is a list of documents from Elasticsearch, sorted by timestamp,
# including all the fields that the config specifies with "include"
def add_data(self, data):
for document in data:
# Convert the timestamp to a time object
login_time = document['#timestamp'].time()
# Convert time_start and time_end to time objects
time_start = dateutil.parser.parse(self.rules['time_start']).time()
time_end = dateutil.parser.parse(self.rules['time_end']).time()
# If time is outside office hours
if login_time < time_start or login_time > time_end:
# To add a match, use self.add_match
self.add_match(document)
# The results of get_match_str will appear in the alert text
def get_match_str(self, match):
return "logged in outside %s and %s" % (self.rules['time_start'], self.rules['time_end'])
def garbage_collect(self, timestamp):
pass
I didn't have the right to write custom rules, so my solution was to make changes in logstash. Added the field hour_of_day, where the value is derived from the timestamp. Thus we are able to create a flatline rule with a filter like this:
filter:
- query:
query_string:
query: "hour_of_day: 0 OR hour_of_day: 1"
hi i'm new in working with datetime objects in django
all I know now is that instead of python's datetime.datetime.now() we should use django's timezone.now(), i've also set TIMEZONE and USE_TZ=True in settings.py
but my problem is now for converting these types of time. as far as I know, even if we use timezone.now() for saving in database, django uses UTC time to store in DB. so I need a simple syntax for converting UTC time into my local time which is set in settings.py and vice versa to get local time from human and return local time.
i've also seen that django has some template tags to do that, but since i am doing this mostly for a REST API with django-rest for an android app, i need to be able to do this in python syntax.
thanks everyone, I hope I could be clear at what I mean :)
In templates, Django will automatically convert your model dates (stored as UTC) to the current time zone. The current time zone is set by settings.TIMEZONE unless you explicitly change it somewhere else. You don't even need to use special template tags. This will convert fine:
{{ MyModel.my_date }}
Outside of templates, there is a tool called localtime that you can use to do the same conversion.
from django.utils.timezone import localtime
...
local_date = localtime(MyModel.my_date)
print( str(MyModel.my_date) ) # UTF time
print( str(local_date) ) # local time
The datetime returned by localtime is time zone aware. If you ever need a time zone naive datetime, you can convert it like this:
my_date = localtime(MyModel.my_date).replace(tzinfo=None)
If, in settings.py we have the following:
from pytz import timezone
LOCAL_TZ = pytz.timezone('CST6CDT') # asume that local timezone is central, but you can use whatever is accurate for your local
Now, you can use this to convert from utc time to local
import pytz
from django.conf import settings
def to_local_dttm(utc_dttm):
return utc_dttm.astimezone(settings.LOCAL_TZ)
def to_utc_dttm(local_dttm):
return local_dttm.astimezone(pytz.utc)