I am a starter in Django/Python.
I have a model MyModel. It has many fields of datetime, char, integer types. Now, I want a values list which has values of the datetime fields containing only the date parts.
I have tried using the dates function [reference] (https://docs.djangoproject.com/en/1.2/ref/models/querysets/#dates), but it works only with one field. I have multiple datetime fields and all have to be retrieved in the required format.
Basically, I want a Django equivalent of :
select stringfield1, date(datefield1), integerfield1, date(datefield2) from mymodel; (PostGreSQL)
Is this even possible? If it is, how should I proceed further?
I'm not sure if Django has a builtin way of doing this. You could use itertools.imap to lazily convert the fields in question into date objects:
from itertools import imap
values = MyModel.objects.values_list('stringfield1', 'datefield1',
'integerfield1', 'datefield2')
values = imap(lambda (s, d1, i, d2): (s, d1.date(), i, d2.date()), values)
(But note that after this you're not dealing with a ValuesListQuerySet anymore but with an itertools.imap object.)
Related
I have some table ports(switch_ip, slot_number, port_number, many, more, columns) and would like to achieve the following PostgreSQL query using Django:
SELECT switch_ip, array_agg((slot_number, port_number, many, more, columns) ORDER BY slot_number, port_number) info
FROM ports
GROUP BY switch_ip
ORDER BY switch_ip
Using django.contrib.postgres.aggregates here's what I got so far:
Port.objects \
.values('switch_ip') \
.annotate(
info=ArrayAgg('slot_number', ordering=('slot_number', 'port_number'))
) \
.order_by('switch_ip')
I am unable to include more than one column in the ArrayAgg. None of ArrayAgg(a, b, c), ArrayAgg((a, b, c)), ArrayAgg([a, b, c]) seem to work. A workaround could involve separate ArrayAggs for each column and each with the same ordering. I would despise this because I have many columns. Is there any nicer workaround, possibly more low-level?
I suspect this is no issue with ArrayAgg itself but rather with tuple expressions in general. Is there any way to have tuples at all in Django queries? For example, what would be the corresponding Django of:
SELECT switch_ip, (slot_number, port_number, many, more, columns) info
FROM ports
If this is not yet possible in Django, how feasible would it be to implement?
I have spent lot of time searching for a working solution and here is a full recipe with code example.
You need to define Array "function" with square brackets in template
from django.db.models.expressions import Func
class Array(Func):
template = '%(function)s[%(expressions)s]'
function = 'ARRAY'
You need to define output field format (it must be array of some django field). For example an array of strings
from django.contrib.postgres.fields import ArrayField
from django.db.models.fields import CharField
out_format = ArrayField(CharField(max_length=200))
Finally make an ArrayAgg expression
from django.db.models import F
annotate = {'2-fields': ArrayAgg(Array(F('field1'), F('field2'), output_field=out_format), distinct=True) }
model.objects.all().annotate(**annotate)
(Optional) If field1 or field2 are not CharFields, you may include Cast as an argument of Array
from django.db.models.functions import Cast
annotate = {'2-fields': ArrayAgg(Array(Cast(F('field1'), output_field=CharField(max_length=200)), F('field2'), output_field=out_format), distinct=True) }
Having done a bit more research I guess one could add the missing tuple functionality as follows:
Create a new model field type named TupleField. The implementation might look kind of similar to django.contrib.postgres.fields.ArrayField. TupleField would be rather awkward because I don't think any RDBMS allows for composite types to be used as column types so usage of TupleField would be limited to (possibly intermediate?) query results.
Create a new subclass of django.db.models.Expression which wraps multiple expressions on its own (like Func in general, so looking at Func's implementation might be worthwile) and evaluates to a TupleField. Name this subclass TupleExpression for example.
Then I could simply annotate with ArrayAgg(TupleExpression('slot_number', 'port_number', 'many', 'more', 'columns'), ordering=('slot_number', 'port_number')) to solve my original problem. This would annotate each switch_ip with correctly-ordered arrays of tuples where each tuple represents one switch port.
I have data which is loaded into a dataframe. This dataframe then needs to be saved to a django model. The major problem is that some data which should go into IntegerField or FloatField are empty strings "". On the other side, some data which should be saved into a CharField is represented as np.nan. This leads to the following errors:
ValueError: Field 'position_lat' expected a number but got nan.
If I replace the np.nan with an empty string, using data[database]["df"].replace(np.nan, "", regex = True, inplace = True), I end up with the following error:
ValueError: Field 'position_lat' expected a number but got ''.
So what I would like to do, is to check in the model whether a FloatField or IntegerField gets either np.nan or an empty string and replace it with an empty value. The same for CharField, which should convert integers (if applicable) to strings or np.nan to an empty string.
How could this be implemented? Using ModelManager or customized fields? Or any better approaches? Sorting the CSV files out is not an option.
import pandas as pd
import numpy as np
from .models import Record
my_dataframe = pd.read_csv("data.csv")
record = Record
entries = []
for e in my_dataframe.T.to_dict().values():
entries.append(record(**e))
record.objects.bulk_create(entries)
Maybe the problem was not clear, nevertheless, I would like to post my solution. I create a new dict which only contain keys with corresponding values.
entries = []
for e in my_dataframe.T.to_dict().values():
e = {k: v for k, v in e.items() if v}
entries.append(record(**e))
record.objects.bulk_create(entries)
I have a problem with understating how to deal DateTimeField.
I have model with DateTimeField, view which returns jsons containing this field and another view which use provided data (as string) to filter result
# model
date = models.DateTimeField(auto_now=True)
# first view
return str((self.date.isoformat() for date in ...))
# f. exp: ['2019-11-19T15:22:47.788254+00:00']
# second view
Row.objects.filter(data__lte=data_from_GET)
If I have used 2019-11-19T15:22:47.788254+00:00 I reciver error
ValidationError at /csv/
["'2019-11-19T15:22:47.788254+00:00' value has an invalid format. It must be in YYYY-MM-DD HH:MM[:ss[.uuuuuu]][TZ] format."]
I can not find an easy way (without string manipulation) to return data in a format acceptable by the filter method.
What is interesting:
2019-11-19T15:22:47.788254-00:00 not ok
2019-11-19 15:22:47.788254+00:00 not ok
2019-11-19 15:22:47.788254-00:00 ok
You could use EpochUTC and integer values to compare different dates. If you need to display these dates you can transform them into strings using the new Date constructor.
clicks = SellerClick.objects.extra({'date' : "date(timestamp)"}).values('date').annotate(count=Count('timestamp'))
The model has a datetime field called timestamp that was are using. I first, convert the datetime field to just a date field. Then the rest is guessing. I need to group by, and then count how many objects are of each date.
So the desired result would be a date, then a count, based on how many objects have that date in the timestamp field.
I prefer to use annotate over extra
from django.db.models.expressions import RawSQL
SellerClick.objects.annotate(
date=RawSQL('date(date_joined)',[]),
).values('date').annotate(count=Count('date')))
You've got everything but an initial queryset there. The extra sql you're passing doesn't include a select so you need to give it something to act on.
clicks = SellerClick.objects.all()
.extra({'date' : "date(timestamp)"})
.values('date')
.annotate(count=Count('timestamp'))
Ref: StackOverflow: Count number of records by date in Django
in models i have start and end date.
How to get all element where start and end date are diffrents.
>>> Entry.objects.exclude(start = end)
>>> NameError: name 'end' is not defined
I have no idea please help.
https://docs.djangoproject.com/en/dev/topics/db/queries/#filters-can-reference-fields-on-the-model
In the examples given so far, we have constructed filters that compare the value of a model field with a constant. But what if you want to compare the value of a model field with another field on the same model?
Django provides the F() object to allow such comparisons. Instances of F() act as a reference to a model field within a query. These references can then be used in query filters to compare the values of two different fields on the same model instance.
For your case, the following should work.
from django.db.models import F
Entry.objects.exclude(start=F('end'))