I'm really stuck in how to do this and appreciate the help. I have three models, two of them shares the same foreign key field:
class AccountsPlan (models.Model):
code = models.CharField(max_length=7, unique=True,)
name = models.CharField(max_length=100, unique=True,)
class Planning (models.Model):
accountplan = models.ForeignKey(AccountsPlan, on_delete=models.PROTECT)
month = models.DateField()
amount = models.DecimalField(max_digits=14, decimal_places=2,)
class Revenue (models.Model):
accountplan = models.ForeignKey(AccountsPlan, on_delete=models.PROTECT)
receipt_date = models.DateField()
amount = models.DecimalField(max_digits=14, decimal_places=2,)
And I have this view that annotates the sum for each model by the foreign key name with a form that filters by date:
def proj_planning(request):
form = ProjectionFilterForm(request.POST or None)
# some definitions for the form
planned = Planning.objects.values('accountplan__name').annotate(Sum('amount')).order_by('accountplan__code').filter(
month__range=[start_date, planned_end_date])
done = Revenue.objects.values('accountplan__name').annotate(Sum('amount')).filter(
receipt_date__range=[start_date, end_date],).order_by('accountplan__code')
if request.method == 'POST':
planned = Planning.objects.values('accountplan__name').annotate(Sum('amount')).order_by(
'accountplan__code').filter(month__range=[start_date, planned_end_date],
accountplan__name__icontains=form['accountplan'].value())
done = Revenue.objects.values('accountplan__name').annotate(
Sum('amount')).filter(receipt_date__range=[start_date, end_date],
accountplan__name__icontains=form['accountplan'].value())
comp_zippedlist = zip(planned, done)
return render(request, 'confi/pages/planning/proj_planning.html', context={
'form': form,
'list': comp_zippedlist,
})
The code kinda works (it doesn't throw any errors), but the thing is, I will never have an exact amount of data for each model.
For example: if I have 6 different records in the Planning model, but only 4 records in the Revenues model, the zipped list will only show the 4 records in the Revenues and not the other 2 records in the Planning model, which makes sense.
But what I need is to show all 6 records from the Planning model with the corresponding 4 records from the Revenue model and the other 2, blank or something like that. Ultimately, I will need to compare the two values and show some kind of status based on the two amount field values, that's why I have to show all of them.
The result I expect is something like this:
Accountplan | Planned | Done
__________________________________
Accountplan A | 500 | 250
Accountplan B | 1000 | 860
Accountplan C | 800 | None
Accountplan D | 150 | 350
Accountplan E | 300 | None
Accountplan F | 700 | 900
I tried to use prefetch_related to do this, using this code:
done = AccountsPlan.objects.prefetch_related('revenue_set').values(
'name').annotate(Sum('revenue__amount')).order_by('code')
And while it shows exactly what I want, if I apply a filter in the receipt_date field of the Revenue model, it will have the same result as before, witch again, make sense.
I thought of looping through every revenue record with no accountplan and apply a temporary value 0 to them, but the accountplan is required, so that idea made no sense.
I also looked at the multi-table inheritance but since the relation with the child models is one-to-one, it does not apply in this case (unless I misread something).
Is there a way to have the behavior described and possible with the prefetch_related code, but keeping all foreign key fiels after a filter? Thanks!
I am using Django to interface with another (JAVA) application that based on some events generates at runtime tables in the db with the same model. So I don't have a direct control over the DB. For example:
Sensor1
id | value | time
1 10 2018-10-11
Sensor2
id | value | time
1 12 2018-10-11
Currently, my Django model is looking something like this:
class Sensor(models.Model):
value = models.IntegerField()
time = models.DatetimeField()
class Meta:
managed = False
db_table = "Sensor1"
Do you have any clue if I can set the model somehow to be able to gain the data from a different table based on the query?
Ideally, something that would allow me to get the data in the fashion of:
config_tables=['Sensor1','Sensor2']
for table in config_tables:
data = Sensor.objects.table(table).objects.all()
...
Other possibility might be also to have a SQL query that executes on different tables, so perhaps something like:
SELECT * FROM %s;
It seems that the best solution so far is to make a custom SQL Query so in this example it could be something like this in models.py:
def get_all_parameters(db_table):
return Parameters.objects.raw('SELECT * FROM %s' % db_table)
and call it as:
get_all_parameters('Sensor1')
or as:
TABLES = ['Sensor1', 'Sensor2']
for table in TABLES:
parameters = get_all_parameters(table)
...
class Customer(models.Model):
name = models.CharField(max_length=200)
# ..
class CustomerTicket(models.Model):
customer = models.OneToOneField(Customer)
date = models.DateTimeField(auto_now_add=True)
# ..
I want to query all customers. And, adding for each customer its ticket if it has one in the date range - so I will get the ticket object only if it is in the given date range, otherwise the ticket field would be null.
Try this:
from django.db import models
customers = Customer.objects.select_related('customerticket').annotate(
ticket=models.Case(models.When(models.Q(customerticket__date__gt=date1) & models.Q(customerticket__date__lt=date2), then=models.F('customerticket')))
)
And you will get ticket as a computed field. Note that when referencing relational fields such as ForeignKey or OneToOneField, F() returns the primary key value rather than a model instance, which means your ticket field will have value of the primary key.
To get customers with related tickets with one query you can use select_related. To make complex condition you can use Q
from django.db.models import Q
Customer.objects.select_related('customerticket').filter(Q(customerticket__date__range=["2018-04-11", "2018-04-12"]) | Q(customerticket__isnull=True))
This will filter data by customerticket date.
Use queryset.filter:
from django.utils import timezone
Customer.objects.exclude(customerticket=None).filter(customerticket__date=timezone.now())
I have the following models:
class Item(models.model):
price = models.floatField()
...
and :
class purchase(models.model):
user = models.ForeignKey(User)
item = models.ForeignKey(Item)
date = models.DateField()
I would like to make a query which will provide top 10 users who purchased the most in the last week .I would also like to have the Sum of the purchased they completed in that week.
So the output I would like to show on my page is similar to this :
Top buyers of the week :
1. User_1 | 150 Points
2. User_2 | 130 Points
...
10. User_10 | 10 Points
Is it possible to use annotate for that ? or it should be break ot several queries ?
Well let's give it a go (still needs to be tested for optimized SQL):
from datetime import timedelta
from django.db.models import Sum
from django.contrib.auth.models import User
from django.utils import timezone
some_day_last_week = timezone.now().date() - timedelta(days=7)
rows = User.objects.filter(purchases__date__gte=some_day_last_week)\
.annotate(item_sum=Sum('purchases__item__price'))\
.order_by('item_sum')[:10]
print [(u.username, u.item_sum) for u in rows]
I would like to save a queryset criteria to the DB for reuse.
So, if I have a queryset like:
Client.objects.filter(state='AL')
# I'm simplifying the problem for readability. In reality I could have
# a very complex queryset, with multiple filters, excludes and even Q() objects.
I would like to save to the DB not the results of the queryset (i.e. the individual client records that have a state field matching 'AL'); but the queryset itself (i.e. the criteria used in filtering the Client model).
The ultimate goal is to have a "saved filter" that can be read from the DB and used by multiple django applications.
At first I thought I could serialize the queryset and save that. But serializing a queryset actually executes the query - and then I end up with a static list of clients in Alabama at the time of serialization. I want the list to be dynamic (i.e. each time I read the queryset from the DB it should execute and retrieve the most current list of clients in Alabama).
Edit: Alternatively, is it possible to obtain a list of filters applied to a queryset?
Something like:
qs = Client.objects.filter(state='AL')
filters = qs.getFilters()
print filters
{ 'state': 'AL' }
You can do as jcd says, storing the sql.
You can also store the conditions.
In [44]: q=Q( Q(content_type__model="User") | Q(content_type__model="Group"),content_type__app_label="auth")
In [45]: c={'name__startswith':'Can add'}
In [46]: Permission.objects.filter(q).filter(**c)
Out[46]: [<Permission: auth | group | Can add group>, <Permission: auth | user | Can add user>]
In [48]: q2=Q( Q(content_type__model="User") | Q(content_type__model="Group"),content_type__app_label="auth", name__startswith='Can add')
In [49]: Permission.objects.filter(q2)
Out[49]: [<Permission: auth | group | Can add group>, <Permission: auth | user | Can add user>]
In that example you see that the conditions are the objects c and q (although they can be joined in one object, q2). You can then serialize these objects and store them on the database as strings.
--edit--
If you need to have all the conditions on a single database record, you can store them in a dictionary
{'filter_conditions': (cond_1, cond_2, cond_3), 'exclude_conditions': (cond_4, cond_5)}
and then serialize the dictionary.
You can store the sql generated by the query using the queryset's _as_sql() method. The method takes a database connection as an argument, so you'd do:
from app.models import MyModel
from django.db import connection
qs = MyModel.filter(pk__gt=56, published_date__lt=datetime.now())
store_query(qs._as_sql(connection))
You can use http://github.com/denz/django-stored-queryset for that
You can pickle the Query object (not the QuerySet):
>>> import pickle
>>> query = pickle.loads(s) # Assuming 's' is the pickled string.
>>> qs = MyModel.objects.all()
>>> qs.query = query # Restore the original 'query'.
Docs: https://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets
But: You can’t share pickles between versions
you can create your own model to store your queries.
First field can contains fk to ContentTypes
Second field can be just text field with your query etc.
And after that you can use Q object to set queryset for your model.
The current answer was unclear to me as I don't have much experience with pickle. In 2022, I've found that turning a dict into JSON worked well. I'll show you what I did below. I believe pickling still works, so at the end I will show some more thoughts there.
models.py - example database structure
class Transaction(models.Model):
id = models.CharField(max_length=24, primary_key=True)
date = models.DateField(null=False)
amount = models.IntegerField(null=False)
info = models.CharField()
account = models.ForiegnKey(Account, on_delete=models.SET_NULL, null=True)
category = models.ForeignKey(Category, on_delete=models.SET_NULL, null=True, blank=False, default=None)
class Account(models.Model):
name = models.CharField()
email = models.EmailField()
class Category(models.Model):
name = models.CharField(unique=True)
class Rule(models.Model):
category = models.ForeignKey(Category, on_delete=models.SET_NULL, blank=False, null=True, default=None)
criteria = models.JSONField(default=dict) # this will hold our query
My models store financial transactions, the category the transaction fits into (e.g., salaried income, 1099 income, office expenses, labor expenses, etc...), and a rule to save a query to automatically categorize future transactions without having to remember the query every year when doing taxes.
I know, for example, that all my transactions with my consulting clients should be marked as 1099 income. So I want to create a rule for clients that will grab each monthly transaction and mark it as 1099 income.
Making the query the old-fashioned way
>>> from transactions.models import Category, Rule, Transaction
>>>
>>> client1_transactions = Transaction.objects.filter(account__name="Client One")
<QuerySet [<Transaction: Transaction object (1111111)>, <Transaction: Transaction object (1111112)>, <Transaction: Transaction object (1111113)...]>
>>> client1_transactions.count()
12
Twelve transactions, one for each month. Beautiful.
But how do we save this to the database?
Save query to database in JSONField
We now have Django 4.0 and a bunch of support for JSONField.
I've been able to grab the filtering values out of a form POST request, then add them in view logic.
urls.py
from transactions import views
app_name = "transactions"
urlpatterns = [
path("categorize", views.categorize, name="categorize"),
path("", views.list, name="list"),
]
transactions/list.html
<form action="{% url 'transactions:categorize' %}" method="POST">
{% csrf_token %}
<label for="info">Info field contains...</label>
<input id="info" type="text" name="info">
<label for="account">Account name contains...</label>
<input id="account" type="text" name="account">
<label for="category">New category should be...</label>
<input id="category" type="text" name="category">
<button type="submit">Make a Rule</button>
</form>
views.py
def categorize(request):
# get POST data from our form
info = request.POST.get("info", "")
account = request.POST.get("account", "")
category = request.POST.get("category", "")
# set up query
query = {}
if info:
query["info__icontains"] = info
if account:
query["account__name__icontains"] = account
# update the database
category_obj, _ = Category.objects.get_or_create(name=category)
transactions = Transaction.objects.filter(**query).order_by("-date")
Rule.objects.get_or_create(category=category_obj, criteria=query)
transactions.update(category=category_obj)
# render the template
return render(
request,
"transactions/list.html",
{
"transactions": transactions.select_related("account"),
},
)
That's pretty much it!
My example here is a little contrived, so please forgive any errors.
How to do it with pickle
I actually lied before. I have a little experience with pickle and I do like it, but I am not sure on how to save it to the database. My guess is that you'd then save the pickled string to a BinaryField.
Perhaps something like this:
>>> # imports
>>> import pickle # standard library
>>> from transactions.models import Category, Rule, Transaction # my own stuff
>>>
>>> # create the query
>>> qs_to_save = Transaction.objects.filter(account__name="Client 1")
>>> qs_to_save.count()
12
>>>
>>> # create the pickle
>>> saved_pickle = pickle.dumps(qs_to_save.query)
>>> type(saved_pickle)
<class 'bytes'>
>>>
>>> # save to database
>>> # make sure `criteria = models.BinaryField()` above in models.py
>>> # I'm unsure about this
>>> test_category, _ = Category.objects.get_or_create(name="Test Category")
>>> test_rule = Rule.objects.create(category=test_category, criteria=saved_pickle)
>>>
>>> # remake queryset at a later date
>>> new_qs = Transaction.objects.all()
>>> new_qs.query = pickle.loads(test_rule.criteria)
>>> new_qs.count()
12
Going even further beyond
I found a way to make this all work with my htmx live search, allowing me to see the results of my query on the front end of my site before saving.
This answer is already too long, so here's a link to a post if you care about that: Saving a Django Query to the Database.