I inherited a Django application that serves high school sports information. I recently received a request to modify a certain set of data so information is displayed only from the current season, which I accomplished this way:
teams = Team.objects.filter(season__school_year__in=school_year_filter_list)
(school_year_filter_list contains [1,3])
This query causes an error in pgpool (a Postgres database pooling/replication utility) so I cannot use it. As a side note, the query works properly in the Python shell, and when I bypass pgpool and use Postgres directly. However, our network architecture dictates the use of pgpool, so I am trying to find an alternate way to retrieve the same data.
Can you help me determine another way to get all Teams with a Season in the current SchoolYear?
The (simplified) models look like this:
class Team(models.Model):
season = models.ForeignKey(Season)
class Season(models.Model):
school_year = models.ForeignKey(SchoolYear, blank=True, null=True)
class SchoolYear(models.Model):
school_year = models.CharField(max_length=150)
The '_schoolyear' table looks like:
id | school_year
----+-------------
1 | 2010-2011
2 | 2009-2010
3 | 2011-2012
In the end, I modified another model to make this work. Rather than parsing years to get active seasons, I added an "active" flag to the SchoolYear model and modified my query to check for that flag:
def queryset(self, request):
qs = super(PlayerYearAdmin, self).queryset(request)
return qs.filter(team__season__school_year__active=True)
Have you tried
teams = Team.objects.filter(season__school_year__id__in=school_year_filter_list)
?
Isn't it right that your school_year_filter_list is a list of school year ids?
Small note: I see 'django-admin' tag, so I think that you add season__school_year_in=[1, 2] to URL query string like /admin/sports/team/?season_school_year__in=[1, 2] Is it correct?
If so, are you shure that school_year_filter_list contains list [1,3], but not a string containing repr of list '[1,3]'
Can you provide us data about error that pgpool or Django ORM returns?
Related
Let's say we have the following simplistic models:
class Category(models.Model):
name = models.CharField(max_length=264)
def __str__(self):
return self.name
class Meta:
verbose_name_plural = "categories"
class Status(models.Model):
name = models.CharField(max_length=264)
def __str__(self):
return self.name
class Meta:
verbose_name_plural = "status"
class Product(models.Model):
title = models.CharField(max_length=264)
description = models.CharField(max_length=264)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
price = models.DecimalField(max_digits=10)
status = models.ForeignKey(Status, on_delete=models.CASCADE)
My aim is to get some statistics, like total products, total sales, average sales etc, based on which price bin each product belongs to.
So, the price bins could be something like 0-100, 100-500, 500-1000, etc.
I know how to use pandas to do something like that:
Binning column with python pandas
I am searching for a way to do this with the Django ORM.
One of my thoughts is to convert the queryset into a list and apply a function to get the apropriate price bin and then do the statistics.
Another thought which I am not sure how to impliment, is the same as the one above but just apply the bin function to the field in the queryset I am interested in.
There are three pathways I can see.
First is composing the SQL you want to use directly and putting it to your database with a modification of your models manager class. .objects.raw("[sql goes here]"). This answer shows how to define group with a simple function on the content - something like that could work?
SELECT FLOOR(grade/5.00)*5 As Grade,
COUNT(*) AS [Grade Count]
FROM TableName
GROUP BY FLOOR(Grade/5.00)*5
ORDER BY 1
Second is that there is no reason you can't move the queryset (with .values() or .values_list()) into a pandas dataframe or similar and then bin it, as you mentioned. There is probably a bit of an efficiency loss in terms of getting the queryset into a dataframe and then processing it, but I am not sure that it would certainly or always be bad. If its easier to compose and maintain, that might be fine.
The third way I would try (which I think is what you really want) is chaining .annotate() to label points with the bin they belong in, and the aggregate count function to count how many are in each bin. This is more advanced ORM work than I've done, but I think you'd start looking at something like the docs section on conditional aggregation. I've adapted this slightly to create the 'price_class' column first, with annotate.
Product.objects.annotate(price_class=floor(F('price')/100).aggregate(
class_zero=Count('pk', filter=Q(price_class=0)),
class_one=Count('pk', filter=Q(price_class=1)),
class_two=Count('pk', filter=Q(price_class=2)), # etc etc
)
I'm not sure if that 'floor' is going to work, and you may need 'expression wrapper' to ensure the push price_class into the write type of output_field. All the best.
models.py:
class Ingredient(models.Model):
_est_param = None
param = models.ManyToManyField(Establishment, blank=True, null=True, related_name='+', through='IngredientParam')
def est_param(self, establishment):
if not self._est_param:
self._est_param, created = self.ingredientparam_set\
.get_or_create(establishment=establishment)
return self._est_param
class IngredientParam(models.Model):
#ingredient params
active = models.BooleanField(default=False)
ingredient = models.ForeignKey(Ingredient)
establishment = models.ForeignKey(Establishment)
I need to fetch all Ingredient with parametrs for Establishment. First I fetch Ingredients.objects.all() and use all params like Ingredients.objects.all()[0].est_param(establishment).active. How I can use django 1.4 prefetch_related to make less sql queries? May be I can use other way to store individual Establishment properties for Ingredient?
Django 1.7 adds the Prefetch object you can put into prefetch_related. It allows you to specify a queryset which should provide the filtering. I'm having some problems with it at the moment for getting a singular (latest) entry from a list, but it seems to work very well when trying to get all the related entries.
You could also checkout django-prefetch which is part of this question which does not seem a duplicate of this question because of the vastly different wording.
The following code would fetch all the ingredients and their parameters in 2 queries:
ingredients = Ingredients.objects.all().prefetch_related('ingredientparam_set')
You could then access the parameters you're interested in without further database queries.
I'm trying to figure out how to design my model. I've been going over the documentation, and it ultimately seems like I should be using the "through" attribute, but I just can't figure out how to get it to work how I want.
If someone could take a look and point out what I'm missing, that would be really helpful. I have pasted my model below.
This is what I am trying to do:
1) Have a list of server types
2) Each server type will need to have different parts available to that specific server type
3) The asset has a FK to the servermodel, which has a M2M to the parts specific to that server type.
My question is, how can each "Asset" store meta data for each "Part" specific to that "Asset"? For example, each "Asset" should have it's own last_used data for the part that's assigned to it.
Thanks! :)
class Part(models.Model):
part_description = models.CharField(max_length=30,unique=1)
last_used = models.CharField(max_length=30)
def __unicode__(self):
return self.part_description
class ServerModel(models.Model):
server_model = models.CharField(max_length=30,unique=1)
parts = models.ManyToManyField(Part)
def __unicode__(self):
return self.server_model
class Asset(models.Model):
server_model = models.ForeignKey(ServerModel)
serial_number = models.CharField(max_length=10,unique=1)
def __unicode__(self):
return self.server_model.server_model
EDIT:
Thank you for the help!
I may have not explained myself clearly, though. It's probably my confusing model names.
Example:
ServerModel stores the type of server being used, say "Dell Server 2000".
The "Dell Server 2000" should be assigned specific parts:
"RAM"
"HARD DISK"
"CDROM"
Then, I should be able to create 10x Assets with a FK to the ServerModel. Now, each of these assets should be able to mark when the "RAM" part was last used for this specific asset.
I'm not sure I exactly understand what you want to do, but basically you can solve that with a "through" model, as you expected:
import datetime
class Part(models.Model):
name = models.CharField(max_length=30,unique=1)
class ServerModel(models.Model):
server_model = models.CharField(max_length=30,unique=1)
parts = models.ManyToManyField(Part,through='Asset')
class Asset(models.Model):
server_model = models.ForeignKey(ServerModel)
part = models.ForeignKey(Part)
serial_number = models.CharField(max_length=10,unique=1)
used = models.DateTimeField(default=datetime.datetime.now())
First thing to notice is the relation of the parts to the servermodel using the "through"-model: that way for each Part instance assigned to the "parts"-property of a ServerModel instance a new Asset instance is created (Phew - hope that doesn't sound too complicated). At the time of creation the "used"-property of the Asset instance is set to the current date and time (thats what default=datetime.datetime.now() does).
If you do that, you can then just query the database for the last asset containing your part. That queryset can then be sorted by the "used" property of the Asset model, which is the date when the Asset instance has been created.
ServerModel.objects.filter(parts__name='ThePartYouAreLookingFor').order_by('asset__used')
I'm not absolutely sure if the queryset is correct, so if someone finds huge nonsense in it, feel free to edit ;)
edit:
The models above do not exactly that. But you do not even need a through model for what you want:
class ServerModel(models.Model):
server_model = models.CharField(max_length=30,unique=1)
parts = models.ManyToManyField(Part)
class Asset(models.Model):
server_model = models.ForeignKey(ServerModel)
parts = models.ForeignKey(Part)
serial_number = models.CharField(max_length=10,unique=1)
used = models.DateTimeField(default=datetime.datetime.now())
Basically you can just add assets and then query all assets that have a RAM in parts.
Asset.objects.filter(parts__contains='RAM').order_by('used')
Get the date of the first (or last) result of that queryset and you have the date of the last usage of your 'RAM'-part.
I am trying to add an additional custom field to a django model. I have been having quite a hard time figuring out how to do the following, and I will be awarding a 150pt bounty for the first fully correct answer when it becomes available (after it is available -- see as a reference Improving Python/django view code).
I have the following model, with a custom def that returns a video count for each user --
class UserProfile(models.Model):
user = models.ForeignKey(User, unique=True)
positions = models.ManyToManyField('Position', through ='PositionTimestamp', blank=True)
def count(self):
from django.db import connection
cursor = connection.cursor()
cursor.execute(
"""SELECT (
SELECT COUNT(*)
FROM videos_video v
WHERE v.uploaded_by_id = p.id
OR EXISTS (
SELECT NULL
FROM videos_videocredit c
WHERE c.video_id = v.id
AND c.profile_id = p.id
)
) AS Total_credits
FROM userprofile_userprofile p
WHERE p.id = %d"""%(int(self.pk))
)
return int(cursor.fetchone()[0])
I want to be able to order by the count, i.e., UserProfile.objects.order_by('count'). Of course, I can't do that, which is why I'm asking this question.
Previously, I tried adding a custom model Manager, but the problem with that was I also need to be able to filter by various criteria of the UserProfile model: Specifically, I need to be able to do: UserProfile.objects.filter(positions=x).order_by('count'). In addition, I need to stay in the ORM (cannot have a raw sql output) and I do not want to put the filtering logic into the SQL, because there are various filters, and would require several statements.
How exactly would I do this? Thank you.
My reaction is that you're trying to take a bigger bite than you can chew. Break it into bite size pieces by giving yourself more primitives to work with.
You want to create these two pieces separately so you can call on them:
Does this user get credit for this video? return boolean
For how many videos does this user get credit? return int
Then use a combination of #property, model managers, querysets, and methods that make it easiest to express what you need.
For example you might attach the "credit" to the video model taking a user parameter, or the user model taking a video parameter, or a "credit" manager on users which adds a count of videos for which they have credit.
It's not trivial, but shouldn't be too tricky if you work for it.
"couldn't you use something like the "extra" queryset modifier?"
see the docs
I didn't put this in an answer at first because I wasn't sure it would actually work or if it was what you needed - it was more like a nudge in the (hopefully) right direction.
in the docs on that page there is an example
query
Blog.objects.extra(
select={
'entry_count': 'SELECT COUNT(*) FROM blog_entry WHERE blog_entry.blog_id = blog_blog.id'
},
)
resulting sql
SELECT blog_blog.*, (SELECT COUNT(*) FROM blog_entry WHERE blog_entry.blog_id = blog_blog.id) AS entry_count
FROM blog_blog;
Perhaps doing something like that and accessing the user id which you currently have as p.id as appname_userprofile.id
note:
Im just winging it so try to play around a bit.
perhaps use the shell to output the query as sql and see what you are getting.
models:
class Positions(models.Model):
x = models.IntegerField()
class Meta:
db_table = 'xtest_positions'
class UserProfile(models.Model):
user = models.ForeignKey(User, unique=True)
positions = models.ManyToManyField(Positions)
class Meta:
db_table = 'xtest_users'
class Video(models.Model):
usr = models.ForeignKey(UserProfile)
views = models.IntegerField()
class Meta:
db_table = 'xtest_video'
result:
test = UserProfile.objects.annotate(video_views=Sum('video__views')).order_by('video_views')
for t in test:
print t.video_views
doc: https://docs.djangoproject.com/en/dev/topics/db/aggregation/
This is either what you want, or I've completely misunderstood!.. Anywhoo... Hope it helps!
this is a model of the view table.
class QryDescChar(models.Model):
iid_id = models.IntegerField()
cid_id = models.IntegerField()
cs = models.CharField(max_length=10)
cid = models.IntegerField()
charname = models.CharField(max_length=50)
class Meta:
db_table = u'qry_desc_char'
this is the SQL i use to create the table
CREATE VIEW qry_desc_char as
SELECT
tbl_desc.iid_id,
tbl_desc.cid_id,
tbl_desc.cs,
tbl_char.cid,
tbl_char.charname
FROM tbl_desC,tbl_char
WHERE tbl_desc.cid_id = tbl_char.cid;
i dont know if i need a function in models or views or both. i want to get a list of objects from that database to display it. This might be easy but im new at Django and python so i having some problems
Django 1.1 brought in a new feature that you might find useful. You should be able to do something like:
class QryDescChar(models.Model):
iid_id = models.IntegerField()
cid_id = models.IntegerField()
cs = models.CharField(max_length=10)
cid = models.IntegerField()
charname = models.CharField(max_length=50)
class Meta:
db_table = u'qry_desc_char'
managed = False
The documentation for the managed Meta class option is here. A relevant quote:
If False, no database table creation
or deletion operations will be
performed for this model. This is
useful if the model represents an
existing table or a database view that
has been created by some other means.
This is the only difference when
managed is False. All other aspects of
model handling are exactly the same as
normal.
Once that is done, you should be able to use your model normally. To get a list of objects you'd do something like:
qry_desc_char_list = QryDescChar.objects.all()
To actually get the list into your template you might want to look at generic views, specifically the object_list view.
If your RDBMS lets you create writable views and the view you create has the exact structure than the table Django would create I guess that should work directly.
(This is an old question, but is an area that still trips people up and is still highly relevant to anyone using Django with a pre-existing, normalized schema.)
In your SELECT statement you will need to add a numeric "id" because Django expects one, even on an unmanaged model. You can use the row_number() window function to accomplish this if there isn't a guaranteed unique integer value on the row somewhere (and with views this is often the case).
In this case I'm using an ORDER BY clause with the window function, but you can do anything that's valid, and while you're at it you may as well use a clause that's useful to you in some way. Just make sure you do not try to use Django ORM dot references to relations because they look for the "id" column by default, and yours are fake.
Additionally I would consider renaming my output columns to something more meaningful if you're going to use it within an object. With those changes in place the query would look more like (of course, substitute your own terms for the "AS" clauses):
CREATE VIEW qry_desc_char as
SELECT
row_number() OVER (ORDER BY tbl_char.cid) AS id,
tbl_desc.iid_id AS iid_id,
tbl_desc.cid_id AS cid_id,
tbl_desc.cs AS a_better_name,
tbl_char.cid AS something_descriptive,
tbl_char.charname AS name
FROM tbl_desc,tbl_char
WHERE tbl_desc.cid_id = tbl_char.cid;
Once that is done, in Django your model could look like this:
class QryDescChar(models.Model):
iid_id = models.ForeignKey('WhateverIidIs', related_name='+',
db_column='iid_id', on_delete=models.DO_NOTHING)
cid_id = models.ForeignKey('WhateverCidIs', related_name='+',
db_column='cid_id', on_delete=models.DO_NOTHING)
a_better_name = models.CharField(max_length=10)
something_descriptive = models.IntegerField()
name = models.CharField(max_length=50)
class Meta:
managed = False
db_table = 'qry_desc_char'
You don't need the "_id" part on the end of the id column names, because you can declare the column name on the Django model with something more descriptive using the "db_column" argument as I did above (but here I only it to prevent Django from adding another "_id" to the end of cid_id and iid_id -- which added zero semantic value to your code). Also, note the "on_delete" argument. Django does its own thing when it comes to cascading deletes, and on an interesting data model you don't want this -- and when it comes to views you'll just get an error and an aborted transaction. Prior to Django 1.5 you have to patch it to make DO_NOTHING actually mean "do nothing" -- otherwise it will still try to (needlessly) query and collect all related objects before going through its delete cycle, and the query will fail, halting the entire operation.
Incidentally, I wrote an in-depth explanation of how to do this just the other day.
You are trying to fetch records from a view. This is not correct as a view does not map to a model, a table maps to a model.
You should use Django ORM to fetch QryDescChar objects. Please note that Django ORM will fetch them directly from the table. You can consult Django docs for extra() and select_related() methods which will allow you to fetch related data (data you want to get from the other table) in different ways.