Django - return all associated rows for distinct results set with postgres backend - django

I have data in the following form:
collection_name | type | manufacturer | description | image_url
---------------------------------------------------------------------------
beach | bed | company a | nice bed | 1.jpg
beach | king bed | company a | nice bed | 1.jpg
beach | nightstand | company a | nice ns | 1.jpg
grass | chest | company a | nice chest | 2.jpg
apple | chest | company a | nice chest | 3.jpg
fiver | chest | company b | good chest | 4.jpg
What I need to do, is select all images and only return each image once (distinct), but then return non-distinct row for each image.
My goal is to ensure I display each image only once in my template, but show all records associated with each image.
In the example above, 1.jpg is one image that would show both beds and the nightstand in one image. I would like to show such an image and list associated products with it.
I have seen similar questions, although asking at the SQL/db level and not asking for a pure django solution.
The query I have been using in my view has been something like:
products = product.objects.filter(collection_name=name)
and then iterating over products, retrieving image_url like so:
{% for instance in products %}
{{ instance.image_url }}
{{ endfor }}
I've tried various attempts to limit repeating images in my template, but none have really worked, and attempts to do so in my view have not been successful.
What is the correct way to approach this?
Edit: A relevant excerpt from my models to match the sample data above:
class Product(models.Model):
collection_name = models.TextField(null='true',blank='true')
type = models.TextField(null='true',blank='true')
manufacturer = models.TextField(null='true',blank='true')
description = models.TextField(null='true',blank='true')
image_url = models.TextField(null='true',blank='true')
Edit: My idea of views and logic to attempt to solve this, after reading the docs and looking at other questions (no answers):
Pass the product_id of any product in a collection to the view. Then obtain the collection_name field of a record based on the id field:
collectionname = product.objects.filter(id=id).values('collection_name').distinct()
Then, when we have the collection_name field, return all products for a given collection_name:
products = product.objects.filter(collection_name__in=collectionname)
Then, finally, return a list of image_url results for a given collection name, removing duplicates:
images = product.objects.filter(collection_name__in=collectionname).values('image_url').distinct()
I think this should work, in theory...
Edit:
Currently attempting the following based on Juancarlos' answer below:
products = product.objects.filter(collection_name=name)
collectionname = product.objects.filter(id=id).values('collection_name').distinct()
images = product.objects.filter(collection_name__in=collectionname).values("image_url").distinct()
results = []
for img in images:
pbis = product.objects.filter(collection_name__in=collectionname, image_url=img['image_url'])
obj = {"image": img['image_url'], "items":[{"type":pbi.type} for pbi in pbis]}
results.append(obj)

mabe this logic can help you, i am not sure but you can do something like this:
images = product.objects.filter(collection_name=name).values("image_url").distinct()
results = []
for img in images:
pbis = product.objects.filter(collection_name=name, image_url=img['image_url'])###this get all record tha contains this image
obj = {"image": img['image_url'], "items":[{"attr":pbi.attr, ...} for pbi in pbis]}
results.append(obj)
###this iterate all record by images and you can store items attribute from all recors that contains that image

You need tell to django what field you want distinct, you can use values to do that:
in your case your answer can seen like this:
products = product.objects.filter(collection_name=name).values("image_url").distinct()

Related

Django multi table on same model

I have huge table that needed to be sliced into some smaller table, ex:
campaign_01, campaign_02, ...
While using django queryset with different table name for same model, what I only know to set table name on a model is:
Model._meta.db_table = 'tableXXX'
However this method doesn't work in single shell/request. (only work for first time, but not for the next) -> maybe because it still on same instance?
After the second time we tried to set _meta.db_table = 'tableYYY', it will occur an error "django.db.utils.ProgrammingError: missing FROM-clause entry for table "tableXXX""
I also have tried some suggestion I read for this problem answer like:
class ListingManager(models.Manager):
def get_custom_obj(self, table_name):
self.model._meta.db_table = table_name
return self
class ObjectName(models.Model):
objects = ListingManager()
Try to create an Object manager to get new object, but it not work, it still throw same error as before (on the second time setting _meta.db_table)
The only way to make it work if we want to set multiple times for _meta.db_table is we need to exit() the shell first, then re-enter the shell mode (which means for loop is not gonna work).
I know it can be achieved with raw query 'Insert into tableXXX values ()', but any method to do it using django queryset? Thanks~
Consider creating a wrapper model.
class Model1(models.Model):
# fields...
name = ...
age = ...
class Model2(models.Model):
# fields...
height = ...
weight = ...
class ModelAll(models.Model):
model1 = models.OneToOneField(Model1)
model2 = models.OneToOneField(Model2)
But if you're only doing this for organization, just break the fields up with white space.
This will result in the following tables:
Model1
id | name | age
------------------
1 | "Joe" | 21
Model2
id | height | weight
----------------------
1 | 5.85 | 175
2 | 6.0 | 210
ModelAll
id | model1_id | model2_id
----------------------------
1 | 1 | 2
To access sub model fields:
modelall = ModelAll.objects.get(...)
modelall_name = modelall.model1.name
modelall_height = modelall.model2.height

Django: getting related model data in a specific order in the template

This is a continuation of a previous question.
I'm making a bug scrub website that pulls from a Bugzilla database. Developers make requests to backport bugs to different branches, and this website keeps track of the requests and the scrubbing progress.
The main model is Bugs which has all of the bug details copied from Bugzilla.
I also have a Branches model of all of the different software branches.
class Branch(models.Model):
name = models.CharField(max_length=20)
flag_id = models.PositiveSmallIntegerField()
flag_name = models.CharField(max_length=100)
version = models.CharField(max_length=10)
The main page on the website is a table. The view walks through the Branches and inserts a header cell with the branch name into a headers list in a specific order that is sent to the template.
Each bug will have one or more unlocks request for different branches. At first I created a separate field in the Bugs model for each branch name, but I don't want to have to edit the model/view/template each time we add or retire a branch.
So as an alternative I added a model called Flags, that has a FK to the bug_id, and the branch.
class Flags(models.Model):
bug_id = models.ForeignKey(Bugs,related_name='flaglines')
branch = models.ForeignKey(Branch)
I also have a model Scrub, which contains the scrub info for each bug.
class Scrub(models.Model):
bug_id = models.ForeignKey(Bugs,related_name='scrublines')
user = models.ForeignKey(User)
time = models.DateTimeField()
field = models.CharField(max_length=50)
value = models.CharField(max_length=255)
What I need to figure out how to do, is get the flags (and scrub) data from the related model, and place them in the correct order when the table gets displayed.
My current idea, is in the view, walk through each bug, and then walk through each branch (I can get them in the same order as the headers). For each branch, check if there is a flag and scrub, and then add them to a dictionary. If there were no flags or scrubs, the dictionary would just have empty strings as placeholders. The dictionary for each branch gets added to a 'flags' list that gets added to the queryset data for that bug.
Then in the template, for each bug, walk through each of the flag/scrub lists, and put the dict values into their proper cell in the table.
An example of what the table should look like is this:
| Bug | Branch 1 | Branch 2 | Branch 3| other fields...
----------------------------------------------------------
| 1234 | | ?(flag) | +(flag) | other data
| 2345 | ?(flag) | | ?(flag) | other data
| 3456 | | | ?(flag) | other data
I hope that is clear.
Is that the best way to accomplish this, or is there a better way?
Here's what I came up with, and it works, but I'm not sure if there is a more efficient way to do it.
branches = Branch.objects.filter(is_active=True)
bugs = Bugs.objects.filter(**filter_kwargs).order_by(*orderby_args)
for bug in bugs:
flags_list = [] # list to hold dictionaries
for branch in branches:
flags_dict = {'branch':branch} # dictionary for this branch
try:
flags_dict['flag'] = bug.flaglines.get(branch=branch)
except Flags.DoesNotExist:
flags_dict['flag'] = None
try:
flags_dict['scrub'] = bug.scrublines.get(field=branch.name).value
except Scrub.DoesNotExist:
flags_dict['scrub'] = None
flags_list.append(flags_dict)
bug.allflags = flags_list
Then in the template:
{% for bug in bugs %}
<tr class="bugrow" >
<td><a name="{{bug.bug_id}}">{{bug.bug_id}}</a></td>
{% for flag in bug.allflags %}
<td>{% if flag.flag %}{{flag.flag}} {%if flag.scrub%}<br>({{flag.scrub}}){%endif%}{%endif%}</td>
{% endfor %}

django paginate with non-model object

I'm working on a side project using python and Django. It's a website that tracks the price of some product from some website, then show all the historical price of products.
So, I have this class in Django:
class Product(models.Model):
price = models.FloatField()
date = models.DateTimeField(auto_now = True)
name = models.CharField()
Then, in my views.py, because I want to display products in a table, like so:
+----------+--------+--------+--------+--------+....
| Name | Date 1 | Date 2 | Date 3 |... |....
+----------+--------+--------+--------+--------+....
| Product1 | 100.0 | 120.0 | 70.0 | ... |....
+----------+--------+--------+--------+--------+....
...
I'm using the following class for rendering:
class ProductView(objects):
name = ""
price_history = {}
So that in my template, I can easily convert each product_view object into one table row. I'm also passing through context a sorted list of all available dates, for the purpose of constructing the head of the table, and getting the price of each product on that date.
Then I have logic in views that converts one or more products into this ProductView object. The logic looks something like this:
def conversion():
result_dict = {}
all_products = Product.objects.all()
for product in all_products:
if product.name in result_dict:
result_dict[product.name].append(product)
else:
result_dict[product.name] = [product]
# So result_dict will be like
# {"Product1":[product, product], "Product2":[product],...}
product_views = []
for products in result_dict.values():
# Logic that converts list of Product into ProductView, which is simple.
# Then I'm returning the product_views, sorted based on the price on the
# latest date, None if not available.
return sorted(product_views,
key = lambda x: get_latest_price(latest_date, x),
reverse = True)
As per Daniel Roseman and zymud, adding get_latest_price:
def get_latest_price(date, product_view):
if date in product_view.price_history:
return product_view.price_history[date]
else:
return None
I omitted the logic to get the latest date in conversion. I have a separate table that only records each date I run my price-collecting script that adds new data to the table. So the logic of getting latest date is essentially get the date in OpenDate table with highest ID.
So, the question is, when product grows to a huge amount, how do I paginate that product_views list? e.g. if I want to see 10 products in my web application, how to tell Django to only get those rows out of DB?
I can't (or don't know how to) use django.core.paginator.Paginator, because to create that 10 rows I want, Django needs to select all rows related to that 10 product names. But to figure out which 10 names to select, it first need to get all objects, then figure out which ones have the highest price on the latest date.
It seems to me the only solution would be to add something between Django and DB, like a cache, to store that ProductView objects. but other than that, is there a way to directly paginate produvt_views list?
I'm wondering if this makes sense:
The basic idea is, since I'll need to sort all product_views by the price on the "latest" date, I'll do that bit in DB first, and only get the list of product names to make it "paginatable". Then, I'll do a second DB query, to get all the products that have those product names, then construct that many product_views. Does it make sense?
To clear it a little bit, here comes the code:
So instead of
#def conversion():
all_products = Product.objects.all()
I'm doing this:
#def conversion():
# This would get me the latest available date
latest_date = OpenDate.objects.order_by('-id')[:1]
top_ten_priced_product_names = Product.objects
.filter(date__in = latest_date)
.order_by('-price')
.values_list('name', flat = True)[:10]
all_products_that_i_need = Product.objects
.filter(name__in = top_ten_priced_product_names)
# then I can construct that list of product_views using
# all_products_that_i_need
Then for pages after the first, I can modify that [:10] to say [10:10] or [20:10].
This makes the code pagination easier, and by pulling appropriate code into a separate function, it's also possible to do Ajax and all those fancy stuff.
But, here comes a problem: this solution needs three DB calls for every single query. Right now I'm running everything on the same box, but still I want to reduce this overhead to two(One or Opendate, the other for Product).
Is there a better solution that solves both the pagination problem and with two DB calls?

Need Modeling Help For An Ordering Form

I'd like to create a Django project for my company's purchasing department. This would be my first project in Django, so sorry if this comes off as rudimentary. The workflow would look something like this:
user registers for an account > signs in > can create, edit, view, or delete a purchase order.
I'm getting tripped up on the modeling. Presumably I can create and authenticate users using django.contrib.auth. Also, since this is mainly a form saving/printing application I would use a ModelForm to generate my forms based on my models since the users will be making changes to the form data that will need to be saved. A simplified version of the purchase order form in question looks something like this:
| Vendor | Date | Lead Time | Arrival Date | Buyer_Name |
+--------+-------+-----------+--------------+------------+
| FooBar |1-1-12 | 30 | 2-1-12 | Mr. Bar |
+--------+-------+-----------+--------------+------------+
+--------+-------+-----------+--------------+------------+
| SKU | Description | Quantity | Price | Dimensions |
+--------+-------------+----------+-------+--------------+
|12345 | Soft Bar | 38 | 5.75 | 16 X 5 X 8 |
+--------+-------------+----------+-------+--------------+
|12346 | Hard Bar | 12 | 5.75 | 16 X 5 X 8 |
+--------+-------------+----------+-------+--------------+
|12347 | Medium Bar | 17 | 5.75 | 16 X 5 X 8 |
+--------+-------------+----------+-------+--------------+
As you can see, the main purchase order form has a header that identifies the Vendor being ordered from, the current date, lead time, arrival date, and the buyer's name who is filling the form out. Under that is a line-by-line order detail for three different SKUs. Ideally, each PurchaseOrder should be able to have many SKUs added to it.
What is the best way to model something like this? Do I create a User, PurchaseOrder, and SKU model? Then add a FK to the SKU Model that points to the PurchaseOrder Model's PK or is there some other, more correct, way to do something like this? Thanks in advance for any help.
[Edit]
Django had what I was looking for all along. Since this is essentially a nested form, I could make use of Formsets.
Here are two helpful links to get started:
https://docs.djangoproject.com/en/1.4/topics/forms/formsets/
https://docs.djangoproject.com/en/1.4/topics/forms/modelforms/#model-formsets
Use django's built in user model (you can look at the source to see the definition but it is similar to the code below for these other models). Other than that I would suggest a model for every object you mentioned.
Don't add a FK to the SKU Model since SKU can exist without being in a purchase order (if I understand the problem correctly).
models.py
from django.contrib.auth.models import User
class Vendor(models.Model):
name = models.CharField(max_length=200)
#other fields
class SKU(models.Model):
description = models.CharField(max_length=200)
#other fields
class PurchaseOrder(models.Model):
purchaser = models.ForiegnKey(User)
name = models.CharField(max_length=200)
skus = models.ManyToManyField(SKU) #this is the magic that allows 1 purchase order to be filled with several SKUs
#other fields

Django QuerySet result with null fields in results

I have to get a QuerySet with many-to-many relations with the same number of results as if I executed the query in the database, but can't manage how to do this; I don't care if I can get the results as a QuerySet item or as a values item, but I do care to get the same number of results.
Imagine the following scenario:
class Person(models.Model):
name = models.CharField()
class Car(models.Model):
name = models.CharField()
class House(models.Model):
people = models.ManyToMany(Person)
cars = models.ManyToMany(Car)
house_1 = House.objects.create()
house_2 = House.objects.create()
john = Person.objects.create(name='John')
mary = Person.objects.create(name='Mary')
house_1.people.add(john)
house_1.people.add(mary)
mike = Person.objects.create(name='Mike')
ferrari = Car.objects.create(name='Ferrari')
house_2.people.add(mike)
house_2.cars.add(ferrari)
'''
Expected search result, regardless of the result format (model instances or values):
------------------------------------
| House ID | Car | Person |
| 1 | | John |
| 1 | | Mary |
| 2 | Ferrari | Mike |
------------------------------------
'''
How can I get a list of values, with all 3 results, spanning multiple tables, as here?
I need this so that I can create a report which can potentialy contain null fields, so the duplicated results must be listed.
Thanks!
Try to write SQL query that does that. You can't because it's wrong query to that data structure. Imagine that there will be 2 cars assigned to house 1. Should it be 1-[car-1]-John, 1-[car-2]-Merry or 1-[car-2]-John, 1-[car-1]-Merry?