i'm struggling with complicated Django queries, mainly because I try to do something that might be complicated.
Basically, I got this :
models.py :
from django.contrib.gis.db import models
class Show(models.Model):
name = models.CharField()
class Venue(models.Models):
name = models.CharField()
coords = models.PointField()
objects = models.GeoManager()
class Representation(models.Model):
datetime = models.DateTimeField()
venue = models.ForeignKey(Venue)
show = models.ForeignKey(Show)
Now, what I want to do is, get 5 incoming shows that are close to user (user.geoloc is a Point). One of the complicated thing is that some of my users may live in area where there are no venue, and, my solution is just, if there is not enough shows close to them, to search in a bigger area.
view.py :
from django.contrib.gis.measure import D
DISTANCE_CLOSE = 1000 #arbitrary number
## this thing is not working, because it's not how it should be done
## but the logic is clearer in this
def get_nearest_shows_not_working(request):
list_shows = {}
while len(list_shows<5)
list_shows = Show.representation_set.filter(venue__coords__distance_lte=(user.geoloc, D(m=DISTANCE_CLOSE))).order_by('datetime'):[5]
DISTANCE_CLOSE = int(DISTANCE_CLOSE*1.2)
return render(request, 'template.html', locals())
def get_nearest_shows_ducktape(request):
list_shows = set()
while len(list_show) < 5:
list_rep = Representation.objects.filter(venue__coords__distance_lte=(user.geoloc, D(m=DISTANCE_CLOSE))).order_by('datetime')
for rep in list_rep:
list_shows.add(rep.show)
DISTANCE_CLOSE = int(DISTANCE_CLOSE*1.2)
list_shows = list_shows[:5]
return render(request, 'template.html', locals())
What am I missing ? in Python, there shoud be one and only right way to do something, and here, I'm just messing with complicated things, this looks unpythonic to me.
You would need something like this:
Show.representation_set.distance(user.geoloc, field_name='venue__coords') \
.order_by('distance')
Unfortunately reversed one to one or one to many relationships are not supported with GeoQuerySet methods (including distance). I already created a feature request ticket for this.
So for now I think you're stuck with:
some hack in Python, not always efficient (like you wrote)
using raw sql queries
Related
The below model have EditorManager,
class EditorManager(models.Manager):
def get_queryset(self):
return super().get_queryset().filter(role='E')
class Person(models.Model):
first_name = models.CharField(max_length=50)
role = models.CharField(max_length=1, choices=[('A', _('Author')), ('E', _('Editor'))])
people = models.Manager()
editors = EditorManager()
If I query Person.objects.filter(role='E') or Person.editors.all() I gets same result.
then, Why do we go for writing EditorManager() ?
The above code is from Django documentation (https://docs.djangoproject.com/en/3.0/topics/db/managers/).
As mentioned in the Documentation:
using multiple managers on the same model. You can attach as many Manager() instances to a model as you’d like. This is a non-repetitive way to define common “filters” for your models.
Since you just have one action, it may be hard for you to see the benefits. However, as your code gets larger, say:
good = Book.objects.filter(author="PersonA", stars=5).order_by("-date_created").exclude(outdated=True)
normal = Book.objects.filter(author="PersonA", stars=3).order_by("-date_created").exclude(outdated=True)
bad = Book.objects.filter(author="PersonA", stars=1).order_by("-date_created").exclude(outdated=True)
You can see that's an awful lot of code. With managers, you can do something like this:
class AuthorAManager(models.Manager):
def get_queryset(self):
return super().get_queryset().filter(author="PersonA").order_by("-date_created").exclude(outdated=True)
class Book(models.Model):
# ...
author_a = AuthorAManager()
good = Book.author_a.filter(stars=5)
normal = Book.author_a.filter(stars=3)
bad = Book.author_a.filter(stars=1)
Overall, it can make your code look a lot cleaner and understandable. As you said, you can't see the difference right now as you haven't gone into complex/repeating handles, but as your project expands, I'd say it's a worthwhile investment.
I have been struggling all morning trying to figure out how to compare two different querysets. I have two manytomanyfields in my model, and I'm trying to figure out if they are identical.
I research this by viewing this particular issue: How do I test Django QuerySets are equal?
I am using class based views...and I have a model with two manytomanyfields...
My model...
class Author(models.Model):
title = models.ManyToManyField(User,blank=True,related_name='title')
title1 = models.ManyToManyField(User,blank=True,related_name='title1)
My View...
class AuthorDetailView(DetailView):
model = Author
def get_context_data(self, **kwargs):
context = super(AuthorDetailView, self).get_context_data(**kwargs)
title = list(Author.objects.filter(title))
title1 = list(Author.objects.filter(title1))
test_instance = Author.objects.all()
proxy4 = self.assertQuerySetEqual(Author.objects.all(), map(repr, [test_instance]))
I am trying to compare fields title and title1. However when try to run the code above I continually get a 'View' object has no attribute 'assertQuerysetEqual'. I can't even get this function to work. I running Django 1.11 and Postgresql. Perhaps this function doesn't work with Postgresql? Any help to get me on the right track is appreciated. Been playing with this and researching all morning with no luck. Thanks in advance.
Update...I have also been playing with various versions of trying to compare title_set.all and title1_set.all in the views....this is working intermittently...but the two are always returning that they are equal.
self.assertQuerySetEqual is about unit-testing.
All you need is compare two lists in your view. It can be done with sets.
set(title) == set(title1)
Your AuthorDetailView need to inherits from django.test.TestCase in order to get access to self.assertQuerySetEqual which is a testing feature.
So basically you need to do like this example:
from django.test import TestCase
...
class AuthorDetailView(DetailView, TestCase):
...
Otherwise for comparing two lists you can use set() like what #Daniil Mashkin said in his answer.
So, more in depth, you can do something like this:
class AuthorDetailView(DetailView):
model = Author
def get_context_data(self, **kwargs):
context = super(AuthorDetailView, self).get_context_data(**kwargs)
title = list(Author.objects.all().values_list('title'))
title1 = list(Author.objects.all().values_list('title1'))
# test if the two lists are equal
equal_ = set(title) == set(title1)
# add the result to the context variable
context.update({'titles_are_equal': equal_})
# return it in order to get the variable `title_are_equal`
# into your template
return context
For more informations visit the django official documentaiton
I am running into a database issue in my unit tests. I think it has something to do with the way I am using TestCase and setUpData.
When I try to set up my test data with certain values, the tests throw the following error:
django.db.utils.IntegrityError: duplicate key value violates unique constraint
...
psycopg2.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_productgroup_product_name_48ec6f8d_uniq"
DETAIL: Key (product_name)=(Almonds) already exists.
I changed all of my primary keys and it seems to be running fine. It doesn't seem to affect any of the tests.
However, I'm concerned that I am doing something wrong. When it first happened, I reversed about an hour's worth of work on my app (not that much code for a noob), which corrected the problem.
Then when I wrote the changes back in, the same issue presented itself again. TestCase is pasted below. The issue seems to occur after I add the sortrecord items, but corresponds with the items above it.
I don't want to keep going through and changing primary keys and urls in my tests, so if anyone sees something wrong with the way I am using this, please help me out. Thanks!
TestCase
class DetailsPageTest(TestCase):
#classmethod
def setUpTestData(cls):
cls.product1 = ProductGroup.objects.create(
product_name="Almonds"
)
cls.variety1 = Variety.objects.create(
product_group = cls.product1,
variety_name = "non pareil",
husked = False,
finished = False,
)
cls.supplier1 = Supplier.objects.create(
company_name = "Acme",
company_location = "Acme Acres",
contact_info = "Call me!"
)
cls.shipment1 = Purchase.objects.create(
tag=9,
shipment_id=9999,
supplier_id = cls.supplier1,
purchase_date='2015-01-09',
purchase_price=9.99,
product_name=cls.variety1,
pieces=99,
kgs=999,
crackout_estimate=99.9
)
cls.shipment2 = Purchase.objects.create(
tag=8,
shipment_id=8888,
supplier_id=cls.supplier1,
purchase_date='2015-01-08',
purchase_price=8.88,
product_name=cls.variety1,
pieces=88,
kgs=888,
crackout_estimate=88.8
)
cls.shipment3 = Purchase.objects.create(
tag=7,
shipment_id=7777,
supplier_id=cls.supplier1,
purchase_date='2014-01-07',
purchase_price=7.77,
product_name=cls.variety1,
pieces=77,
kgs=777,
crackout_estimate=77.7
)
cls.sortrecord1 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-05",
bags_sorted=20,
turnout=199,
)
cls.sortrecord2 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-07",
bags_sorted=40,
turnout=399,
)
cls.sortrecord3 = SortingRecords.objects.create(
tag=cls.shipment1,
date='2015-02-09',
bags_sorted=30,
turnout=299,
)
Models
from datetime import datetime
from django.db import models
from django.db.models import Q
class ProductGroup(models.Model):
product_name = models.CharField(max_length=140, primary_key=True)
def __str__(self):
return self.product_name
class Meta:
verbose_name = "Product"
class Supplier(models.Model):
company_name = models.CharField(max_length=45)
company_location = models.CharField(max_length=45)
contact_info = models.CharField(max_length=256)
class Meta:
ordering = ["company_name"]
def __str__(self):
return self.company_name
class Variety(models.Model):
product_group = models.ForeignKey(ProductGroup)
variety_name = models.CharField(max_length=140)
husked = models.BooleanField()
finished = models.BooleanField()
description = models.CharField(max_length=500, blank=True)
class Meta:
ordering = ["product_group_id"]
verbose_name_plural = "Varieties"
def __str__(self):
return self.variety_name
class PurchaseYears(models.Manager):
def purchase_years_list(self):
unique_years = Purchase.objects.dates('purchase_date', 'year')
results_list = []
for p in unique_years:
results_list.append(p.year)
return results_list
class Purchase(models.Model):
tag = models.IntegerField(primary_key=True)
product_name = models.ForeignKey(Variety, related_name='purchases')
shipment_id = models.CharField(max_length=24)
supplier_id = models.ForeignKey(Supplier)
purchase_date = models.DateField()
estimated_delivery = models.DateField(null=True, blank=True)
purchase_price = models.DecimalField(max_digits=6, decimal_places=3)
pieces = models.IntegerField()
kgs = models.IntegerField()
crackout_estimate = models.DecimalField(max_digits=6,decimal_places=3, null=True)
crackout_actual = models.DecimalField(max_digits=6,decimal_places=3, null=True)
objects = models.Manager()
purchase_years = PurchaseYears()
# Keep manager as "objects" in case admin, etc. needs it. Filter can be called like so:
# Purchase.objects.purchase_years_list()
# Managers in docs: https://docs.djangoproject.com/en/1.8/intro/tutorial01/
class Meta:
ordering = ["purchase_date"]
def __str__(self):
return self.shipment_id
def _weight_conversion(self):
return round(self.kgs * 2.20462)
lbs = property(_weight_conversion)
class SortingModelsBagsCalulator(models.Manager):
def total_sorted(self, record_date, current_set):
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= record_date]
return sum(sorted)
class SortingRecords(models.Model):
tag = models.ForeignKey(Purchase, related_name='sorting_record')
date = models.DateField()
bags_sorted = models.IntegerField()
turnout = models.IntegerField()
objects = models.Manager()
def __str__(self):
return "%s [%s]" % (self.date, self.tag.tag)
class Meta:
ordering = ["date"]
verbose_name_plural = "Sorting Records"
def _calculate_kgs_sorted(self):
kg_per_bag = self.tag.kgs / self.tag.pieces
kgs_sorted = kg_per_bag * self.bags_sorted
return (round(kgs_sorted, 2))
kgs_sorted = property(_calculate_kgs_sorted)
def _byproduct(self):
waste = self.kgs_sorted - self.turnout
return (round(waste, 2))
byproduct = property(_byproduct)
def _bags_remaining(self):
current_set = SortingRecords.objects.values().filter(~Q(id=self.id), tag=self.tag)
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= self.date]
remaining = self.tag.pieces - sum(sorted) - self.bags_sorted
return remaining
bags_remaining = property(_bags_remaining)
EDIT
It also fails with integers, like so.
django.db.utils.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_purchase_pkey"
DETAIL: Key (tag)=(9) already exists.
UDPATE
So I should have mentioned this earlier, but I completely forgot. I have two unit test files that use the same data. Just for kicks, I matched a primary key in both instances of setUpTestData() to a different value and sure enough, I got the same error.
These two setups were working fine side-by-side before I added more data to one of them. Now, it appears that they need different values. I guess you can only get away with using repeat data for so long.
I continued to get this error without having any duplicate data but I was able to resolve the issue by initializing the object and calling the save() method rather than creating the object via Model.objects.create()
In other words, I did this:
#classmethod
def setUpTestData(cls):
cls.person = Person(first_name="Jane", last_name="Doe")
cls.person.save()
Instead of this:
#classmethod
def setUpTestData(cls):
cls.person = Person.objects.create(first_name="Jane", last_name="Doe")
I've been running into this issue sporadically for months now. I believe I just figured out the root cause and a couple solutions.
Summary
For whatever reason, it seems like the Django test case base classes aren't removing the database records created by let's just call it TestCase1 before running TestCase2. Which, in TestCase2 when it tries to create records in the database using the same IDs as TestCase1 the database raises a DuplicateKey exception because those IDs already exists in the database. And even saying the magic word "please" won't help with database duplicate key errors.
Good news is, there are multiple ways to solve this problem! Here are a couple...
Solution 1
Make sure if you are overriding the class method tearDownClass that you call super().tearDownClass(). If you override tearDownClass() without calling its super, it will in turn never call TransactionTestCase._post_teardown() nor TransactionTestCase._fixture_teardown(). Quoting from the doc string in TransactionTestCase._post_teardown()`:
def _post_teardown(self):
"""
Perform post-test things:
* Flush the contents of the database to leave a clean slate. If the
class has an 'available_apps' attribute, don't fire post_migrate.
* Force-close the connection so the next test gets a clean cursor.
"""
If TestCase.tearDownClass() is not called via super() then the database is not reset in between test cases and you will get the dreaded duplicate key exception.
Solution 2
Override TransactionTestCase and set the class variable serialized_rollback = True, like this:
class MyTestCase(TransactionTestCase):
fixtures = ['test-data.json']
serialized_rollback = True
def test_name_goes_here(self):
pass
Quoting from the source:
class TransactionTestCase(SimpleTestCase):
...
# If transactions aren't available, Django will serialize the database
# contents into a fixture during setup and flush and reload them
# during teardown (as flush does not restore data from migrations).
# This can be slow; this flag allows enabling on a per-case basis.
serialized_rollback = False
When serialized_rollback is set to True, Django test runner rolls back any transactions inserted into the database beween test cases. And batta bing, batta bang... no more duplicate key errors!
Conclusion
There are probably many more ways to implement a solution for the OP's issue, but these two should work nicely. Would definitely love to have more solutions added by others for clarity sake and a deeper understanding of the underlying Django test case base classes. Phew, say that last line real fast three times and you could win a pony!
The log you provided states DETAIL: Key (product_name)=(Almonds) already exists. Did you verify in your db?
To prevent such errors in the future, you should prefix all your test data string by test_
I discovered the issue, as noted at the bottom of the question.
From what I can tell, the database didn't like me using duplicate data in the setUpTestData() methods of two different tests. Changing the primary key values in the second test corrected the problem.
I think the problem here is that you had a tearDownClass method in your TestCase without the call to super method.
In this way the django TestCase lost the transactional functionalities behind the setUpTestData so it doesn't clean your test db after a TestCase is finished.
Check warning in django docs here:
https://docs.djangoproject.com/en/1.10/topics/testing/tools/#django.test.SimpleTestCase.allow_database_queries
I had similar problem that had been caused by providing the primary key value to a test case explicitly.
As discussed in the Django documentation, manually assigning a value to an auto-incrementing field doesn’t update the field’s sequence, which might later cause a conflict.
I have solved it by altering the sequence manually:
from django.db import connection
class MyTestCase(TestCase):
#classmethod
def setUpTestData(cls):
Model.objects.create(id=1)
with connection.cursor() as c:
c.execute(
"""
ALTER SEQUENCE "app_model_id_seq" RESTART WITH 2;
"""
)
Say I have django model that looks something like this:
class Order(models.Model):
number = models...
date = models...
class OrderLine(models.Model):
# One or more lines per order
order = models.ForeginKey(Order)
common_line = models.OneToOneField(CommonLine)
class CommonLine(models.Model):
# common elements of what might be on a line item...
taxes = model...
amount = model...
I want to create a form that uses an inlineformset to edit one or more Lines (both OrderLine and CommonLine) per order.
I can create a formset that works with Order and OrderLine - but how do I get the inline formset to give me all the detailed items from the CommonLine class when displaying the formset. It seems the documentation on inline formsets requires that the inline form - the multiple lines on an order can only map to a single class...
Am I not seeing something in the documentation? I'm sure I can probably override something, I'm just not sure where.
Thanks for any help...
I solved problem with http://yergler.net/blog/2009/09/27/nested-formsets-with-django/. Pleas use the following correction in forms.py file:
instance=None
pk_value = hash(form.prefix)
+ correct_data = None
+ if (self.data):
+ correct_data = self.data;
# store the formset in the .nested property
form.nested = [
- TenantFormset(data=self.data,
+ TenantFormset(data=correct_data,
instance = instance,
Just working on Django 1.4.1 very well.
Some minor changes were needed to make Nathan's code at http://yergler.net/blog/2009/09/27/nested-formsets-with-django/ work in Django 1.3. The line below causes a ManagementForm Error.
TenantFormset = inlineformset_factory(models.Building, models.Tenant, extra=1)
Usings the modelformset_factory and manually defining the queryset seems to work, but I have not implemented the ability to add extras.
TenantFormset = modelformset_factory(models.Tenant, extra=0)
form.nested = [
TenantFormset(
queryset = Tenant.objects.filter(building = pk_value),
prefix = 'value_%s' % pk_value
)
]
I also had to manually pass data to the sub-sub-forms in the is_valid method:
def is_valid(self):
result = super(BaseProtocolEventFormSet, self).is_valid()
for form in self.forms:
if hasattr(form, 'nested'):
for n in form.nested:
n.data = form.data
if form.is_bound:
n.is_bound = True
for nform in n:
nform.data = form.data
if form.is_bound:
nform.is_bound = True
# make sure each nested formset is valid as well
result = result and n.is_valid()
return result
EDIT:
New instances can be created using jQuery. See this question:
This sounds very similar to the approach talked about at help http://yergler.net/blog/2009/09/27/nested-formsets-with-django/ where Nathan writes about how he catered for "a multi-level data model; an example of this kind of model would be modeling City Blocks, where each Block has one or more Buildings, and each Building has one or more Tenants."
Some more explanations can aslo be found here Django Forms Newbie Question
I am sure that someone has a pluggable app (or tutorial) out there that approximates this, but I have having trouble finding it: I want to be able to track the number of "views" a particular object has (just like a question here on stackoverflow has a "view count").
If the user isn't logged in, I wouldn't mind attempting to place a cookie (or log an IP) so they can't inadvertently run up the view count by refreshing the page; and if a user is logged in, only allow them one "view" across sessions/browsers/IP addresses. I don't think I need it any fancier than that.
I figure the best way to do this is with Middleware that is decoupled from the various models I want to track and using an F expression (of sorts) -- other questions on StackOverflow have alluded to this (1), (2), (3).
But I wonder if this code exists out in the wild already -- because I am not the savviest coder and I'm sure someone could do it better. Smile.
Have you seen it?
I am not sure if it's in the best taste to answer my own question but, after a bit of work, I put together an app that solves the problems in earnest: django-hitcount.
You can read about how to use it at the documentation page.
The ideas for django-hitcount came came from both of my two original answers (Teebes -and- vikingosegundo), which really got me started thinking about the whole thing.
This is my first attempt at sharing a pluggable app with the community and hope someone else finds it useful. Thanks!
You should use the django built-in session framework, it already does a lot of this for you. I implemented this in the following way with a Q&A app where I wanted to track views:
in models.py:
class QuestionView(models.Model):
question = models.ForeignKey(Question, related_name='questionviews', on_delete=models.CASCADE)
ip = models.CharField(max_length=40)
session = models.CharField(max_length=40)
created = models.DateTimeField(default=datetime.datetime.now())
in views.py:
def record_view(request, question_id):
question = get_object_or_404(Question, pk=question_id)
if not QuestionView.objects.filter(
question=question,
session=request.session.session_key):
view = QuestionView(question=question,
ip=request.META['REMOTE_ADDR'],
created=datetime.datetime.now(),
session=request.session.session_key)
view.save()
return HttpResponse(u"%s" % QuestionView.objects.filter(question=question).count())
Vikingosegundo is probably right though that using content-type is probably the more reusable solution but definitely don't reinvent the wheel in terms of tracking sessions, Django already does that!
Last thing, you should probably have the view that records the hit be either called via Ajax or a css link so that search engines don't rev up your counts.
Hope that helps!
You could create a generic Hit model
class Hit(models.Model):
date = models.DateTimeField(auto_now=True)
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = generic.GenericForeignKey('content_type', 'object_id')
In your view.py you write this function:
def render_to_response_hit_count(request,template_path,keys,response):
for key in keys:
for i in response[key]:
Hit(content_object=i).save()
return render_to_response(template_path, response)
and the views that you are interested in return
return render_to_response_hit_count(request, 'map/list.html',['list',],
{
'list': l,
})
This approach gives you the power, not only to count the hit, but to filter the hit-history by time, contenttype and so on...
As the hit-table might be growing fast, you should think about a deletion strategy.
I know this question is an old one and also thornomad has put an app to solve the problem and inspire me with me solution. I would like to share this solution since I didn't find much information about this topic and it may help someone else.
My approach is to make a generic model can be used with any view based on the view path (url).
models.py
class UrlHit(models.Model):
url = models.URLField()
hits = models.PositiveIntegerField(default=0)
def __str__(self):
return str(self.url)
def increase(self):
self.hits += 1
self.save()
class HitCount(models.Model):
url_hit = models.ForeignKey(UrlHit, editable=False, on_delete=models.CASCADE)
ip = models.CharField(max_length=40)
session = models.CharField(max_length=40)
date = models.DateTimeField(auto_now=True)
views.py
def get_client_ip(request):
x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
if x_forwarded_for:
ip = x_forwarded_for.split(',')[0]
else:
ip = request.META.get('REMOTE_ADDR')
return ip
def hit_count(request):
if not request.session.session_key:
request.session.save()
s_key = request.session.session_key
ip = get_client_ip(request)
url, url_created = UrlHit.objects.get_or_create(url=request.path)
if url_created:
track, created = HitCount.objects.get_or_create(url_hit=url, ip=ip, session=s_key)
if created:
url.increase()
request.session[ip] = ip
request.session[request.path] = request.path
else:
if ip and request.path not in request.session:
track, created = HitCount.objects.get_or_create(url_hit=url, ip=ip, session=s_key)
if created:
url.increase()
request.session[ip] = ip
request.session[request.path] = request.path
return url.hits
I did this by creating a model PageViews and making a column "Hits" in it. Every time when Homepage url is hit. I increment the first and only row of column Hit and render it to the template. Here how it looks.
Views.py
def Home(request):
if(PageView.objects.count()<=0):
x=PageView.objects.create()
x.save()
else:
x=PageView.objects.all()[0]
x.hits=x.hits+1
x.save()
context={'page':x.hits}
return render(request,'home.html',context=context)
Models.py
class PageView(models.Model):
hits=models.IntegerField(default=0)
I did it using cookies. Don't know if it's a good idea to do that or not. The following code looks for an already set cookie first if it exists it increases the total_view counter if it is not there the it increases both total_views and unique_views. Both total_views and unique_views are a field of a Django model.
def view(request):
...
cookie_state = request.COOKIES.get('viewed_post_%s' % post_name_slug)
response = render_to_response('community/post.html',context_instance=RequestContext(request, context_dict))
if cookie_state:
Post.objects.filter(id=post.id).update(total_views=F('total_views') + 1)
else:
Post.objects.filter(id=post.id).update(unique_views=F('unique_views') + 1)
Post.objects.filter(id=post.id).update(total_views=F('total_views') + 1)
response.set_cookie('viewed_post_%s' % post_name_slug , True, max_age=2678400)
return response