I have some problems with my query - with filter() it's ok but with exclude() doesn't work.
My models:
class Dictionary(DateTimeModel):
base_word = models.ForeignKey(BaseDictionary, related_name=_('dict_words'))
word = models.CharField(max_length=64)
version = models.ForeignKey(Version)
class FrequencyData(DateTimeModel):
word = models.ForeignKey(Dictionary, related_name=_('frequency_data'))
count = models.BigIntegerField(null=True, blank=True)
source = models.ForeignKey(Source, related_name=_('frequency_data'), null=True, blank=True)
user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name=_('frequency_data'))
user_ip_address = models.GenericIPAddressField(null=True, blank=True)
date_of_checking = models.DateTimeField(null=True, blank=True)
is_checked = models.BooleanField(default=False)
And I want to get some words from Dictionary where whose frequencies are not from some user
Dictionary.objects.prefetch_related('frequency_data').filter(frequency_data__user=1)[:100] - OK
Dictionary.objects.prefetch_related('frequency_data').exclude(frequency_data__user=1)[:100] - processor up to 100% and loading
Without prefetch_related the same. What is wrong with exclude?
EDIT
Dictionary db tabel - 120k rows
FrequencyData - 160k rows
EDIT2
psql(9.6.6)
Related
I am studying about Django ORM. I couldn't get an answer from the search, but I'd appreciate it if someone could tell me the related site.
My model is as follows. user1 has2 accounts, and 500,000 transactions belong to one of the accounts.
class Account(models.Model):
class Meta:
db_table = 'account'
ordering = ['created_at']
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
account = models.CharField(max_length=20, null=False, blank=False, primary_key=True)
balance = models.PositiveBigIntegerField(default=0)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class AccountTransaction(models.Model):
class Meta:
db_table = 'account_transaction'
ordering = ['tran_time']
indexes = [
models.Index(fields=['tran_type', 'tran_time', ]),
]
account = models.ForeignKey(Account, on_delete=models.CASCADE)
tran_amt = models.PositiveBigIntegerField()
balance = models.PositiveBigIntegerField()
tran_type = models.CharField(max_length=10, null=False, blank=False)
tran_detail = models.CharField(max_length=100, null=True, default="")
tran_time = models.DateTimeField(auto_now_add=True)
The query time for the above model is as follows.
start = time.time()
rs = request.user.account_set.all().get(account="0000000010").accounttransaction_set.all()
count = rs.count()
print('>>all')
print(time.time() - start) # 0.028000831604003906
start = time.time()
q = Q(tran_time__date__range = ("2000-01-01", "2000-01-03"))
rs = request.user.account_set.all().get(account="0000000010").accounttransaction_set.filter(q)
print('>>filter')
print(time.time() - start) # 0.0019981861114501953
start = time.time()
result = list(rs)
print('>>offset')
print(time.time() - start) # 5.4373579025268555
The result of the query_set is about 3500 in total. (3500 out of 500,000 records were selected).
I've done a number of things, such as setting offset to the result (rs) of query_set, but it still takes a long time to get the actual value from query_set.
I know that the view loads data when approaching actual values such as count(), but what did I do wrong?
From https://docs.djangoproject.com/en/4.1/topics/db/queries/#querysets-are-lazy:
QuerySets are lazy – the act of creating a QuerySet doesn’t involve
any database activity. You can stack filters together all day long,
and Django won’t actually run the query until the QuerySet is
evaluated. Take a look at this example:
q = Entry.objects.filter(headline__startswith="What")
q = q.filter(pub_date__lte=datetime.date.today())
q = q.exclude(body_text__icontains="food")
print(q)
Though this looks like three database hits, in fact it hits the
database only once, at the last line (print(q)). In general, the
results of a QuerySet aren’t fetched from the database until you “ask”
for them. When you do, the QuerySet is evaluated by accessing the
database. For more details on exactly when evaluation takes place, see
When QuerySets are evaluated.
In your example the database is hit only when you're calling list(rs), that's why it takes so long.
I Have Two models
User Model and DailyPresent model with user as foreign_key in DailyPresent model.
class DailyPresentReport(models.Model):
PRESENT = 'present'
ABSENT = 'absent'
ON_LEAVE = 'on_leave'
PRESENT_CHOICES = (
(PRESENT, 'Present'),
(ABSENT, 'Absent'),
(ON_LEAVE, 'On Leave'),
)
user = models.ForeignKey(User, on_delete=models.CASCADE, related_name='daily_present_report')
present = models.CharField(max_length=10, choices=PRESENT_CHOICES, default=ABSENT)
punch_in = models.DateTimeField(null=True, blank=True)
punch_out = models.DateTimeField(null=True, blank=True)
date = models.DateField(null=True, blank=True)
work_time = models.DurationField(null=True, blank=True)
class Meta:
ordering = ['id']
def __str__(self):
return f'{str(self.user)}: {self.present}'
If User logs in then he is automatically made present
but when user doesn't login nothing happens.
Now I want to display a table with all users with Present and Absent Fields.
Please help me.. Thanks in advance
you can use Q module, processing the combination of "or" conditions when searching:
from django.db.models.query_utils import Q
dailyPresents = DailyPresentReport.objects.filter(Q(present="PRESENT") |
Q(present="ABSENT"))
or use .exclude to exclude the On Leave:
dailyPresents = DailyPresentReport.objects.exclude(present="ON_LEAVE")
I have a relatively simple query running on a database table; the filter checks the value of 2 date fields and returns a queryset. The fields checked are authored_report_received and subbed_complete. If authored_report_received has a date set AND subbed_complete has no date set, then a result will be returned.
If both these fields have a date set then no results are returned.
(A simple calculation for the 'days outstanding' between any result returned and the current date is then made.)
This works absolutely fine running locally on a SQLite database but when I upload to heroku to be used with a PostgreSQL database, the filter does not seem to work in that it seems to ignore any date value in the subbed_complete field. I do not understand why not. Is there a difference between the way SQLite and PostgreSQL handles the filters? I can't find any docs which indicate this.
# views.py
class EpgReportSubeditListView(LoginRequiredMixin, EpgUserPassesTestMixin, ListView):
"""EPG-only: create a list view of reports currently being subedited
"""
model = Report
template_name = 'tracker/epg_report_subedit.html'
context_object_name = 'report_subedit'
ordering = ['subeditor']
def get_queryset(self):
today = datetime.now().date()
out = []
for obj in (self.model.objects.filter(subbed_complete__isnull=True) and self.model.objects.filter(authored_report_received__isnull=False)):
setattr(obj, 'days_diff', (today - obj.authored_report_received).days)
out.append(obj)
return out
# models.py
class Report(models.Model):
"""Define fields for Report object (representing a case report)
"""
case = models.ForeignKey(Case, on_delete=models.CASCADE, related_name='reports')
editor = models.ForeignKey(User, on_delete=False,
limit_choices_to={'groups__name': 'editors'},
related_name='editor', null=True)
# editing workflow info
authored_report_received = models.DateField(null=True, blank=True)
subbed_complete = models.DateField(null=True, blank=True)
subeditor = models.ForeignKey(User, on_delete=False,
limit_choices_to={'groups__name': 'subeditors'},
related_name='subeditor', null=True, blank=True)
sent_to_editor = models.DateField(null=True, blank=True)
edited = models.DateField(null=True, blank=True)
reporter_queries = models.CharField(max_length=3,
choices=Binary_Choices,
default="n")
sent_to_deskeditor = models.DateField(null=True, blank=True)
taken_by_deskeditor = models.DateField(null=True, blank=True)
deskeditor = models.ForeignKey(User, on_delete=False,
limit_choices_to={'groups__name': 'deskeditors'},
related_name='deskeditor', null=True, blank=True)
deskedit_complete = models.DateField(null=True, blank=True)
sent_to_counsel = models.DateField(null=True, blank=True)
received_from_counsel = models.DateField(null=True, blank=True)
date_editor_publish_approve = models.DateField(null=True, blank=True)
def get_absolute_url(self):
return reverse("epg_home")
def __str__(self):
return f'{self.case} | f: {self.filename}'
Sqlite and PostgreSQL handle dates differently. subbed_complete is a models.DateField field, but sqlite3:
does not have a storage class set aside for storing dates and/or
times. Instead, the built-in Date And Time Functions of SQLite are
capable of storing dates and times as TEXT, REAL, or INTEGER values:
Applications can chose to store dates and times in any of these formats and freely convert between formats using the built-in date and time functions.
So you should test your code and adjust the query in an another environment with postgresql.
https://www.sqlite.org/datatype3.html
I'm trying to make a program which take values of barcodes with barcode scanner and save the values of barcodes in db, problems occurs when I try to add more than 5-6 barcodes.
It gives me "django.db.utils.DataError: (1406, "Data too long for column 'barcodes' at row 1")" error.
I've made sure that its a model.textfield() but that doesn't solve my problem.
My models look like this:
id = models.IntegerField(primary_key=True)
barcodes = models.CharField(max_length=255)
to_warehouse = models.CharField(max_length=255)
from_warehouse = models.CharField(max_length=255)
total_count = models.IntegerField()
approval_flag = models.IntegerField(default=0)
current_status = models.CharField(max_length=50, blank=True, null=True)
error_message = models.CharField(max_length=255, blank=True, null=True)
created_by = models.CharField(max_length=50, blank=True, null=True)
created_at = models.DateTimeField(blank=True, null=True, auto_now_add=True)
class Meta:
managed = False
db_table = 'transfer_inventory'
def __str__(self):
return "%s" % self.id
My view function that creates the obj for that looks like this:
def change_status(self, barcodes, warehouse_id, request, is_error="", msg=""):
barcode_count = barcodes.count(',')
_list_barcodes = barcodes.split(",")
print("list: ", list)
list_barcodes = []
to_warehouse = Warehouse.objects.filter(id=warehouse_id).values_list('key', flat=True)[0]
try:
current_warehouse = UserWarehouseMapping.objects.filter(
user=request.user).values_list('warehouse__key', flat=True).first()
except:
current_warehouse = "-"
for i in _list_barcodes:
list_barcodes.append(i)
list_barcodes = list_barcodes.pop(len(list_barcodes) - 1)
available_barcodes = list(
Inventory.objects.filter(inventory_status='available').values_list('barcode', flat=True))
InventoryTransfer.objects.create(barcodes=barcodes, to_warehouse=to_warehouse,
total_count=barcode_count,
created_by=request.user.username,
from_warehouse=current_warehouse, current_status="Pending")
In which specific this part is used to create obj:
InventoryTransfer.objects.create(barcodes=barcodes, to_warehouse=to_warehouse,
total_count=barcode_count,
created_by=request.user.username,
from_warehouse=current_warehouse, current_status="Pending")
I'm stuck with this error since very long and I don't know what to do. Any kind of approach will be appreciated. Thank You.
From your model your column barcodes is charfield length 255
barcodes = models.CharField(max_length=255)
You cannot store more than 255 characters as that is what you set as max_length, also it would be beneficiary if you are storing longer texts to change it to
barcodes = models.TextField()
And don't forget to migrate afterwards
Other maybe a bit more reasonable idea would be to have related table for Barcode
As you are filtering through barcodes in your code it is much more efficient
Here is table in class from django first app called proizvodi:
class Meblovi(models.Model):
class Meta:
verbose_name_plural = "Meblovi"
#OSNOVNI_PODACI
ime_proizvoda = models.CharField(max_length=120)
proizvodjac = models.CharField(max_length=120, default="proizvodjac")
sastav = models.CharField(max_length=120, default="sastav")
sirina = models.CharField(max_length=120, default="sirina")
zemlja_porekla = models.CharField(max_length=120, default="zemlja porekla")
stara_cena = models.DecimalField(decimal_places=2,max_digits=10,default=795.00)
nova_cena = models.DecimalField(decimal_places=2,max_digits=10,default=795.00)
na_lageru = models.BooleanField()
rok_isporuke = models.CharField(max_length=120, default="3 dana")
jedinica_mere = models.CharField(max_length=120, default="po dužnom metru")
#SLIKE
glavna_slika = models.ImageField(upload_to='proizvodi/', null=True, blank=True)
#KARAKTERISTIKE
vodootporan = models.BooleanField(default=False)
vodoodbojan = models.BooleanField(default=False)
nezapaljiv = models. BooleanField(default=False)
#OSTALO
izdvojeno = models.BooleanField()
def __str__(self):
return self.ime_proizvoda
Now, here is table from another app:
class Podmeblovi(models.Model):
class Meta:
verbose_name_plural = "Meblovi - podvrste"
#OSNOVNI_PODACI
model_mebla = models.CharField(max_length=120)
mebl = models.ForeignKey(Meblovi, on_delete=models.CASCADE)
slika = models.ImageField(upload_to='proizvodi/', null=True, blank=True)
#BOJE
bela = models.BooleanField(default=False)
svetlo_siva = models.BooleanField(default=False)
tamno_siva = models.BooleanField(default=False)
crna = models.BooleanField(default=False)
bez = models.BooleanField(default=False)
braon = models.BooleanField(default=False)
zuta = models.BooleanField(default=False)
narandzasta = models.BooleanField(default=False)
crvena = models.BooleanField(default=False)
bordo = models.BooleanField(default=False)
svetlo_zelena = models.BooleanField(default=False)
tamno_zelena = models.BooleanField(default=False)
svetlo_plava = models.BooleanField(default=False)
tamno_plava = models.BooleanField(default=False)
pink = models.BooleanField(default=False)
ljubicasta = models.BooleanField(default=False)
#DIZAJN
jednobojno = models.BooleanField(default=False)
sareno = models.BooleanField(default=False)
def __str__(self):
return self.mebl.ime_proizvoda + " " + self.model_mebla
So what I want is to make a sql query which returns content from table Podmeblovi(second app) but only if field from table Meblovi (first app) 'vodootporan' is equal to 1. I tried INNER JOIN and it didn't seem to work. Also, i wanted to use RIGHT JOIN but it says that it is not working yet in Django.
My table from second app is connected to the first app table with foreign key. To explain better, first app table is basic default information table for product. So if we have fabrics product named Portland for example, we got child products of it in second app. They are different colours and designs, but the price and other specs are same for them. So how can I solve this problem?
What i tried:
SELECT * FROM modeli_meblova_podmeblovi
INNER JOIN "proizvodi_meblovi" ON ("proizvodi_meblovi"."vodootporan" = "meblovi")
WHERE "proizvodi_meblovi" = 1
This had no errors indeed, but in my database I have products checked true for field 'vodootporan', but sql didn't return any model from database.
Just to be clear, I need sql query because I have many filters to integrate with this one. Other filters are from base table, not joined one.
You didn't show what you tried. But this should work:
Podmeblovi.objects.filter(mebl__vodootporan=True)