Why Django get result list from query_set too late? - django

Question

I am studying about Django ORM. I couldn't get an answer from the search, but I'd appreciate it if someone could tell me the related site.
My model is as follows. user1 has2 accounts, and 500,000 transactions belong to one of the accounts.
class Account(models.Model):
class Meta:
db_table = 'account'
ordering = ['created_at']
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
account = models.CharField(max_length=20, null=False, blank=False, primary_key=True)
balance = models.PositiveBigIntegerField(default=0)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class AccountTransaction(models.Model):
class Meta:
db_table = 'account_transaction'
ordering = ['tran_time']
indexes = [
models.Index(fields=['tran_type', 'tran_time', ]),
]
account = models.ForeignKey(Account, on_delete=models.CASCADE)
tran_amt = models.PositiveBigIntegerField()
balance = models.PositiveBigIntegerField()
tran_type = models.CharField(max_length=10, null=False, blank=False)
tran_detail = models.CharField(max_length=100, null=True, default="")
tran_time = models.DateTimeField(auto_now_add=True)
The query time for the above model is as follows.
start = time.time()
rs = request.user.account_set.all().get(account="0000000010").accounttransaction_set.all()
count = rs.count()
print('>>all')
print(time.time() - start) # 0.028000831604003906
start = time.time()
q = Q(tran_time__date__range = ("2000-01-01", "2000-01-03"))
rs = request.user.account_set.all().get(account="0000000010").accounttransaction_set.filter(q)
print('>>filter')
print(time.time() - start) # 0.0019981861114501953
start = time.time()
result = list(rs)
print('>>offset')
print(time.time() - start) # 5.4373579025268555
The result of the query_set is about 3500 in total. (3500 out of 500,000 records were selected).
I've done a number of things, such as setting offset to the result (rs) of query_set, but it still takes a long time to get the actual value from query_set.
I know that the view loads data when approaching actual values such as count(), but what did I do wrong?

Answer 1

From https://docs.djangoproject.com/en/4.1/topics/db/queries/#querysets-are-lazy:
QuerySets are lazy – the act of creating a QuerySet doesn’t involve
any database activity. You can stack filters together all day long,
and Django won’t actually run the query until the QuerySet is
evaluated. Take a look at this example:
q = Entry.objects.filter(headline__startswith="What")
q = q.filter(pub_date__lte=datetime.date.today())
q = q.exclude(body_text__icontains="food")
print(q)
Though this looks like three database hits, in fact it hits the
database only once, at the last line (print(q)). In general, the
results of a QuerySet aren’t fetched from the database until you “ask”
for them. When you do, the QuerySet is evaluated by accessing the
database. For more details on exactly when evaluation takes place, see
When QuerySets are evaluated.
In your example the database is hit only when you're calling list(rs), that's why it takes so long.