Django connected SQL queries with filters - django

Example:
class Room(models.Model):
assigned_floor = models.ForeignKey(Floor, null=True, on_delete=models.CASCADE)
room_nr = models.CharField(db_index=True, max_length=4, unique=True, null=True)
locked = models.BooleanField(db_index=True, default=False)
last_cleaning = models.DateTimeField(db_index=True, auto_now_add=True, null=True)
...
class Floor(models.Model):
assigned_building = models.ForeignKey(Building, on_delete=models.CASCADE)
wall_color = models.CharField(db_index=True, max_length=255, blank=True, null=True)
...
class Building(models.Model):
name = models.CharField(db_index=True, max_length=255, unique=True, null=True)
number = models.PositiveIntegerField(db_index=True)
color = models.CharField(db_index=True, max_length=255, null=True)
...
I want to output all rooms in a table sorted by Building.number.
Data which I want to print for each room:
Building.number, Building.color, Building.name, Floor.wall_color, Room.last_cleaning
Furthermore I want to allow optional filters:
Room.locked, Room.last_cleaning, Floor.wall_color, Building.number, Building.color
With one table it's no Problem for me, but I don't know how I archive this with three tables.
kwargs = {'number': 123}
kwargs['color'] = 'blue'
all_buildings = Building.objects.filter(**kwargs).order_by('-number')
Can you please help me? Do I need write raw SQL queries or can I archive this with the Django model query APIs?
I'm using the latest Django version with PostgreSQL.

No raw sql needed:
room_queryset = Room.objects.filter(assigned_floor__wall_color='blue')
^^
# A double unterscore declares the following attribute to be a field of the object referenced in the foregoing foreign key field.
for room in room_queryset:
print(room.assigned_floor.assigned_building.number)
print(room.assigned_floor.assigned_building.color)
print(room.assigned_floor.assigned_building.name)
print(room.assigned_floor.wall_color)
print(room.last_cleaning)

Related

Django One to Many Loose Relationship

Backstory
Data is being pulled from an accounting system that can have department, market, and client relationship data associated with it. The relationship data are all TEXT/CHAR fields. They are not integer columns. There are over 2 million rows.
The problem
The problem I've been running into is how to add the lines without relationship validation that could fail because the related table (like Market) is missing the value or has been changed (because we are looking far back in the past). The naming of the columns in the Django database (more detail below), and querying Django models with a join that don't have a relationship attribute on the class.
models.py
from typing import Optional
from django.db import models
class Line(models.Model):
entry_number: int = models.IntegerField(db_index=True)
posting_date: date = models.DateField()
document_number: Optional[str] = models.CharField(max_length=150, null=True, default=None)
description: Optional[str] = models.CharField(max_length=150, null=True, default=None)
department: Optional[str] = models.CharField(max_length=150, null=True, default=None)
market: Optional[str] = models.CharField(max_length=150, null=True, default=None)
amount: Decimal = models.DecimalField(max_digits=18, decimal_places=2)
client: Optional[str] = models.CharField(max_length=150, null=True, default=None)
# This relationship works
account = models.ForeignKey(Account, on_delete=models.DO_NOTHING, related_name='lines')
class Department(models.Model):
code: str = models.CharField(max_length=10, db_index=True, unique=True, primary_key=True)
name: str = models.CharField(max_length=100, null=True)
class Market(models.Model):
code: str = models.CharField(max_length=10, db_index=True, unique=True, primary_key=True)
name: str = models.CharField(max_length=100, null=True)
The data for the line is filled in the a sql statement grabbing the data from the accounting system.
In SqlAlchemy
What I am looking for is something like this which can be represented in sqlalchemy.
class Line(...):
# snipped
client_rel: Client = relationship("Client", primaryjoin=client == foreign(Client.code), viewonly=True)
department_rel: Department = relationship("Department", primaryjoin=department == foreign(Department.code), viewonly=True)
market_rel: Market = relationship("Market", primaryjoin=market == foreign(Market.code), viewonly=True)
# snipped the other related classes
The SQLAlchemy code allows for primaryjoins which are relationships only enforced at the python level not in the database. It also allows for the relatanship's loose foreign key to be named almost anything. As far as I can tell Django's foreign key column has to be "name"_id and can't be changed.

When I open the model, one field value is replaced automatically

Have a stupid issue: When I open the employee model (in django admin), one field (employee_type) value is replaced automatically, idk why...
Example: I create employee, define employee type as manager, save. In DB value is manager. After that, I open employee and see employee type as sewer and if I save, that value will saved in DB.
I created text choices for this field, and In DB field is defined as enum.
I tried to research the issue, value isn't used from DB, always first value from text choices is used.
By the way, I created same fields (enum in DB and text choices in the model) in other model. And all works properly.
How can I fix it???
Models.py with the issue:
class Employee(models.Model):
class EmployeeType(models.TextChoices):
SEWER = 'SEWER', _('Sewer')
MANAGER = 'MANAGER', _('Manager')
UNDEFINED = 'UNDEFINED', _('Undefined')
user = models.OneToOneField(User,
models.CASCADE,
db_column='user',
verbose_name=_('User'),
primary_key=True)
employee_type = models.CharField(db_column='Employee type',
verbose_name=_('Employee type'),
max_length=9,
choices=EmployeeType.choices,
default=EmployeeType.UNDEFINED)
phone = models.CharField(db_column='Phone',
verbose_name=_('Phone'),
max_length=255)
work_xp = models.IntegerField(db_column='Work XP',
verbose_name=_('Work XP'),
blank=True,
null=True)
Another one models.py. With same fields but without issues:
class Order(models.Model):
class OrderStatus(models.TextChoices):
CREATED = 'Created', _('Created')
CANCELLED = 'Cancelled', _('Cancelled')
IN_PROGRESS = 'In progress', _('In progress')
COMPLETED = 'Completed', _('Completed')
PASSED_TO_CLIENT = 'Passed to the client', _('Passed to the client')
RETURNED_FOR_REWORK = 'Returned for rework', _('Returned for rework')
class Urgency(models.TextChoices):
LOW = 'Low', _('Low urgency')
MEDIUM = 'Medium', _('Medium urgency')
HIGH = 'High', _('High urgency')
VERY_HIGH = 'Very high', _('Very high urgency')
class LabourIntensity(models.TextChoices):
LOW = 'Low', _('1-3 days')
MEDIUM = 'Medium', _('4-6 days')
HIGH = 'High', _('7-9 days')
VERY_HIGH = 'Very high', _('10+ days')
class PaymentStatus(models.TextChoices):
PENDING = 'Pending payment', _('Pending payment')
PREPAYMENT_MADE = 'Prepayment made', _('Prepayment made')
PAID = 'Paid', _('Paid')
id_service = models.ForeignKey(Service,
models.SET_NULL,
db_column='id_Service',
verbose_name=_('Service'),
blank=True,
null=True)
status = models.CharField(db_column='Status',
verbose_name=_('Status'),
max_length=20,
choices=OrderStatus.choices,
default=OrderStatus.CREATED)
payment_status = models.CharField(db_column='Payment status',
verbose_name=_('Payment status'),
max_length=15,
choices=PaymentStatus.choices,
blank=True,
null=True)
prepayment = models.DecimalField(db_column='Prepayment',
verbose_name=_('Prepayment'),
max_digits=19,
decimal_places=2,
blank=True,
null=True)
cost = models.DecimalField(db_column='Cost',
verbose_name=_('Cost'),
max_digits=19,
decimal_places=2,
blank=True,
null=True)
start_date = models.DateTimeField(db_column='Start date',
verbose_name=_('Start date'),
blank=True,
null=True)
end_date = models.DateTimeField(db_column='End date',
verbose_name=_('End date'),
blank=True,
null=True)
id_client = models.ForeignKey(Client,
models.SET_NULL,
db_column='id_Client',
verbose_name=_('Client'),
blank=True,
null=True)
id_employee = models.ForeignKey(Employee,
models.SET_NULL,
db_column='id_Employee',
verbose_name=_('Employee'),
blank=True,
null=True)
labour_intensity = models.CharField(db_column='Labour intensity',
verbose_name=_('Labour intensity'),
max_length=9,
choices=LabourIntensity.choices,
default=LabourIntensity.LOW)
urgency = models.CharField(db_column='Urgency',
verbose_name=_('Urgency'),
max_length=9,
choices=Urgency.choices,
default=Urgency.LOW)
materials = models.ManyToManyField(Material, through='OrderMaterials')
comment = models.TextField(db_column='Comment',
verbose_name=_('Comment'),
blank=True,
null=True)
I solved this issue.
In DB a type of field "employee_type" was defined as:
ENUM('Undefined','Sewer','Manager')
But the model has EmployeeType choices with uppercase.
Solution: I changed lowercase to uppercase of the values in the field type in DB:
ENUM('UNDEFINED','SEWER','MANAGER')
Now everything works fine.

Export xlsx file with model having reverse foreign key relation and make that reverse foreign key relation as separate column

I am using django import_export package for export my data into xlsx file.
I am having issue while exporting the data to excel in the format I need.
models.py
class Fans(models.Model):
"""
Model for survey answering people
"""
fan_id = models.AutoField(db_column='FAN_ID', primary_key=True)
first_name = models.CharField(
db_column='FIRST_NAME', max_length=45, blank=True, null=True)
last_name = models.CharField(
db_column='LAST_NAME', max_length=45, blank=True, null=True)
phone = models.CharField(
db_column='PHONE', max_length=45, blank=True, null=True)
email = models.CharField(
db_column='EMAIL', max_length=45, blank=True, null=True)
gender = models.CharField(
db_column='GENDER', max_length=45, blank=True, null=True)
class Responses(models.Model):
"""
Model for responses given by fans
"""
survey = models.ForeignKey(
Surveys, on_delete=models.CASCADE, db_column='SURVEY_ID', related_query_name="part")
fan = models.ForeignKey(Fans, on_delete=models.CASCADE,
db_column='FAN_ID', related_query_name="given", related_name="given")
survey_question = models.ForeignKey(
SurveyQuestions, on_delete=models.DO_NOTHING, db_column='SURVEY_QUESTION_ID',
related_query_name="response")
response = models.CharField(
db_column='RESPONSE', max_length=255, blank=True, null=True)
correct_answer = models.IntegerField(
db_column='CORRECT_ANSWER', blank=True, null=True)
load_id = models.IntegerField(db_column='LOAD_ID', blank=True, null=True)
class SurveyQuestions(models.Model):
"""
Model for surveys questions
"""
survey = models.ForeignKey(Surveys, on_delete=models.CASCADE,
db_column='SURVEY_ID', related_query_name="question")
survey_question_id = models.AutoField(
db_column='SURVEY_QUESTION_ID', primary_key=True)
survey_question_name = models.CharField(
db_column='SURVEY_QUESTION_NAME', max_length=255)
question = models.CharField(
db_column='QUESTION', max_length=255, blank=True, null=True)
response_type = models.CharField(
db_column='RESPONSE_TYPE', max_length=255, blank=True, null=True)
load_date = models.DateField(db_column='LOAD_DATE', auto_now_add=True)
I want to export data of the fans with recorded responses in the following format:
first_name, last_name, phone, email, question1, question2, question3
abc, xyz, 1234566780, abc#gmail.com, response1, response2, response3
Here, the first four fields are directly from Fans model, however the last three column headers represent "question" field from SurveyQuestions model and values come from the "response" field of Responses model.
Till now, I am able to achieve the following format:
first_name, last_name, phone, email, given
abc, xyz, 1234566780, abc#gmail.com, {question1: response1, question2: response2, question3: response3}
given field is json of question-response as key-value pair.
admin.py
class FanResource(resources.ModelResource):
"""
Resource for exporting to excel
"""
given = fields.Field()
class Meta:
model = Fans
fields = ("first_name", "last_name", "email",
"phone", "given")
def dehydrate_given(self, instance):
res = {}
for x in instance.given.values('response', 'survey_question__question'):
res[x['survey_question__question']] = x['response']
return json.dumps(res)
Any help would be appreciated. Thanks in advance!!
14/10/20 UPDATE
Using the answer below, I was able to achieve the format required. Code is as follows:
def after_export(self, queryset, data, *args, **kwargs):
survey_questions = {x["survey_question_id"]: x["question"] for x in SurveyQuestions.objects.filter(
survey=self.survey).values('survey_question_id', 'question')}
for k, v in survey_questions.items():
res = []
for x in queryset:
try:
res.append(x.given.get(survey_question=k).response)
except ObjectDoesNotExist:
res.append(None)
data.append_col(res, header=v)
Now, the issue is that it is taking too long as it is hitting the database for each entry. And other issue is order is not proper (i.e responses are not in same line as per the corresponding fans).
I think the way to achieve this would be to override after_export(), and manipulate the exported dataset. For example:
def after_export(self, queryset, data, *args, **kwargs):
response1 = [i for i in range(data.height)]
data.append_col(response1, header="response1")
response2 = [i for i in range(data.height)]
data.append_col(response2, header="response2")
This will append new columns to the end of the export. In after_export() you will have access to both the data set and the queryset, so hopefully you can manipulate this data to fill the 'response' columns correctly.

Django ORM with Key/Value Table structure

I'm trying to get the django orm to replicate a call on a database for my current table structure:
Tables:
ServiceItems {Id, name, user, date_created}
ServiceItemsProps {fk_to_ServiceItems Item_id, Id, key, value}
I'm trying to select items from the ServiceItem table with multiple keys from the ServiceItemsProps table as columns.
I can accomplish this with a query like the following:
> select tbl1.value as bouncebacks, tbl2.value as assignees from
> service_items join service_item_props as tbl1 on tbl1.item_id =
> service_items.id join service_item_props as tbl2 on tbl2.item_id =
> service_items.id where service_items.item_type='CARD' and
> tbl1.key='bouncebacks' and tbl2.key='assignees'
But I'm not able to figure out how to reproduce this in Django's ORM. I would like to not inject raw SQL into the statements here, because codebase portability is important.
Section of models.py
class ServiceItems(models.Model):
class Meta:
db_table = 'service_items'
unique_together = ('service', 'item_type', 'item_id')
service = models.ForeignKey(Service, blank=False, db_column='service_id', on_delete=models.CASCADE)
item_type = models.CharField(max_length=255, blank=False)
url = models.TextField(blank=True, null=True)
item_id = models.TextField(blank=True, null=True)
item_creation_user = models.TextField(blank=True, null=True)
item_creation_date = models.DateTimeField(blank=True, null=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class ServiceItemProps(models.Model):
class Meta:
db_table = 'service_item_props'
item = models.ForeignKey(ServiceItems, blank=False, db_column='item_id', on_delete=models.CASCADE)
prop_id = models.TextField(blank=True, null=True)
key = models.CharField(max_length=255, blank=False)
value = models.TextField(blank=True, null=True)
# change one line to make it easier to query
item = models.ForeignKey(ServiceItems, blank=False, db_column='item_id', on_delete=models.CASCADE, related_name='item_props')
Query should become:
ServiceItem.objects.filter(Q(item_type='CARD') & (Q(item_props__key='bouncebacks') | Q(item_props__key='assignees'))
==============================================================
I think I misunderstood your query.
I believe this is a good case to use .raw() .
Try this one instead:
qs = ServiceItemProps.objects.raw('''
SELECT sip1.*, sip2.value as other_value
FROM {item_table} as service_items
INNER JOIN {props_table} as sip1 on sip1.item_id = service_items.id
INNER JOIN {props_table} as sip2 on sip2.item_id = service_items.id
WHERE service_items.item_type='CARD' and sip1.key='bouncebacks' and sip2.key='assignees'
'''.format(item_table=ServiceItems._meta.db_table, props_table=ServiceItemProps._meta.db_table)
for itemprop in qs:
print(qs.value, qs.other_value)

Django: Are multiple Generic Relations in one Model bad design?

I have a model with multiple Generic Relations that has become very complicated to use in my templates. The model is a 'Gig' or musical event that takes place at a 'Venue' and/or a 'Festival' and has a 'Musician' and/or an 'Ensemble'.
Where it gets complicated is that each 'Gig' has a presenter, promoter and an agent. These are setup as generic relations to other models such as 'PresenterCompany'. A Presenter company could be a promoter, presenter, or agent, or all of them for the same gig. Here are the models (simplified for ref):
class Gig(models.Model):
description = models.CharField(max_length=100, blank=True)
date = models.DateTimeField()
venue = models.ForeignKey(Venue)
festival = models.ForeignKey(Festival, blank = True, null=True)
musician = models.ManyToManyField(Musician, blank=True)
ensembles = models.ManyToManyField(Ensemble, blank = True)
presenter_content_type = models.ForeignKey(ContentType,
limit_choices_to={"model__in": ("Individual", "ArtsOrganization",'Presenter', "BookingAgent","Festival", "OtherOrganization","PresenterCompany", "Venue")}, related_name = "Presenter Type", verbose_name = "Presenter",blank=True, null=True)
presenter_id = models.IntegerField(db_index=True, blank=True, null=True, verbose_name='Presenter ID')
presenter = generic.GenericForeignKey('presenter_content_type','presenter_id')
promoter_content_type = models.ForeignKey(ContentType,
limit_choices_to={"model__in": ("Individual", "ArtsOrganization","BookingAgent","Presenter", "Festival", "OtherOrganization","PresenterCompany", "Venue")}, related_name = "promotor", verbose_name='Promoter Type', blank=True, null=True)
promoter_id = models.IntegerField(db_index=True, blank=True, null=True, verbose_name='Promoter ID')
promoter = generic.GenericForeignKey('promoter_content_type','promoter_id')
agent_content_type = models.ForeignKey(ContentType,
limit_choices_to={"model__in": ("Individual", "BookingAgent")}, related_name="agent", verbose_name='Agent Type', blank=True, null=True)
agent_id = models.IntegerField(db_index=True, blank=True, null=True, verbose_name='Agent ID')
agent = generic.GenericForeignKey('agent_content_type','agent_id')
class PresenterCompany(models.Model):
name = models.CharField(max_length=70)
address =GenericRelation(Address)
presented_gig = GenericRelation('Gig',
content_type_field='presenter_content_type',
object_id_field='presenter_id',
related_name='presenter_presented_gig'
)
promoted_gig = GenericRelation('Gig',
content_type_field='promoter_content_type',
object_id_field='promoter_id',
related_name='presenter_promoted_gig'
)
booked_gig = GenericRelation('Gig',
content_type_field='promoter_content_type',
object_id_field='promoter_id',
related_name='presenter_booked_gig'
)
The main issue is that when I try to get all of the gigs for a presenter company, I have to write three different for loops for each role i.e. {% for gig in presentercompany.presented_gig.all %}, and so on... This seems like redundant code.
Is there a better way to structure this such as using intermediary models for presenter, promoter, and agent? Thanks for your advice!
Generic relationships can definitely be hard to deal with. I would only use them when there is no other option.
In your case, I see a couple other options. You could have a ManyToMany relationship between PresenterCompany and Gig using a through table to specify the type of relationship (https://docs.djangoproject.com/en/2.0/topics/db/models/#extra-fields-on-many-to-many-relationships):
class Gig(models.Model):
description = models.CharField(max_length=100, blank=True)
date = models.DateTimeField()
venue = models.ForeignKey(Venue)
festival = models.ForeignKey(Festival, blank=True, null=True)
musician = models.ManyToManyField(Musician, blank=True)
ensembles = models.ManyToManyField(Ensemble, blank=True)
class PresenterCompanyGigRelationship(models.Model):
gig = models.ForeignKey(Gig, on_delete=models.CASCADE)
presenter_company = models.ForeignKey(
'PresenterCompany', on_delete=models.CASCADE)
relationship = models.CharField(
max_length=10,
choices=(
('presenter', 'Presenter'),
('promoter', 'Promoter'),
('agent', 'Agent'),
))
class PresenterCompany(models.Model):
name = models.CharField(max_length=70)
git = models.ManyToManyField(Gig, through=PresenterCompanyGigRelationship)