I am using Django to interface with another (JAVA) application that based on some events generates at runtime tables in the db with the same model. So I don't have a direct control over the DB. For example:
Sensor1
id | value | time
1 10 2018-10-11
Sensor2
id | value | time
1 12 2018-10-11
Currently, my Django model is looking something like this:
class Sensor(models.Model):
value = models.IntegerField()
time = models.DatetimeField()
class Meta:
managed = False
db_table = "Sensor1"
Do you have any clue if I can set the model somehow to be able to gain the data from a different table based on the query?
Ideally, something that would allow me to get the data in the fashion of:
config_tables=['Sensor1','Sensor2']
for table in config_tables:
data = Sensor.objects.table(table).objects.all()
...
Other possibility might be also to have a SQL query that executes on different tables, so perhaps something like:
SELECT * FROM %s;
It seems that the best solution so far is to make a custom SQL Query so in this example it could be something like this in models.py:
def get_all_parameters(db_table):
return Parameters.objects.raw('SELECT * FROM %s' % db_table)
and call it as:
get_all_parameters('Sensor1')
or as:
TABLES = ['Sensor1', 'Sensor2']
for table in TABLES:
parameters = get_all_parameters(table)
...
Related
I have two models, the simple version would be this:
class Users:
name = models.CharField()
birthdate = models.CharField()
# other fields that play no role in calculations or filters, but I simply need to display
class UserLogs:
user_id = models.ForeignKey(to='Users', related_name='user_daily_logs', on_delete=models.CASCADE)
reference_date = models.DateField()
hours_spent_in_chats = models.DecimalField()
hours_spent_in_p_channels = models.DecimalField()
hours_spent_in_challenges = models.DecimalField()
# other fields that play no role in calculations or filters, but I simply need to display
What I need to write is a query that will return all the fields of all users, with the latest log (reference_date) for each user. So for n users and m logs, the query should return n records. It is guaranteed that each user has at least one log record.
Restrictions:
the query needs to be written in django orm
the query needs to start from the user model. So Anything that goes like Users.objects... is ok. Anything that goes like UserLogs.objects... is not. That's because of filters and logic in the viewset, which is beyond my control
It has to be a single query, and no iterations in python, pandas or itertools are allowed. The Queryset will be directly processed by a serializer.
I shouldn't have to specify the names of the columns that need to be returned, one by one. The query must return all the columns from both models
Attempt no. 1 returns only user id and the log date (for obvious reasons). However, it is the right date, but I just need to get the other columns:
test = User.objects.select_related("user_daily_logs").values("user_daily_logs__user_id").annotate(
max_date=Max("user_daily_logs__reference_date"))
Attempt no. 2 generates as error (Cannot resolve expression type, unknown output_field):
logs = UserLogs.objects.filter(user_id=OuterRef('pk')).order_by('-reference_date')[:1]
users = Users.objects.annotate(latest_log = Subquery(logs))
This seems impossible taking into account all the restrictions.
One approach would be to use prefetch_related
users = User.objects.all().prefetch_related(
models.Prefetch(
'user_daily_logs',
queryset=UserLogs.objects.filter().order_by('-reference_date'),
to_attr="latest_log"
)
)
This will do two db queries and return all logs for every user which may or not be a problem depending on the number of records. If you need only logs for the current day as the name suggest, you can add that to filter and reduce the number of UserLogs records. Of course you need to get the first element from the list.
users.daily_logs[0]
For that you can create a #property on the User model which could look roughly like this
#property
def latest_log(self):
if not hasattr('daily_logs'):
return None
return self.daily_logs[0]
user.latest_log
You can also go a step further and try the following SubQuery inside Prefetch to limit the queryset to one element but I am not sure on the performance with this one (credits Django prefetch_related with limit).
users = User.objects.all().prefetch_related(
models.Prefetch(
'user_daily_logs',
queryset=UserLogs.objects.filter(id__in=Subquery(UserLogs.objects.filter(user_id=OuterRef('user_id')).order_by('-reference_date').values_list('id', flat=True)[:1] ) ),
to_attr="latest_log"
)
)
Using the Django QuerySet API, how can I perform multiple joins between the same two tables/models? See the following untested code for illustration purposes:
class DataPacket(models.Model):
time = models.DateTimeField(auto_now_add=True)
class Field(models.Model):
packet = models.ForeignKey(DataPacket, models.CASCADE)
name = models.CharField(max_length=25)
value = models.FloatField()
I want to grab a list of data packets with only specific named fields. I tried something like this:
pp = DataPacket.prefetch_related('field_set')
result = []
for p in pp:
o = {
f.name: f.value
for f in p.field_set.all()
if f.name in ('latitude', 'longitude')
}
o['time'] = p.time
result.append(o)
But this has proven extremely inefficient because I'm working with hundreds to thousands of packets with a lot of other fields besides the latitude and longitude fields I want.
Is there a Django QuerySet call which translates into an efficient SQL query performing two inner joins from the datapacket table to the field table on different rows? I can do it with raw SQL, as follows (assuming the Django application is named myapp) (again, untested code for illustration purposes):
from django.db import connection
with connection.cursor() as cursor:
cursor.execute('''
SELECT p.time AS time, f1.value AS lat, f2.value AS lon
FROM myapp_datapacket AS p
INNER JOIN myapp_field as f1 ON p.id = f1.packet_id
INNER JOIN myapp_field as f2 ON p.id = f2.packet_id
WHERE f1.name = 'latitude' AND f2.name = 'longitude'
''')
result = list(cursor)
But instinct tells me not to use the low-level DB api if I don't have to do so. Possible reasons to back that up might be that my SQL code might not be compatible with all the DBMs Django supports, or I feel like I'm more at risk of trashing my database by misunderstanding a SQL command than I am at misunderstanding the Django API call, etc.
Try Performing raw SQL queries in django. As well as select related in raw request.
prefetch on raw query:
from django.db.models.query import prefetch_related_objects
raw_queryset = list(raw_queryset)
prefetch_related_objects(raw_queryset, ['a_related_lookup',
'another_related_lookup', ...])
Your example:
from django.db.models.query import prefetch_related_objects
raw_DataPacket = list(DataPacket.objects.raw)
pp = prefetch_related_objects(raw_DataPacket, ['field_set'])
Example of prefetch_related with Raw Queryset:
models:
class Country:
name = CharField()
class City:
country = models.ForeignKey(Country)
name = models.CharField()
prefetch_related:
from django.db.models.query import prefetch_related_objects
#raw querysets do not have len()
#thats why we need to evaluate them to list
cities = list(City.objects.raw("select * from city inner join country on city.country_id = country.id where name = 'london'"))
prefetch_related_objects(cities, ['country'])
Answer provided from information from these sources: djangoproject - performing raw queries | Related Stackoverflow Question | Google docs question
I have following models:
class Dictionary(models.Model):
word = models.CharField(unique=True)
class ProcessedText(models.Model):
text_id = models.ForeiginKey('Text')
word_id = models.ForeignKey('Dictionary')
class UserDictionary(models.Model):
word_id = models.ForeignKey('Dictionary')
user_id = models.ForeignKye('User')
I want to make query using django ORM same with next sql
SELECT * FROM ProcessedText, UserDictionary WHERE
ProcessedText.text_id = text_id
AND ProcessedText.word_id = UserDictionary.word_id
AND UserDictionary.user_id = user_id
How to do it in one query without using cycles?
This might help you:
How do I select from multiple tables in one query with Django?
And also you may have to restructure your models to enable select_related concept of django.
I have an application where users select their own display columns. Each display column has a specified formula. To compute that formula, I need to join few related columns (one-to-one relationship) and compute the value.
The models are like (this is just an example model, actual has more than 100 fields):
class CompanyCode(models.Model):
"""Various Company Codes"""
nse_code = models.CharField(max_length=20)
bse_code = models.CharField(max_length=20)
isin_code = models.CharField(max_length=20)
class Quarter(models.Model):
"""Company Quarterly Result Figures"""
company_code = models.OneToOneField(CompanyCode)
sales_now = models.IntegerField()
sales_previous = models.IntegerField()
I tried doing:
ratios = {'growth':'quarter__sales_now / quarter__sales_previous'}
CompanyCode.objects.extra(select=ratios)
# raises "Unknown column 'quarter__sales_now' in 'field list'"
I also tried using raw query:
query = ','.join(['round((%s),2) AS %s' % (formula, ratio_name)
for ratio_name, formula in ratios.iteritems()])
companies = CompanyCode.objects.raw("""
SELECT `backend_companycode`.`id`, %s
FROM `backend_companycode`
INNER JOIN `backend_quarter` ON ( `backend_companycode`.`id` = `backend_companyquarter`.`company_code_id` )
""", [query])
#This just gives empty result
So please give me a little clue as to how I can use related columns preferably using 'extra' command. Thanks.
By now the Django documentation says that one should use extra as a last resort.
So here is a query without extra():
from django.db.models import F
CompanyCode.objects.annotate(
growth=F('quarter__sales_now') / F('quarter__sales_previous'),
)
Since the calculation is being done on a single Quarter instance, where's the need to do it in the SELECT? You could just define a ratio method/property on the Quarter model:
#property
def quarter(self):
return self.sales_now / self.sales_previous
and call it where necessary
Ok, I found it out. In above using:
CompanyCode.objects.select_related('quarter').extra(select=ratios)
solved the problem.
Basically, to access any related model data through 'extra', we just need to ensure that that model is joined in our query. Using select_related, the query automatically joins the mentioned models.
Thanks :).
I have a third party Django App (Satchmo) which has a model called Product which I make extensive use of in my Django site.
I want to add the ability to search for products via color. So I have created a new model called ProductColor. This model looks roughly like this...
class ProductColor(models.Model):
products = models.ManyToManyField(Product)
r = models.IntegerField()
g = models.IntegerField()
b = models.IntegerField()
name = models.CharField(max_length=32)
When a store product's data is loaded into the site, the product's color data is used to create a ProductColor object which will point to that Product object.The plan is to allow a user to search for a product by searching a color range.
I can't seem to figure out how to put this query into a QuerySet. I can make this...
# If the color ranges look something like this...
r_range, g_range, b_range = ((3,130),(0,255),(0,255))
# Then my query looks like
colors_in_range = ProductColor.objects.select_related('products')
if r_range:
colors_in_range = colors_in_range.filter(
Q(r__gte=r_range[0])
| Q(r__lte=r_range[1])
)
if g_range:
colors_in_range = colors_in_range.filter(
Q(g__gte=g_range[0])
| Q(g__lte=g_range[1])
)
if b_range:
colors_in_range = colors_in_range.filter(
Q(b__gte=b_range[0])
| Q(b__lte=b_range[1])
)
So I end up with a QuerySet which contains all of the ProductColor objects in that color range. I could then build a list of Products by accessing the products ManyToMany attribute of each ProductColor attribute.
What I really need is a valid QuerySet of Products. This is because there is going to be other logic which is performed on these results and it needs to operate on a QuerySet object.
So my question is how can I build the QuerySet that I really want? And failing that, is there an efficient way to re-build the QuerySet (preferably without hitting the database again)?
If you want to get a Product queryset you have to filter the Product objects and filter via the reverse relation for product color:
products = Product.objects.filter(productcolor_set__r__gte=x).distinct()
You can use the range field lookup:
You can use range anywhere you can use
BETWEEN in SQL -- for dates, numbers
and even characters.
your query:
r_range, g_range, b_range = ((3,130),(0,255),(0,255))
products = Product.objects.filter(productcolor_set__r__range=r_range,
productcolor_set__g__range=g_range,
productcolor_set__b__range=b_range).distinct()