Related
I have a list of devices in my database, like this:
id: 1, category_json: [1, 3]
id: 2, category_json: [1, 4]
id: 3, category_json: [4, 35]
The field 'category_json' is a JSONField. I get an array of categories that the users wants from the frontend like this:
categories = [3]
I need to make a query in django to look up for all devices that has any of the categories above, I did this in django:
category_filtering = None
for category in categories:
if not category_filtering:
category_filtering = Q(category_json__contains=category)
else:
category_filtering = category_filtering | Q(category_json__contains=category)
all_devices: QuerySet = Device.objects.filter(Q(status=1) & Q(company__in=companies)) & category_filtering
That code above reproduces the following query:
SELECT `devices`.`id`, `devices`.`serial`, `devices`.`title`, `devices`.`product_id`, `devices`.`company_id`, `devices`.`server_access_key`, `devices`.`device_local_access_key`, `devices`.`mac_address`, `devices`.`status`, `devices`.`owner_device_id`, `devices`.`connected_equipament`, `devices`.`gps_x`, `devices`.`gps_y`, `devices`.`profile_id`, `devices`.`softwareVer`, `devices`.`hardwareVer`, `devices`.`isVirtual`, `devices`.`timezone`, `devices`.`installed_phase`, `devices`.`installation_local`, `devices`.`group`, `devices`.`relay_state`, `devices`.`last_sync`, `devices`.`phases_ordering`, `devices`.`created_at`, `devices`.`installation_date`, `devices`.`installation_state`, `devices`.`last_change`, `devices`.`last_online`, `devices`.`comments`, `devices`.`enable_bbd_home`, `devices`.`enable_bbd_pro`, `devices`.`enable_tseries_analytics`, `devices`.`enable_gd_production`, `devices`.`card_profile_id`, `devices`.`firmware_app_ver`, `devices`.`firmware_metrol_ver`, `devices`.`commissioning_date`, `devices`.`state`, `devices`.`area_square_meters`, `devices`.`category`, `devices`.`local`, `devices`.`code`, `devices`.`enable_advanced_monitoring`, `devices`.`send_data`, `devices`.`olson_timezone`, `devices`.`language`, `devices`.`extra_fields`, `devices`.`category_json` FROM `devices` WHERE (`devices`.`status` = 1 AND `devices`.`company_id` IN (SELECT U0.`id` FROM `companies` U0 WHERE (U0.`id` IN ((
WITH RECURSIVE ids(id) as (
select 297 as id
union all
select companies.id as id from ids join companies where ids.id = companies.company_owner
)
select id from ids)) AND U0.`is_active` = True)) AND JSON_CONTAINS(`devices`.`category_json`, (CAST("3" AS JSON))))
In django is returning an empty Queryset, like this:
all_devices <QuerySet []>
But when I just copy this query and run in MySQL Workbench, it works fine, it brings the device I want. What am I doing wrong??
Sorry for my bad english
I'm retrieving all records, and I would like to display the record's age for those records that are older than 5 minutes.
The output should be something like this (in this example, two records: 1.8.9.1 and 2.7.3.1 are older than 5 minutes) :
ip ... status
---------------------
1.8.9.1 ... 3 hours
2.7.3.1 ... 7 minutes
1.1.1.1 ... up
1.1.1.2 ... up
1.1.1.3 ... up
1.1.1.4 ... up
1.1.1.5 ... up
Here's my current code:
Interfaces.objects.all()
.annotate(
age = (datetime.utcnow() - F('timestamp')), # 0:00:08.535704
age2 = Epoch(datetime.utcnow() - F('timestamp')), # 8.535704
# age3 = int(Epoch(datetime.utcnow() - F('timestamp'))/300),
current_time=Value(str(datetime.utcnow()),
output_field=null_char_field),
)
.order_by('age','ip')
age and age2 both work, but the problem is that I want the records that are older than 5 minutes sorted by age, and the rest by ip
So I'm trying to set age to 0, if it's less than 5 minutes.
If I would do it directly in postgresql, I'd use this query:
select ip, <other fields>,
case when extract('epoch' from now() - "timestamp") > 300
then extract('epoch' from now() - "timestamp")
else 0
end
Is there a way to do it in django?
I figured it out:
Interfaces.objects.all()
.annotate(
age=Case(
When(timestamp__lt=datetime.utcnow() - timedelta(minutes=5),
then=Cast(Epoch(datetime.utcnow() - F('timestamp')),
NullIntegerField)),
default=0,
output_field=NullIntegerField
),
)
.order_by('age','ip')
By the way, my imports and relevant settings:
from django.db.models import F, Func, Case, When, IntegerField
from django.db.models.functions import Coalesce, Cast
NullIntegerField = IntegerField(null=True)
class Epoch(Func):
function = 'EXTRACT'
template = "%(function)s('epoch' from %(expressions)s)"
This website ended up being the most helpful: https://micropyramid.com/blog/django-conditional-expression-in-queries/
You can do it in other way also which will be faster.
Get current time, subtract from that 5 minutes, after that search all the Interfaces
where age is less or equal than the subtracted date.
example:
current_time = datetime.now()
older_than_five = current_time - datetime.timedelta(minutes=5)
Interfaces.objects.all()
.annotate(
age=Case(
When(age__lt=older_than_five, then=Value(0)),
default=F('age')
)
)
.order_by('age','ip')
I have a simple query in Django. I have Django toolbar installed to check the SQL queries and the corresponding data
My model:
class RecipePosition(models.Model):
name = models.CharField(max_length=200,blank=True,help_text="If left blank will be same as Ingredient name Eg: Tomato pulp")
mass_quantity = models.DecimalField(max_digits=19, decimal_places=10,null=True,blank=True,default=0,validators=[MinValueValidator(0)])
title = models.CharField(max_length=200,blank=True)
updated = models.DateTimeField(auto_now=True, auto_now_add=False)
timestamp = models.DateTimeField(auto_now=False, auto_now_add=True)
I have the below django query with filter.
RecipePosition.objects.all().filter(mass_quantity__gt = 0)
Django gets all the objects whose mass_quantity is greater than 0.
But when i check the sql in the django - toolbar. it shows:
SELECT "recipes_recipeposition"."id",
"recipes_recipeposition"."name",
"recipes_recipeposition"."mass_quantity",
"recipes_recipeposition"."title",
"recipes_recipeposition"."updated",
"recipes_recipeposition"."timestamp"
FROM "recipes_recipeposition"
WHERE "recipes_recipeposition"."mass_quantity" > 'Decimal(''0'')'
ORDER BY "recipes_recipeposition"."sequence_number" ASC
I tried this command in sqlite browser also, but it didn't show any results.
Why django-toolbar is not showing the correct SQL?
As per me the sql should be:
SELECT "recipes_recipeposition"."id",
"recipes_recipeposition"."name",
"recipes_recipeposition"."mass_quantity",
"recipes_recipeposition"."title",
"recipes_recipeposition"."updated",
"recipes_recipeposition"."timestamp"
FROM "recipes_recipeposition"
WHERE "recipes_recipeposition"."mass_quantity" > 0
ORDER BY "recipes_recipeposition"."sequence_number" ASC
and this when tested in sqlite browser shows the results.
Also when I tested this on shell_plus with --print-sql --ipython shows
$ python manage.py shell_plus --print-sql --ipython
System check identified some issues:
# Shell Plus Model Imports
from recipes.models import Recipe, RecipePosition
Python 3.6.4 (default, Jan 5 2018, 02:35:40)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: RecipePosition.objects.all().filter(mass_quantity__gt=0)
Out[1]: SELECT "recipes_recipeposition"."id",
"recipes_recipeposition"."name",
"recipes_recipeposition"."mass_quantity",
"recipes_recipeposition"."title",
"recipes_recipeposition"."updated",
"recipes_recipeposition"."timestamp"
FROM "recipes_recipeposition"
WHERE "recipes_recipeposition"."mass_quantity" > '0'
ORDER BY "recipes_recipeposition"."sequence_number" ASC
LIMIT 21
Only on django-toolbar it shows Decimal() thing here on Django shell it shows WHERE "recipes_recipeposition"."mass_quantity" > '0'
I also tried debugsqlshell as mentioned in the django-toolbar documentation. Its shows "recipes_recipeposition"."mass_quantity" > '0' rather than "recipes_recipeposition"."mass_quantity" > 'Decimal(''0'')'
$ python manage.py debugsqlshell
Python 3.6.4 (default, Jan 5 2018, 02:35:40)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [2]: from recipes.models import Recipe, RecipePosition
In [3]: RecipePosition.objects.all().filter(mass_quantity__gt = 0)
Out[3]: SELECT "recipes_recipeposition"."id",
"recipes_recipeposition"."name",
"recipes_recipeposition"."mass_quantity",
"recipes_recipeposition"."title",
"recipes_recipeposition"."updated",
"recipes_recipeposition"."timestamp"
FROM "recipes_recipeposition"
WHERE "recipes_recipeposition"."mass_quantity" > '0'
ORDER BY "recipes_recipeposition"."sequence_number" ASC
LIMIT 21 [1.58ms]
I dont know why django-toobar is using "recipes_recipeposition"."mass_quantity" > 'Decimal(''0'')' instead of "recipes_recipeposition"."mass_quantity" > '0'
I want to rely on django-toolbar, but now i am worried.
Good news, I think you need to make the most minute of changes!
Instead of:
RecipePosition.objects.all().filter(mass_quantity__gt = 0)
You require:
RecipePosition.objects.all().filter(mass_quantity__gt=0.0)
Finally after lot of struggle and going through the code. I did the following changes in the source code and then everything worked the way i want.
####################
http://127.0.0.1:8001/static/debug_toolbar/css/toolbar.css
# By doing this the sql will show in multiple lines with indent
#djDebug .djDebugSql {
#word-break:break-word;
z-index:100000002;
}
#####################
# replace \n with <br> and space with nbsp and dont use Boldkeyword
# By doing this the sql will show in multiple lines with indent
def reformat_sql(sql):
stack = sqlparse.engine.FilterStack()
options = formatter.validate_options({'reindent':True,'indent_width':True,})
stack = formatter.build_filter_stack(stack, options)
#stack.preprocess.append(BoldKeywordFilter()) # add our custom filter
stack.postprocess.append(sqlparse.filters.SerializerUnicode()) # tokens -> strings
#return swap_fields(''.join(stack.run(sql)))
return swap_fields(''.join(stack.run(sql)).replace("\n", "<br/>").replace(" ", " "))
#####################
in file /lib/python3.6/site-packages/debug_toolbar/panels/sql/tracking.py
# because of this the greater than 0 is shown as deciman(0.0)
# change for decimal wrap p in rev_typecast_decimal
def _record(self, method, sql, params):
start_time = time()
try:
return method(sql, params)
finally:
stop_time = time()
duration = (stop_time - start_time) * 1000
if dt_settings.get_config()['ENABLE_STACKTRACES']:
stacktrace = tidy_stacktrace(reversed(get_stack()))
else:
stacktrace = []
_params = ''
try:
_params = json.dumps([self._decode(rev_typecast_decimal(p)) for p in params])
#_params = json.dumps([self._decode(p) for p in params]) I
###########################
hare = []
if params is not None:
hare = [self._decode(rev_typecast_decimal(p)) for p in params]
try:
_params = json.dumps([self._decode(rev_typecast_decimal(p)) for p in params])
# _params = json.dumps([self._decode(p) for p in params])
except Exception:
pass # object not JSON serializable
template_info = get_template_info()
alias = getattr(self.db, 'alias', 'default')
conn = self.db.connection
vendor = getattr(conn, 'vendor', 'unknown')
params1 = {
'vendor': vendor,
'alias': alias,
# 'sql': self.db.ops.last_executed_query(
# self.cursor, sql, self._quote_params(params)),
'sql': self.db.ops.last_executed_query(
self.cursor, sql, hare),
'duration': duration,
'raw_sql': sql,
'params': _params,
'stacktrace': stacktrace,
'start_time': start_time,
'stop_time': stop_time,
'is_slow': duration > dt_settings.get_config()['SQL_WARNING_THRESHOLD'],
'is_select': sql.lower().strip().startswith('select'),
'template_info': template_info,
}
################################################
The final out put looks like, with decimal(0.0) replaces with 0 and well formatted sql
SELECT "recipes_recipeposition"."id",
"recipes_recipeposition"."name",
"recipes_recipeposition"."recipe_id",
"recipes_recipeposition"."ingredient_id",
"recipes_recipeposition"."recipeposition_slug",
"recipes_recipeposition"."cooking_unit",
"recipes_recipeposition"."mass_unit_id",
"recipes_recipeposition"."mass_quantity",
"recipes_recipeposition"."volume_unit_id",
"recipes_recipeposition"."volume_quantity",
"recipes_recipeposition"."pieces_unit_id",
"recipes_recipeposition"."pieces_quantity",
"recipes_recipeposition"."cooking_notes",
"recipes_recipeposition"."sequence_number",
"recipes_recipeposition"."title",
"recipes_recipeposition"."updated",
"recipes_recipeposition"."timestamp",
"ingredients_ingredient"."rate" AS "ingredient__rate",
CASE
WHEN "ingredients_ingredient"."munit" = 'kg' THEN 'kg'
WHEN "ingredients_ingredient"."munit" = 'ltr' THEN 'ltr'
WHEN "ingredients_ingredient"."munit" = 'pcs' THEN 'pcs'
ELSE 'False'
END AS "ingredient__cost_unit",
CASE
WHEN "ingredients_ingredient"."munit" = 'kg' THEN CASE
WHEN ("recipes_recipeposition"."mass_unit_id" IS NOT NULL
AND "recipes_recipeposition"."mass_quantity" IS NOT NULL
AND "recipes_recipeposition"."mass_quantity" > '0') THEN CAST(("recipes_recipeposition"."mass_quantity" * "single_measurements_singlemeasurements"."quantity") AS NUMERIC)
ELSE 'False'
END
WHEN "ingredients_ingredient"."munit" = 'ltr' THEN 'ltr'
WHEN "ingredients_ingredient"."munit" = 'pcs' THEN 'pcs'
ELSE 'False'
END AS "reciposition_cost_quantity"
FROM "recipes_recipeposition"
LEFT OUTER JOIN "ingredients_ingredient" ON ("recipes_recipeposition"."ingredient_id" = "ingredients_ingredient"."id")
LEFT OUTER JOIN "single_measurements_singlemeasurements" ON ("recipes_recipeposition"."mass_unit_id" = "single_measurements_singlemeasurements"."id")
WHERE "recipes_recipeposition"."recipe_id" = '1'
ORDER BY "recipes_recipeposition"."sequence_number" ASC
In my application I have models Visits & Post &
class Visit < ActiveRecord::Base
belongs_to :post, :counter_cache => true
class Post < ActiveRecord::Base
has_many :visits
When a visitor visits a post, I am adding it to my visits table with post_id and price (price is decimal).
In my dashboard, I want to show which posts they have viewed (grouped) and how much they have earned.
For instance:
1) post 1, viewed 54 times and earned $1.6, 2) post 2, viewed 39 times and earned $1.1, etc
I have tried with:
- a = Visit.group(:post_id).where(user: current_user).sum(:price)
- a.each do |n|
%p
= n
This gives me, each post_id, but price is just * BigDecimal:7fb2625f9238,'0.15E1* & I can't find post title by doing n.post.title & n.post.title gives me error: undefined method 'post'
This is result I get:
[44, #<BigDecimal:7fb2625f9238,'0.15E1',18(36)>]
[45, #<BigDecimal:7fb2625f8dd8,'0.13E1',18(36)>]
[46, #<BigDecimal:7fb2625f8928,'0.3E-1',9(36)>]
I have also tried with:
- Visit.select([:post_id, :price]).where(user: current_user).each do |e|
%p
= e.post.title
= e.cpc_bid
This option gives me all the posts and prices individually and not combined.
Results are like:
Post title 1, 0.15
Post title 1, 0.01
Post title 2, 0.1
Post title 1, 0.15
Post title 2, 0.1
Post title 2, 0.1
Post title 2, 0.1
Post title 1, 0.15
I also tried with:
- Visit.select([:post_id, :price]).group(:post_id).where(user: current_user).each do |e|
%p
= e.post.title
= e.price
This option gives me only one of the visits on the post with its price.
Results are:
Post title 2, 0.1
Post title 1, 0.15
My last try was:
- Visit.joins(:post).group(:post_id).select('sum(price) as earnings', :post_id, :title, 'count(visits.id) as total_views').where(user: current_user).each do |e|
%p
= e.title
= e.price
This gives me this error:
PG::GroupingError: ERROR: column "posts.title" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...ECT sum(price) as earnings, "visits"."post_id", "title", c...
How can I combine them together with sum of price on all post, with its post title.
You need to join tables and group
Post.joins(:visits).group(:id)
.where(visits: { user_id: current_user.id})
.select("*, sum(price) as total_price, count(visits.id) as total_views")
It adds to post instance accessors total_price and total_views.
Answer from #MikDiet works perfectly, only issue I had was that I was getting error from PostgreSql:
PG::GroupingError: ERROR: column "posts.title" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...ECT sum(price) as earnings, "visits"."post_id", "title", c...
and I changed it to:
Visit.joins(:post).group([:post_id, :title]).select('sum(cpc_bid) as earnings', :post_id, :title, 'count(visits.id) as total_views').where(influencer: current_user)
And it worked.
NB: I couldn't find the answer without #MikDiet's help and all credits goes to him
I have below fields in my Django model
class cube(models.Model):
pid = models.IntegerField()
lc = models.CharField(max_length=100)
sid = models.IntegerField()
st = models.IntegerField()
wid = models.IntegerField()
wt = models.IntegerField()
I have below input set coming from clients
input_set1 object -
[{"sid":1,"st":7},{"sid":7,"st":5},{"sid":5,"st":9},{"sid":2,"st":7}]
input_set2 object -
[{"wid":3,"wt":5},{"wid":1,"wt":7},{"wid":4,"wt":8},{"wid":2,"wt":5},{"wid":5,"wt":5}]
Below is my requirement which is in mysql notation :
select pid,lc from cube
where (((sid=1) AND (st>=7)) AND ((sid=7) AND (sid>=5)) AND ((sid=5) AND
(st>=9)) AND ((sid=2) AND (st>=7)))
AND (((wid=3) AND (wt>=5)) AND ((wid=1) AND (wt>=7)) AND ((wid=4) AND
(wt>=8)) AND ((wid=2) AND (wt>=5)) AND ((wid=5) AND (wt>=5)))
Input {sid,st} and {wid,wt} sets can contain upto 10 items each -
like {sid,st} can be upto 10 for each input and same with {wid,wt}
How can I write this sql notation in django ?
I would like to use the input parameters as lists like below in my view, so that it would be generic for the input sets -
input_set1=[{1,7},{7,5},{5,9},{2,7}]
input_set2=[{3,5},{1,7},{4,8},{2,5},{5,5}]
fieldsReq=['pid','lc']
queryset=cube.objects.values_list(fieldsReq).filter(reduce(operator.and_, (Q(sid__contains=x) for x in ['3', '5', '6']),(Q(rt__contains=x) for x in ['4', '8', '3']))) #Am not sure how to write the condition here
I would like to map input_set1 to {sid,st}
and input_set2 to {wid,wt}, but found it difficulty in writing.
How can I achieve this in Django notation to get the fields from mysql table ?
Here's my attempt for sid, it should apply to wid as well.
from django.db.models import Q
# original input
input = [{"sid":1,"st":7},{"sid":7,"st":5},{"sid":5,"st":9},{"sid":2,"st":7}]
# a list of Q objects
queries = [Q(sid=i['sid'], st__gte=i['st']) for i in input]
# 'AND' all conditions
sid_query = reduce(lambda x, y: x & y, queries)
fieldsReq=['pid','lc']
queryset=cube.objects.values_list(fieldsReq).filter(sid_query)