LOOP AT... GROUP BY with dynamic group key - grouping

I am trying to loop by grouping data with dynamic group parameter.
We can use dynamic queries on WHERE conditions on loops but I do not know whether it is possible to use a dynamic string in group condition.
Here is the sample where user decides by which field to group and then put additional logic based on the decision:
DATA query TYPE string.
IF i_condition_type = 'ERNAM'.
query = |ERNAM = MARA-ERNAM|.
ELSE.
query = |ERSDA = MARA-ERSDA|.
ENDIF.
LOOP AT lt_mara INTO DATA(mara) GROUP BY ( (query) ) "syntax error
ASSIGNING FIELD-SYMBOL(<group>).
LOOP AT GROUP <group> ASSIGNING FIELD-SYMBOL(<line_data>).
"//do something
ENDLOOP.
ENDLOOP.
Is there any way to do this? I am also open to other ideas besides grouping because if I can't dynamically group, I will copy so many lines and change only the group key.

As pointed out in the comments, LOOP AT ... GROUP BY doesn't support dynamic group-by clauses from strings.
In this simple example you could create your grouping key dynamically at runtime by creating it with an inline expression like COND or SWITCH:
LOOP AT lt_mara INTO DATA(mara) GROUP BY
SWITCH string(
i_condition_type
WHEN 'ERNAM' THEN mara-ernam
WHEN 'ERSDA' THEN mara-ersda
ELSE ''
)
But your key-building logic might be too complex to express with an expression (or at least an expression which is still readable by a human being). In that case there is something else you can do: group on values returned by a method:
LOOP AT lt_mara INTO DATA(mara)
GROUP BY my_grouping_method( line = mara
condition = i_condition_type )
The implementation of that method can then include any logic you need to form the grouping key at runtime:
METHOD my_grouping_method.
IF condition = 'ERNAM'.
result = line-ernam.
ELSE.
result = line-ersda.
ENDIF.
ENDMETHOD.
The grouping method can also be a method of a different object. So you could represent your grouping condition as an own class. That would allow you to write code like this:
DATA(lo_group_condition) = NEW zcl_mara_group_condition( 'ERNAM' ).
LOOP AT lt_mara INTO DATA(mara)
GROUP BY lo_group_condition->get_key_from( mara )

Related

Django annotate and LEFT OUTER JOIN with desired WHERE Clause

Django 1.10.6
Asset.objects.annotate(
coupon_saved=Count(
Q(coupons__device_id='8ae83c6fa52765061360f5459025cb85e6dc8905')
)
).all().query
produces the following query:
SELECT
"assets_asset"."id",
"assets_asset"."title",
"assets_asset"."description",
"assets_asset"."created",
"assets_asset"."modified",
"assets_asset"."uid",
"assets_asset"."org_id",
"assets_asset"."subtitle",
"assets_asset"."is_active",
"assets_asset"."is_generic",
"assets_asset"."file_standalone",
"assets_asset"."file_ios",
"assets_asset"."file_android",
"assets_asset"."file_preview",
"assets_asset"."json_metadata",
"assets_asset"."file_icon",
"assets_asset"."file_image",
"assets_asset"."video_mobile",
"assets_asset"."video_standalone",
"assets_asset"."file_coupon",
"assets_asset"."where_to_buy",
COUNT("games_coupon"."device_id" = 8ae83c6fa52765061360f5459025cb85e6dc8905) AS "coupon_saved"
FROM
"assets_asset"
LEFT OUTER JOIN
"games_coupon"
ON ("assets_asset"."id" = "games_coupon"."asset_id")
GROUP BY
"assets_asset"."id"
I need to get that device_id=X into LEFT OUTER JOIN definition below.
How to achieve?
TL;DR:
The condition should be in filter.
qs = (
Asset.objects
.filter(coupons__device_id='8ae83c6fa52765061360f5459025cb85e6dc8905')
.annotate(coupon_saved=Count('coupons'))
)
If you want only count > 0 then it can be filtered.
qs = qs.filter(coupon_saved__gt=0)
Footnotes: A one to many query is compiled to LEFT OUTER JOIN in order to be possible to get also base objects (Asset) with zero children. JOINs in Django are based every times on a ForeignKey to the primary key or similarly on OneToOne or ManyToMany, other conditions are compiled to WHERE.
Conditions in annotation (that you used) are possible e.g. as part of Conditional Expressions but it is more complicated to be used correctly and useful e.g. if you want to get many aggregations with many conditions by one query without subqueries and if a full scan is acceptable. This is probably not a subject of a question.

How to create a filter from request.GET parameters? [duplicate]

This question already has an answer here:
Closed 10 years ago.
Possible Duplicate:
filter using Q object with dynamic from user?
I am working on a filter feature in my app. I send a comma separated string via jquery to Django (within jquery I replace the space with a +, so that it can be sent over the wire).
/?ajax&sales_item=t2,+t1
Now in the view when I retrieve the GET parameters, I can see Django has already replaced the + with a space, which is great. Then I split the keywords by comma and strip the whitespace.
sales_item_raw = request.GET['sales_item']
sales_item_keywords = sales_item_raw.split(',')
I need first to check if the given names even exist as sales item. I have to use a icontains, hence sales_items can be more than one item.
for item in sales_item_keywords:
sales_items = profile.company.salesitem_set.filter(item_description__icontains=item.strip())
Last but not least the queryset is used to filter deals for the given sales_items:
deals_queryset = deals_queryset.filter(sales_item__in=sales_items)
If the user filters for only one keyword that would work fine, however if there are two keywords the sales_items will be obviously overwritten in each loop iteration.
What is the most performant way to solve this? Shall I just append the content of sales_itemsin each iteration to a list outside the loop? And eventually send the new list to the final deals_queryset.filter?
I am not sure if this is a good way to solve this...
Use Django's Q object to create "or" logic in your filter.
# create a chain of Qs, one for each item, and "or" them together
q_filters = Q(item_description__icontains=sales_item_keywords[0].strip())
for item in sales_item_keywords[1:]:
q_filters = q_filters | Q(item_description__icontains=item.strip())
# do a single filter with the chained Qs
profile.company.salesitem_set.filter(q_filters)
This is ugly code, as I'm not sure how to handle the initial Q elegantly, as I'm not sure what is an "empty" Q to which you can chain all other Qs, including the first one. (I'm guessing you could use Q(pk=pk) but that's ugly in a different way.)
EDIT: Ignacio's link above shows the way, i.e.
q_filters = reduce(operator.or_, (Q(item_description__icontains=item.strip()) for item in sales_items_keywords))
profile.company.salesitem_set.filter(q_filters)

Using .extra() on fields created by .annotate() in Django

I want to retrieve a sum of two fields (which are aggregations themselves) for each object in a table.
The following may describe a bit better what I'm after but results in an Unknown column in field list-Error:
items = MyModel.objects.annotate(
field1=Sum("relatedModel__someField"),
field2=Sum("relatedModel__someField")).extra(
select={"sum_field1_field2": "field1 + field2"})
I also tried using F() for the field lookups but that gives me an invalid sql statement.
Any ideas on how to solve this are much appreciated.
it this what you want?
items = MyModel.objects.extra(
select = {'sum_field1_field2': 'SUM(relatedModel__someField) + SUM(relatedModel__someField)'},
)
To make it work for many to many or for many to one (reverse) relations, you may use the following:
items = MyModel.objects.extra(
select = {'sum_field1_field2': 'SUM("relatedModel"."someField") + SUM("relatedModel"."someField")'},
)
But this will break also if you need another annotate, like for a count, because extra will add the statement to the GROUP BY clause, whereas aggregate functions are not allowed in there.

Word count query in Django

Given a model with both Boolean and TextField fields, I want to do a query that finds records that match some criteria AND have more than "n" words in the TextField. Is this possible? e..g.:
class Item(models.Model):
...
notes = models.TextField(blank=True,)
has_media = models.BooleanField(default=False)
completed = models.BooleanField(default=False)
...
This is easy:
items = Item.objects.filter(completed=True,has_media=True)
but how can I filter for a subset of those records where the "notes" field has more than, say, 25 words?
Try this:
Item.objects.extra(where=["LENGTH(notes) - LENGTH(REPLACE(notes, ' ', ''))+1 > %s"], params=[25])
This code uses Django's extra queryset method to add a custom WHERE clause. The calculation in the WHERE clause basically counts the occurances of the "space" character, assuming that all words are prefixed by exactly one space character. Adding one to the result accounts for the first word.
Of course, this calculation is only an approximation to the real word count, so if it has to be precise, I'd do the word count in Python.
I dont know what SQL need to be run in order for the DB to do the work, which is really what we want, but you can monkey-patch it.
Make an extra fields named wordcount or something, then extend the save method and make it count all the words in notes before saving the model.
The it is trivial to loop over and there is still no chance that this denormalization of data will break since the save method is always run on save.
But there might be a better way, but if all else fails, this is what I would do.

How to filter empty or NULL names in a QuerySet?

I have first_name, last_name & alias (optional) which I need to search for. So, I need a query to give me all the names that have an alias set.
Only if I could do:
Name.objects.filter(alias!="")
So, what is the equivalent to the above?
You could do this:
Name.objects.exclude(alias__isnull=True)
If you need to exclude null values and empty strings, the preferred way to do so is to chain together the conditions like so:
Name.objects.exclude(alias__isnull=True).exclude(alias__exact='')
Chaining these methods together basically checks each condition independently: in the above example, we exclude rows where alias is either null or an empty string, so you get all Name objects that have a not-null, not-empty alias field. The generated SQL would look something like:
SELECT * FROM Name WHERE alias IS NOT NULL AND alias != ""
You can also pass multiple arguments to a single call to exclude, which would ensure that only objects that meet every condition get excluded:
Name.objects.exclude(some_field=True, other_field=True)
Here, rows in which some_field and other_field are true get excluded, so we get all rows where both fields are not true. The generated SQL code would look a little like this:
SELECT * FROM Name WHERE NOT (some_field = TRUE AND other_field = TRUE)
Alternatively, if your logic is more complex than that, you could use Django's Q objects:
from django.db.models import Q
Name.objects.exclude(Q(alias__isnull=True) | Q(alias__exact=''))
For more info see this page and this page in the Django docs.
As an aside: My SQL examples are just an analogy--the actual generated SQL code will probably look different. You'll get a deeper understanding of how Django queries work by actually looking at the SQL they generate.
Name.objects.filter(alias__gt='',alias__isnull=False)
Firstly, the Django docs strongly recommend not using NULL values for string-based fields such as CharField or TextField. Read the documentation for the explanation:
https://docs.djangoproject.com/en/dev/ref/models/fields/#null
Solution:
You can also chain together methods on QuerySets, I think. Try this:
Name.objects.exclude(alias__isnull=True).exclude(alias="")
That should give you the set you're looking for.
1. When using exclude, keep the following in mind to avoid common mistakes:
Should not add multiple conditions into an exclude() block like filter(). To exclude multiple conditions, you should use multiple exclude().
Example: (NOT a AND NOT b)
Entry.objects.exclude(title='').exclude(headline='')
equal to
SELECT... WHERE NOT title = '' AND NOT headline = ''
======================================================
2. Only use multiple when you really know about it:
Example: NOT (a AND b)
Entry.objects.exclude(title='', headline='')
equal to
SELECT.. WHERE NOT (title = '' AND headline = '')
If you want to exclude null (None), empty string (""), as well as a string containing white spaces (" "), you can use the __regex along with __isnull filter option
Name.objects.filter(
alias__isnull = False,
alias__regex = r"\S+"
)
alias__isnull=False excludes all the columns null columns
aliax__regex = r"\S+" makes sure that the column value contains at least one or more non whitespace characters.
From Django 1.8,
from django.db.models.functions import Length
Name.objects.annotate(alias_length=Length('alias')).filter(alias_length__gt=0)
You can simply do this:
Name.objects.exclude(alias="").exclude(alias=None)
It's really just that simple. filter is used to match and exclude is to match everything but what it specifies. This would evaluate into SQL as NOT alias='' AND alias IS NOT NULL.
Another approach using a generic isempty lookup, that can be used with any field.
It can also be used by django rest_framework or other apps that use django lookups:
from distutils.util import strtobool
from django.db.models import Field
from django.db.models.lookups import BuiltinLookup
#Field.register_lookup
class IsEmpty(BuiltinLookup):
lookup_name = 'isempty'
prepare_rhs = False
def as_sql(self, compiler, connection):
sql, params = compiler.compile(self.lhs)
condition = self.rhs if isinstance(self.rhs, bool) else bool(strtobool(self.rhs))
if condition:
return "%s IS NULL or %s = ''" % (sql, sql), params
else:
return "%s <> ''" % sql, params
You can then use it like this:
Name.objects.filter(alias__isempty=False)
this is another simple way to do it .
Name.objects.exclude(alias=None)