Django count manytomany relation shows wrong number - django

I currently have a queryset which aims to extract a list of used colors associated with products, and return response with a list of colors and the amount of of products they're attached to. The colors originate in its own model called ProductColor and is referenced through a ManyToMany relationship in the Product model.
In the example bellow there are 3 products which has these colors registered:
Product 1: White
Product 2: White, Red
Product 3: White
Wished output should be something like:
[
{
"colors__name": "White",
"count": 3
},
{
"colors__name": "Red",
"count": 1
},
]
However, the closest i get is:
[
{
"colors__name": "White",
"count": 1
},
{
"colors__name": "White",
"count": 1
},
{
"colors__name": "Red",
"count": 1
},
{
"colors__name": "White",
"count": 1
}
]
The queryset is structured like this:
products = Product.objects.filter(
category__parent__name__iexact=category,
status='available'
).values('colors__name', count=Count('colors', distinct=True))
I've tried to add .distinct() at the end of the queryset, but then it returns:
[
{
"colors__name": "White",
"count": 1
},
{
"colors__name": "Red",
"count": 1
}
]
I've also tried using an annotation through .annotate(count=Count('colors')), but then it returns:
[
{
"colors__name": "White",
"count": 7
},
{
"colors__name": "Red",
"count": 3
}
]
How can i make sure it displays the correct amount next to the correct color (White: 4, Red: 1)?

Related

Group queryset by field

I am working with Django and Django REST framework. I have a model called Selection that contains field called category, when i query the model to send the result to the frontend i get it with the following structure:
[
{
"id": 1,
"category": "SHOES",
"products": 122,
"created_at": "2021-09-11",
},
{
"id": 2,
"category": "SHOES",
"products": 4,
"created_at": "2021-10-07",
},
{
"id": 3,
"category": "CLOTHES",
"products": 0,
"created_at": "2021-10-08",
},
]
I need to put the selections of the same category in an array and remove the grouped-by category, like this:
{
"SHOES": [
{
"id": 1,
"products": 122,
"created_at": "2021-09-11",
},
{
"id": 2,
"products": 4,
"created_at": "2021-10-07",
}
],
"CLOTHES": [
{
"id": 3,
"category": "CLOTHES",
"products": 0,
"created_at": "2021-10-08",
}
]
}
I considered to making it with Javascript in the frontend, but before i wanted to know if there's a way to do this from Django.
Yes, you need to do some customisation in your code.
Get all categories by using values_list('category', flat=True) with your queryset
Iterate through them filtering category one by one
response_list = []
categories = Selection.objects.values_list('category', flat=True)
for category in categories:
data = Selection.objects.filter(category=category)
data = {
category: SelectionSerializer(data, many=True).data,
}
response_list.append(data)

Django ORM group by week day and sum the day of each separately

I am having trouble to query based on weekday.
This is my models:
class Sell(models.Model):
total_sell = models.IntegerField()
date = models.DateField(auto_now_add=True)
and this is my query:
weekly_sell = Sell.objects.annotate(
weekday=ExtractWeekDay('date'),
total=Sum('total_sell')
).values(
'weekday',
'total'
)
But data i am getting and it is not my expected.
Like I have an entry in the table
Sunday sell 4
Sunday sell 7
Friday sell 10
So i am expecting it should return these data:
[
[
{
"weekday": 7,
"total": 11
},
{
"weekday": 6,
"total": 0
},
{
"weekday": 5,
"total": 10
},
{
"weekday": 4,
"total": 4
},
{
"weekday": 3,
"total": 0
},
{
"weekday": 2,
"total": 0
},
{
"weekday": 1,
"total": 10
},
]
]
But problem is, it not returning data that way i want,
It is returning these:
[
[
{
"weekday": 7,
"total": 4
},
{
"weekday": 7,
"total": 5
},
{
"weekday": 5,
"total": 10
},
]
]
I don't know what's wrong with this. Can anyone please help me in this case?
In your query, grouping is being done on id first. You can confirm this by printing out the raw query: print(weekly_sell.query).
This is because, when you run an annotation, django automatically groups by the id field of the model. So you need to add .values("<<column>>") to ask django ORM to group by this column instead of id.
You should use the following query:
weekly_sell = Sell.objects.annotate(
weekday=ExtractWeekDay('date'),
).values(
'weekday',
).annotate(
total=Sum('total_sell')
).values(
'weekday',
'total'
)
If you print out the raw query for the above queryset, you will notice that now the grouping is not on id and is on weekday instead.

How to use a field as a filter group?

Currently have a list of Records that have fields child (Child model has Parent) and date.
class Record(models.Model):
child = models.ForeignKey(Child)
date = DateField()
... other fields
I would like to generate a daily list of Records that are grouped at a date level.
i.e. Parent > 2020-06-06 > Child + Record other fields
Right now I've devised a way to brute force through all the records at the serializer level if I select from the Parent level.
records = serializers.SerializerMethodField()
def get_records(self, obj):
qs = Record.objects.filter(child__parent=obj)
dates_dict = defaultdict(list)
for row in qs.all():
date = row.date.strftime('%Y-%m-%d')
dates_dict[date].append(RecordSerializer(row).data)
return dates_dict
However, this messes with pagination as it creates a single record because there's only 1 Parent. I do not need the Parent data at all. What I would like is for the data to be paginated on the date level.
{
"id": 1,
"name": "Parent",
"reports": {
"2020-05-20": [
{
"child": {
"id": 1,
},
..other fields
},
{
"child": {
"id": 2,
},
..other fields
},
{
"child": {
"id": 3,
},
..other fields
},
]
}
}
How would I paginate this if I'm filtering on the Python level? I am trying to achieve this:
{
"count": 100,
"next": "NEXT",
"prev": "PREV",
"results": {
"2020-05-20": [
{
"child": {
"id": 1,
},
..other fields
},
{
"child": {
"id": 2,
},
..other fields
},
{
"child": {
"id": 3,
},
..other fields
},
]
}
}
Thank you

How do I extract data from "List" field

I'm getting JSON data from webservice and trying to make a table . Datadisk is presented as List and clicking into each item will navigate further down the hiearchy like denoted in screenshots below. I need to concatate storageAccountType for each item with | sign, so if there were 2 list items for Greg-VM and it had Standard_LRS for first one and Premium_LRS for second one then new column will list Standard_LRS | Premium_LRS for that row.
Input returned by function is below
[
{
"name": "rhazuremspdemo",
"disk": {
"id": "/subscriptions/24ba3e4c-45e3-4d55-8132-6731cf25547f/resourceGroups/AzureMSPDemo/providers/Microsoft.Compute/disks/rhazuremspdemo_OsDisk_1_346353b875794dd4a7a5c5938abfb7df",
"storageAccountType": "StandardSSD_LRS"
},
"datadisk": []
},
{
"name": "w12azuremspdemo",
"disk": {
"id": "/subscriptions/24ba3e4c-45e3-4d55-8132-6731cf25547f/resourceGroups/AzureMSPDemo/providers/Microsoft.Compute/disks/w12azuremspdemo_OsDisk_1_09788205f8eb429faa082866ffee0f18",
"storageAccountType": "Premium_LRS"
},
"datadisk": []
},
{
"name": "Greg-VM",
"disk": {
"id": "/subscriptions/24ba3e4c-45e3-4d55-8132-6731cf25547f/resourceGroups/GREG/providers/Microsoft.Compute/disks/Greg-VM_OsDisk_1_63ed471fef3e4f568314dfa56ebac5d2",
"storageAccountType": "Premium_LRS"
},
"datadisk": [
{
"name": "Data",
"createOption": "Attach",
"diskSizeGB": 10,
"managedDisk": {
"id": "/subscriptions/24ba3e4c-45e3-4d55-8132-6731cf25547f/resourceGroups/GREG/providers/Microsoft.Compute/disks/Data",
"storageAccountType": "Standard_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 0
},
{
"name": "Disk2",
"createOption": "Attach",
"diskSizeGB": 10,
"managedDisk": {
"id": "/subscriptions/24ba3e4c-45e3-4d55-8132-6731cf25547f/resourceGroups/GREG/providers/Microsoft.Compute/disks/Disk2",
"storageAccountType": "Standard_LRS"
},
"caching": "None",
"toBeDetached": false,
"lun": 1
}
]
}
]
How do I do that?
Thanks,
G
This should help you. It steps through the process.
If you have a scenario like this
you can use Add custom Column and type the follwing expression:
=Table.Columns([TableName], "ColumnName")
to get it as list:
Now you can left click on the Custom column and chose Extract Values....
Choose Custom and your delimiter | and hit OK
This way the data who was in your list will now be in the same row with the delimiter

How to interpret user search query (in Elasticsearch)

I would like to serve my visitors the best results possible when they use our search feature.
To achieve this I would like to interpret the search query.
For example a user searches for 'red beds for kids 120cm'
I would like to interpret it as following:
Category-Filter is "beds" AND "children"
Color-filter is red
Size-filter is 120cm
Are there ready to go tools for Elasticsearch?
Will I need NLP in front of Elasticsearch?
Elasticsearch is pretty powerful on its own and is very much capable of returning the most relevant results to full-text search queries, provided that data is indexed and queried adequately.
Under the hood it always performs text analysis for full-text searches (for fields of type text). A text analyzer consists of a character filter, tokenizer and a token filter.
For instance, synonym token filter can replace kids with children in the user query.
Above that search queries on modern websites are often facilitated via category selectors in the UI, which can easily be implemented with querying keyword fields of Elasticsearch.
It might be enough to model your data correctly and tune its indexing to implement the search you need - and if that is not enough, you can always add some extra layer of NLP-like logic on the client side, like #2ps suggested.
Now let me show a toy example of what you can achieve with a synonym token filter and copy_to feature.
Let's define the mapping
Let's pretend that our products are characterized by the following properties: Category, Color, and Size.LengthCM.
The mapping will look something like:
PUT /my_index
{
"mappings": {
"properties": {
"Category": {
"type": "keyword",
"copy_to": "DescriptionAuto"
},
"Color": {
"type": "keyword",
"copy_to": "DescriptionAuto"
},
"Size": {
"properties": {
"LengthCM": {
"type": "integer",
"copy_to": "DescriptionAuto"
}
}
},
"DescriptionAuto": {
"type": "text",
"analyzer": "MySynonymAnalyzer"
}
}
},
"settings": {
"index": {
"analysis": {
"analyzer": {
"MySynonymAnalyzer": {
"tokenizer": "standard",
"filter": [
"MySynonymFilter"
]
}
},
"filter": {
"MySynonymFilter": {
"type": "synonym",
"lenient": true,
"synonyms": [
"kid, kids => children"
]
}
}
}
}
}
}
Notice that we selected type keyword for the fields Category and Color.
Now, what about these copy_to and synonym?
What will copy_to do?
Every time we send an object for indexing into our index, value of the keyword field Category will be copied to a full-text field DescritpionAuto. This is what copy_to does.
What will synonym do?
To enable synonym we need to define a custom analyzer, see MySynonymAnalyzer which we defined under "settings" above.
Roughly, it will replace every token that matches something on the left of => with the token on the right.
How will the documents look like?
Let's insert a few example documents:
POST /my_index/_doc
{
"Category": [
"beds",
"adult"
],
"Color": "red",
"Size": {
"LengthCM": 150
}
}
POST /my_index/_doc
{
"Category": [
"beds",
"children"
],
"Color": "red",
"Size": {
"LengthCM": 120
}
}
POST /my_index/_doc
{
"Category": [
"couches",
"adult",
"family"
],
"Color": "blue",
"Size": {
"LengthCM": 200
}
}
POST /my_index/_doc
{
"Category": [
"couches",
"adult",
"family"
],
"Color": "red",
"Size": {
"LengthCM": 200
}
}
As you can see, DescriptionAuto is not present in the original documents - though due to copy_to we will be able to query it.
Let's see how.
Performing the search!
Now we can try out our index with a simple query_string query:
POST /my_index/_doc/_search
{
"query": {
"query_string": {
"query": "red beds for kids 120cm",
"default_field": "DescriptionAuto"
}
}
}
The results will look something like the following:
"hits": {
...
"max_score": 2.3611186,
"hits": [
{
...
"_score": 2.3611186,
"_source": {
"Category": [
"beds",
"children"
],
"Color": "red",
"Size": {
"LengthCM": 120
}
}
},
{
...
"_score": 1.0998137,
"_source": {
"Category": [
"beds",
"adult"
],
"Color": "red",
"Size": {
"LengthCM": 150
}
}
},
{
...
"_score": 0.34116736,
"_source": {
"Category": [
"couches",
"adult",
"family"
],
"Color": "red",
"Size": {
"LengthCM": 200
}
}
}
]
}
The document with categories beds and children and color red is on top. And its relevance score is twice bigger than of its follow-up!
How can I check how Elasticsearch interpreted the user's query?
It is easy to do via analyze API:
POST /my_index/_analyze
{
"text": "red bed for kids 120cm",
"analyzer": "MySynonymAnalyzer"
}
{
"tokens": [
{
"token": "red",
"start_offset": 0,
"end_offset": 3,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "bed",
"start_offset": 4,
"end_offset": 7,
"type": "<ALPHANUM>",
"position": 1
},
{
"token": "for",
"start_offset": 8,
"end_offset": 11,
"type": "<ALPHANUM>",
"position": 2
},
{
"token": "children",
"start_offset": 12,
"end_offset": 16,
"type": "SYNONYM",
"position": 3
},
{
"token": "120cm",
"start_offset": 17,
"end_offset": 22,
"type": "<ALPHANUM>",
"position": 4
}
]
}
As you can see, there is no token kids, but there is token children.
On a side note, in this example Elasticsearch wasn't able, though, to parse the size of the bed: token 120cm didn't match to anything, since all sizes are integers, like 120, 150, etc. Another layer of tweaking will be needed to extract 120 from 120cm token.
I hope this gives an idea of what can be achieved with Elasticsearch's built-in text analysis capabilities!