Separate nested python dictionary into 2 separate dictionaries - python-2.7

So I have a nested dictionary in python as follows:
{'name': 'Waffles',
'subCategories': [{'menu': [{'name': 'Fig & Honey with Fresh Cream','price': 120},
{'name': 'Toffeed Banana', 'price': 110}],
'name': 'Sweet',
'description': 'Sweet and yummy'},
{'menu': [{'name': 'Mushroom Cheese Gratin','price': 175},
{'name': 'Pepper Chicken Waffle', 'price': 180}],
'name': 'Savoury'
'description' : 'Salty and yummy'}]
}
What I am looking at is to separate out the dict into 2 dicts as follows:
{'name': 'Waffles(Sweet)',
'menu': [{'name': 'Fig & Honey with Fresh Cream','price': 120},
{'name': 'Toffeed Banana', 'price': 110}],
'description' : 'Sweet and yummy'}
{'name': 'Waffles(Savoury)',
'menu': [{'name': 'Mushroom Cheese Gratin','price': 175},
{'name': 'Pepper Chicken Waffle', 'price': 180}],
'description': 'Salty and yummy'}
Note that the name key is a combination of the same key in the outer and inner dicts
What would be the best way to tackle this ?

Hoping the code is self explantory!!
import pprint
d = {'name': 'Waffles',
'subCategories': [
{'menu': [{'name': 'Fig & Honey with Fresh Cream','price': 120},
{'name': 'Toffeed Banana', 'price': 110}],
'name': 'Sweet',
'description': 'Sweet and yummy'},
{'menu': [{'name': 'Mushroom Cheese Gratin','price': 175},
{'name': 'Pepper Chicken Waffle', 'price': 180}],
'name': 'Savoury',
'description' : 'Salty and yummy'}]
}
menu = []
for category in d.get('subCategories', []):
category['name'] = "{}({})".format(d['name'], category.get('name', ''))
menu.append(category)
pprint.pprint(menu)
and the sample output
[{'description': 'Sweet and yummy',
'menu': [{'name': 'Fig & Honey with Fresh Cream', 'price': 120},
{'name': 'Toffeed Banana', 'price': 110}],
'name': 'Waffles(Sweet)'},
{'description': 'Salty and yummy',
'menu': [{'name': 'Mushroom Cheese Gratin', 'price': 175},
{'name': 'Pepper Chicken Waffle', 'price': 180}],
'name': 'Waffles(Savoury)'}]

This is the code to do what you ask. Note that it can be optimized, but was left as is for readability.
dict = #your dictionary
# list to hold your new dictionaries (if there are more than one)
dict_list = []
# loop to extract
for item in dict['subCategories']:
d = {}
id = item['name']
# this could be made more compact
d.update({'name':'Waffles('+id+')'})
d.update({'menu':item['menu']})
d.update({'description':item['description']})
dict_list.append(d)
# print new dictionaries
for i in dict_list:
print(i)
Note that the program could be made more robust to handle arbitrary yamls (or json, I don't know what format this dictionary came from) if needed. Here the keys for extraction are hard coded.
Best of luck!
ps: there was formatting error in your starting dictionary

Related

How Specify Unique Column for QuestDB Import?

I am importing data to QuestDB with the HTTP method in Python.
The table (test_data) has the following properties:
'name': 'time', 'size': 8, 'type': 'TIMESTAMP'
'name': 'open', 'size': 8, 'type': 'DOUBLE'
'name': 'high', 'size': 8, 'type': 'DOUBLE'
'name': 'low', 'size': 8, 'type': 'DOUBLE'
'name': 'close', 'size': 8, 'type': 'DOUBLE'
'name': 'volume', 'size': 8, 'type': 'DOUBLE'
'name': 'ts', 'size': 8, 'type': 'TIMESTAMP'
Note: the 'time' column is the designated timestamp
The imported data is sourced from a pandas dataframe. The dataframe has the same headers and the 'time' column is the index and the 'ts' column is the timestamp from when the data was acquired. The code shown below regarding the import function.
def to_csv_str(table):
output = io.StringIO()
csv.writer(output, dialect="excel").writerows(table)
return output.getvalue().encode("utf-8")
def write_to_table(df, table="test_data"):
table_name = table
table = [[df.index.name] + df.columns.tolist()] +df.reset_index().values.tolist()
table_csv = to_csv_str(table)
schema = json.dumps([])
response = requests.post(
"http://localhost:9000/imp",
params={"fmt": "json"},
files={"schema": schema, "data": (table_name, table_csv)},
).json()
pprint.pprint(response)
The import executes successfully the first time. If I was to rerun the import for the same data (all values are the same except the 'ts' column for when the data was acquired), then one additional row will be appended with all of the same values but the 'ts' column. How can I have the 'time' column be defined in such a way that it is forced to be unique and any import with a row that has a duplicate 'time' value will be omitted?
Example screenshots for a 6 row import below:
Initial import with all rows successfull image 1
Reissued import with only 5 errors (expected 6) image 2
Table data from the web console
image 3

Get respectives values from Django annotate method

I have the following query:
result = data.values('collaborator').annotate(amount=Count('cc'))
top = result.order_by('-amount')[:3]
This one, get the collaborator field from data, data is a Django Queryset, i am trying to make like a GROUP BY query, and it's functional, but when i call the .values() method on the top variable, it's returning all the models instances as dicts into a queryset, i need the annotate method result as a list of dicts:
The following is the top variable content on shell:
<QuerySet [{'collaborator': '1092788966', 'amount': 20}, {'collaborator': '1083692812', 'amount': 20}, {'collaborator': '1083572767', 'amount': 20}]>
But when i make list(top.values()) i get the following result:
[{'name': 'Alyse Caffin', 'cc': '1043346592', 'location': 'Wu’an', 'gender': 'MASCULINO', 'voting_place': 'Corporación Educativa American School Barranquilla', 'table_number': '6', 'status': 'ESPERADO', 'amount': 1}, {'name': 'Barthel Hanlin', 'cc': '1043238706', 'location': 'General Santos', 'gender': 'MASCULINO', 'voting_place': 'Colegio San José – Compañía de Jesús Barranquilla', 'table_number': '10', 'status': 'PENDIENTE', 'amount': 1}, {'name': 'Harv Gertz', 'cc': '1043550513', 'location': 'Makueni', 'gender': 'FEMENINO', 'voting_place': 'Corporación Educativa American School Barranquilla', 'table_number': '7', 'status': 'ESPERADO', 'amount': 1}]
I just want the result to be like:
[{'collaborator': '1092788966', 'amount': 20}, {'collaborator': '1083692812', 'amount': 20}, {'collaborator': '1083572767', 'amount': 20}]
there is something wrong, maybe a typo (also it seems you do not show the full query... something like data=yourmodel.objects.filter... is missing before):
The output of list(top.values()) returns a completely different model's fields then what you post as top Queryset- are you sure you really did:
result = data.values('collaborator').annotate(amount=Count('cc'))
top = result.order_by('-amount')[:3]
list(top.values())
because it should deliver what you expect (provided that data is a Queryset)

What's the structure of Telegram's updates to webhook Flask app?

I'm trying to program a Telegram bot using webhook in a Flask app with telepot, in PythonAnywhere. And so, I want to know what's the structure of the updates comming from Telegram, so as to know what's there and how is it called, and use it in the bot, essentially.
I've tried to log the message it receives to the console (though I'm not sure where the console should be on PythonAnywhere), and also to write in a file in the same server, through python, but that's not working either.
#This that seemed easy didn't work either in the Flask web app
with open('log.txt', 'a') as f:
f.write('Is this working?')
It feels like I'm missing some easy information everyone takes for granted, but I can't figure out what that is.
There was indeed something I didn't notice. Posting in case it helps anyone.
On the web app section of PythonAnywhere there are three Log Files links where you can see the kinds of things that would apear on the console in a regular Python app.
Those links look like this:
username.eu.pythonanywhere.com.access.log
username.eu.pythonanywhere.com.error.log
username.eu.pythonanywhere.com.server.log #no .eu on the american PythonAnywhere
And server.log is where console print statements end up.
Also, regular messages from Telegram users look like this when they arrive to Flask:
{
'update_id': 123456789,
'message': {
'message_id': 42,
'from': {
'id': 42424242,
'is_bot': False,
'first_name': 'Joaquim',
'last_name': 'Pernil Rinoceronts',
'username': 'Juqim',
'language_code': 'ca'
},
'chat': {
'id': 42424242,
'first_name': 'Joaquim',
'last_name': 'Pernil Rinoceronts',
'username': 'Juqim',
'type': 'private'
},
'date': 1562247903,
'text': 'Patata'
}
}
Stickers have their info where 'text' would be:
'sticker': {
'width': 512,
'height': 512,
'emoji': '😒',
'set_name': 'Ruscamems',
'thumb': {
'file_id': 'AAQEABNgnrsaAAQkkp4QRiVF1rokAAIC',
'file_size': 4840,
'width': 128,
'height': 128
},
'file_id': 'CAADBAADBQADkvulAumgmwOAjdfYAg',
'file_size': 36612
}
Images have 'photo' instead, and they come in a collection of different sizes:
'photo':[
{
'file_id': 'AgADBAADVrAxG2wj8FCs-f6v7AGFUQvo-RkABFGq4cIH4_MaHXIFAAEC',
'file_size': 2101,
'width': 66,
'height': 90
},
{
#same but bigger (different id too)
},
... #I got 4 sizes.
]
I guess I'll post the callback too and we'll have most of the interesting stuff:
{
'update_id': 123456793,
'callback_query': {
'id': '424242424242424242',
'from': { #Who pressed the Button
'id': 42424242,
'is_bot': False,
'first_name': 'Joaquim',
'last_name': 'Pernil Rinoceronts',
'username': 'Juqim',
'language_code': 'ca'
},
'message': { #What message was the button in
'message_id': 123,
'from': {
'id': 434343434,
'is_bot': True,
'first_name': 'The Bot Name',
'username': 'name_bot'
},
'chat': {
'id': 42424242,
'first_name': 'Joaquim',
'last_name': 'Pernil Rinoceronts',
'username': 'Juqim',
'type': 'private'
},
'date': 1562252792,
'text': 'A viam si funciona això',
'reply_markup': { #Keyboard pressed
'inline_keyboard': [[{'text': 'Cliccami', 'callback_data': 'clicat'}]]
}
},
'chat_instance': '1234123412341234242',
'data': 'clicat' #Callback data (button pressed)
}
}

Django extract field form html

I have a html page that returns data in a form. The forms is a multiple choice checkbox field with "name=states1". When I run print(vars(self)) in my forms.py I see that the data is there ['MD', 'WV', 'WY'] (see below) however, when I try to extract it with print(self.data['states1']) I just get 'WY' and print(len(self.data['states1'])) returns 2.
{'instance': <AA: AAobject>, '_validate_unique': False, 'is_bound': True, 'data': <QueryDict: {'csrfmiddlewaretoken': ['AAAAA']: ['1'], 'states1': ['MD', 'WV', 'WY'], 'states': [''], 'price': ['2.00'], 'user': ['1', '1']}>, ...
However, when I run
import json
print(json.dumps(self.data, indent=2))
I get the following output
{
"csrfmiddlewaretoken": "AAAAA",
"roles": "1",
"states1": "WY",
"states": "",
"price": "2.00",
"user": "1"
}
The data is just gone. Why is my data being restricted to the last field, when I can clearly see it when looking at vars?
forms.py
class UserProfileChangeForm(forms.ModelForm):
states = forms.CharField(widget=USStateSelect(), initial='TX', required=False)
class Meta:
model = SkilledLaborer
fields = ['user','roles','travel_flag','price','states']
def clean_states(self):
print('states')
states = self.cleaned_data['states']
print(type(vars(self)))
print(vars(self).keys())
print(vars(self))
print(self.data.keys())
print(self.data['states1'])
print(len(self.data['states1']))
import json
print(json.dumps(self.data, indent=2))
try:
self.data['states1']
except:
pass
return states

Wrongly big numbers when use multiple of Sum, Count aggregations in annotate

I have these models:
User:
email = EmailField()
Payment:
user = ForeignKey(User)
sum = DecimalField()
GuestAccount:
user = ForeignKey(User)
guest = ForeignKey(User)
I want to get user emails, amount of money that came from every user
and number of its guests accounts.
My query:
User.objects.annotate(
money=Sum('payment__sum'),
guests_number=Count('guestaccount')
).values('email', 'money', 'guests_number')
But money and guests_number in the result of the query are bigger then they really are:
{'guests_number': 0, 'email': 'a#b.cd', 'money': None}
{'guests_number': 20, 'email': 'user1#mail.com', 'money': Decimal('6600.00')}
{'guests_number': 4, 'email': 'user1000#test.com', 'money': Decimal('2500.00')}
{'guests_number': 0, 'email': 'zzzz#bbbbb.com', 'money': None}
I noticed that I get correct data if I split the query into 2 separate queries:
User.objects.annotate(money=Sum('payment__sum')).values('email', 'money')
User.objects.annotate(guests_number=Count('guestaccount')).values('email', 'guests_number')
Correct result of 1st half:
{'email': 'a#b.cd', 'money': None}
{'email': 'user1#mail.com', 'money': Decimal('1650.00')}
{'email': 'user1000#test.com', 'money': Decimal('1250.00')}
{'email': 'zzzz#bbbbb.com', 'money': None}
Correct result of 2nd half:
{'email': 'a#b.cd', 'guests_number': 0}
{'email': 'user1#mail.com', 'guests_number': 4}
{'email': 'user1000#test.com', 'guests_number': 2}
{'email': 'zzzz#bbbbb.com', 'guests_number': 0}
Also I noticed that I can add distinct=True in Count aggregation:
User.objects.annotate(
money=Sum('payment__sum'),
guests_number=Count('guestaccount', distinct=True)
).values('email', 'money', 'guests_number')
It fixes guests_number:
{'guests_number': 0, 'email': 'a#b.cd', 'money': None}
{'guests_number': 4, 'email': 'user1#mail.com', 'money': Decimal('6600.00')}
{'guests_number': 2, 'email': 'user1000#test.com', 'money': Decimal('2500.00')}
{'guests_number': 0, 'email': 'zzzz#bbbbb.com', 'money': None}
Unfortunatly, there are no distinct parameter in Sum aggregation.
What is wrong with my query? How to fix these numbers getting bigger with every aggregation in annotate?
Raw SQL query investigation showed that the problem comes from multiple LEFT OUTER JOINs. So I ended up with raw SQL:
User.objects.extra(select={
"money": """
SELECT SUM("website_payment"."sum")
FROM "website_payment"
WHERE "website_user"."id" = "website_payment"."user_id"
""",
"guests_number": """
SELECT COUNT("guests_guestaccount"."id")
FROM "guests_guestaccount"
WHERE "website_user"."id" = "guests_guestaccount"."user_id"
""",
}
).values('email', 'money', 'guests_number')
But I need to annotate these fields into queried objects and extra don't do it.