json proper encoding and decoding not working with flask, python - python-2.7

I'm taking a text-input from html being json object and trying to work upon that. But when I'm trying the following code, I'm getting error/(page not rendering) in encoding and decoding JSON.
#app.route('/', methods=['POST'])
def my_form_post():
text = request.form['text']
#getting text-input as text = {'a':'1','b':'2'}
json_input = json.dumps(text)
ordered_json = json.loads(text, object_pairs_hook=ordereddict.OrderedDict)
print ordered_json
processed_text = htmlConvertor(ordered_json)
#rep(jso)
return render_template("my-form.html",processed_text=processed_text)
But when I'm tying to do so with a local JSON variable as jso everything working fine. The same input when I provide with html-input, it's giving an error and I can;t even see the error except displaying Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
#app.route('/', methods=['POST'])
def my_form_post():
jso = '''{
"glossary": {
"title": "example glossary",
"GlossDiv": {
"title": "S",
"GlossList": {
"GlossEntry": {
"ID": "SGML",
"SortAs": "SGML",
"GlossTerm": "Standard Generalized Markup Language",
"Acronym": "SGML",
"Abbrev": "ISO 8879:1986",
"GlossDef": {
"para": "A meta-markup language, used to create markup languages such as DocBook.",
"GlossSeeAlso": ["GML", "XML"]
},
"GlossSee": "markup"
}
}
}
}
}'''
json_input = json.dumps(jso)
ordered_json = json.loads(jso, object_pairs_hook=ordereddict.OrderedDict)
print ordered_json
processed_text = htmlConvertor(ordered_json)
#rep(jso)
return render_template("my-form.html",processed_text=processed_text)
UPDATE:
Everything working fine now, but for the integers, it isn't working.
For eg:
{"name":"yo","price":"250"}
works perfectly but
{"name":"yo","price":250}
ain't.
What's the solution for that? Any specific answer or I would have to check for integer in python and then convert it to string before applying any JSON related methods and functioning.

Not sure if this is your problem, but {'a':'1','b':'2'} is not a valid JSON object because of the single quotes:
>>> json.loads("{'a':'1','b':'2'}")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
obj, end = self._scanner.iterscan(s, **kw).next()
File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
rval, next_pos = action(m, context)
File "/usr/lib64/python2.6/json/decoder.py", line 171, in JSONObject
raise ValueError(errmsg("Expecting property name", s, end))
ValueError: Expecting property name: line 1 column 1 (char 1)
If instead you use double quotes everything works fine:
>>> json.loads("{\"a\":\"1\",\"b\":\"2\"}")
{u'a': u'1', u'b': u'2'}
Also note that to get stack traces intead of code 500 errors when there is an exception you have to start your flask server as follows:
app.run(debug = True)

Related

Getting error of PipelineActivity must have one and only one member when using Boto3 and Create_Pipeline

I have a python program that is using boto3 to create an IoT Analytics path. My program was able to successfully create the channel and the datastore but fails when I try to connect the two through the create pipeline function. My code is as follows:
dactivity = [{
"channel": {
"channelName": channel["channelName"],
"name": IoTAConfig["channelName"],
"next" : IoTAConfig["datastoreName"]
},
"datastore": {
"datastoreName": ds["datastoreName"],
"name": IoTAConfig["datastoreName"]
}
}]
pipeline = iota.create_pipeline(
pipelineActivities = dactivity,
pipelineName = IoTAConfig["pipelineName"]
)
The error code is as follows:
Traceback (most recent call last):
File "createFullGG.py", line 478, in <module>
createIoTA()
File "createFullGG.py", line 268, in createIoTA
pipelineName = IoTAConfig["pipelineName"]
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the UpdatePipeline operation: PipelineActivity must have one and only one member
According to the documentation pipeline activities can contain from 1 to 25 entries as long as they are in an array of 1 object. I have no idea why this continues to fail. Any help is appreciated.
The public documentation looks a little confusing because of the way that optional elements are represented, the good news is that this is an easy fix.
A corrected version of what you are trying would be written as;
dactivity=[
{
"channel": {
"channelName": channel["channelName"],
"name": IoTAConfig["channelName"],
"next" : IoTAConfig["datastoreName"]
}
},
{
"datastore": {
"datastoreName": ds["datastoreName"],
"name": IoTAConfig["datastoreName"]
}
}
]
response = client.create_pipeline(
pipelineActivities = dactivity,
pipelineName = IoTAConfig["pipelineName"]
)
So it's an array of activities that you are providing, like [ {A1},{A2} ] if that makes sense?
Does that help?

json.loads() giving "ValueError: Expecting property name:"

I'm trying to convert sttr data type to dict in python using json.loads() function.
But I'm getting the error as:
File "/usr/local/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/l`enter code here`ocal/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python2.7/json/decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1)
In comments you gave the following example of the text you are trying to load as JSON:
[
{
'entity_class': 'Hardware Entities (AHV)',
'status': 'recommended',
'uuid': 'de74178a-cbc7-4c69-ae2f-9e7042bf8e98',
'zprotobuf': 'eNolzD1LAzEYB/DFScHBRXAKoUMr5khiLi/dCgpdCg7qIiLPJU/KQS4nyeEL6ne3p+vv/3K8CGiUMBaY77xhymvHAGVkDg1XsosWnT1bcI42tqZlwbWOqWAis5oD08J3VinOOwMXp',
'version': '2.4-1542668003',
'dependencies': '[{"entity_class": "Dell Update Manager", "version": "1.8-0.112", "exact": "false", "entity_model": "PT Agent on AHV (el6)"}]',
'entity_uuid': '00e8f575-d959-4d7f-860a-61cb84400b7a',
'order': 4,
}
]
JSON should use double quotes, not single quotes. The above looks like native python data struct syntax. Here is the above dumped from native python to JSON.
import json
native_python_data = [
{
'entity_class': 'Hardware Entities (AHV)',
'status': 'recommended',
'uuid': 'de74178a-cbc7-4c69-ae2f-9e7042bf8e98',
'zprotobuf': 'eNolzD1LAzEYB/DFScHBRXAKoUMr5khiLi/dCgpdCg7qIiLPJU/KQS4nyeEL6ne3p+vv/3K8CGiUMBaY77xhymvHAGVkDg1XsosWnT1bcI42tqZlwbWOqWAis5oD08J3VinOOwMXp',
'version': '2.4-1542668003',
'dependencies': [
{
"entity_class": "Dell Update Manager",
"version": "1.8-0.112",
"exact": "false",
"entity_model": "PT Agent on AHV (el6)"
}
],
'entity_uuid': '00e8f575-d959-4d7f-860a-61cb84400b7a',
'order': 4,
}
]
json_string_with_escaping = json.dumps(js, indent=4)
print json_string_with_escaping

In Python, Unable to retrieve "dictionary inside a list" from stdout

I am executing a curl command and the output is in json format like this, which i am redirecting to a outputfile
[{ "createdDateTime": "2015-03-24T12:06:45.403+0000", "Id": "123", "ToDo": "VIEW"}, { "createdDateTime": "2015-03-25T06:30:31.624+0000", "Id": "456", "ToDo": "VIEW"}]
I am trying the below code to access the Id information from this data.
I thought it is a dictionary inside a list. But python processes it in a different way. please provide your inputs.
import subprocess
import shlex
cmd="curl \-X GET 'xxx'"
shlex.split(cmd)
p1=subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE)
stdout, stderr = p1.communicate()
print stdout
jsonFile = open('jsonFile.txt', 'w')
jsonFile.writelines(str(stdout))
jsonFile.close()
jsonFile = open('jsonFile.txt', 'r')
jsonout = [jsonFile.readlines()]
Ids = [d['Id'] for d in jsonout]
print Ids
Output:
[['[{ "createdDateTime": "2015-03-24T12:06:45.403+0000", "Id": "123", "ToDo": "VIEW"}, { "createdDateTime": "2015-03-25T06:30:31.624+0000", "Id": "456", "ToDo": "VIEW"}]']]
Traceback (most recent call last):
File "file.py", line 16, in
Ids = [d['Id'] for d in jsonout]
TypeError: list indices must be integers, not str
Python has an comprehensive json module to deal with tasks like that.
import json
with open ('jsonFile.txt', 'r') as f:
jsonout = json.load(f)
Your problem is that you're not using a json parser library. Try this instead:
import json
jsonout=json.loads(open('jsonFile.txt', 'r').read())
Ids = [d['Id'] for d in jsonout]
print Ids
For me, it prints [u'123', u'456']
And oh, I should add that unlike I do, you should consider also closing the file.

couchbase python sdk ascii exception

First of all, this is the exception
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\scrapy\middleware.py", line 62, in _process_chain
return process_chain(self.methods[methodname], obj, *args)
File "C:\Python27\lib\site-packages\scrapy\utils\defer.py", line 65, in process_chain
d.callback(input)
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 383, in callback
self._startRunCallbacks(result)
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 491, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 578, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\ScrapyProjects\General_Spider_code_version_4\General_Spider_code_version_4\pipelines.py", line 14, in process_item
connection.set(fileName, dict(item)) #write the item to the couchbase database
File "C:\Python27\lib\site-packages\couchbase-1.2.5-py2.7-win-amd64.egg\couchbase\connection.py", line 331, in set
persist_to, replicate_to)
File "C:\Python27\lib\site-packages\couchbase-1.2.5-py2.7-win-amd64.egg\couchbase\_bootstrap.py", line 99, in _json_encode_wrapper
return json.dumps(*args, ensure_ascii=False, separators=(',', ':'))
File "C:\Python27\lib\json\__init__.py", line 250, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "C:\Python27\lib\json\encoder.py", line 210, in encode
return ''.join(chunks)
couchbase.exceptions.ValueFormatError: <Couldn't encode value, inner_cause='ascii' codec can't decode byte 0xe2 in position 5: ordinal not in range(128), C Source=(src\convert.c,131), OBJ={'bathrooms': 1.0, 'furnished': 'No', 'ad_title': 'Large Studio For Rent in IMPZ just 45K/4chqs(KK)', 'agent_fees': -1, 'size': 550.0, 'category': 'Apartment', 'company_rera_number': '12913', 'agent_company': 'AL ANAS REAL ESTATE BROKER', 'ded_licence_number': '700590', 'source': 'dubizzleproperty', 'location': 'UAE \xe2\x80\xaa>\xe2\x80\xaa Dubai \xe2\x80\xaa>\xe2\x80\xaa IMPZ International Media Production Zone ; 3.1 km from Meadows Town Centre \xc2\xa0', 'image_links': [u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/73ff34e2a38c7b104401c9e5c54b03628971053f/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/24ec831f6b4afb47fecc1c3e0991cf3090c90c24/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/77fee11394090aaea2d668cfe2754b92d6e36264/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/5d4113319ccbabcdd65b0ffe7302da59b374b5fe/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/8070689f309759d5860e97aa35d3f0eac425dc1d/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/8e86702847e69d485d147629dd2e48e1ad831e63/main.jpeg'], 'latitude': -1, 'description': 'Central A/C & Heating , Balcony , Shared Pool , Built in Wardrobes , Walk-in Closet , Shared Gym , Security , Built in Kitchen Appliances', 'bedrooms': 'Studio', 'rent_is_paid': 'Quarterly', 'action': 'Rent', 'link': 'http://dubai.dubizzle.com/property-for-rent/residential/apartmentflat/2015/2/18/large-studio-for-rent-in-impz-just-45k4chq-2/?back=ZHViYWkuZHViaXp6bGUuY29tL3Byb3BlcnR5LWZvci1yZW50L3Jlc2lkZW50aWFsL2FwYXJ0bWVudGZsYXQv&pos=1', 'longitude': -1, 'property_reference': '', 'yearly_cost': 45000.0, 'agent_mobile': -1, 'posting_date': '2015-02-19'}>
I am trying to store a dictionary on a couchbase. i am using this code
connection.set(fileName, dict(item))
to transfer the item to dictionary. as you see from the error message. i have a unicode values, which is according to python sdk couchbase is fine, could you help me please?
Your values are not unicode. Keep in mind that a str object containing valid unicode escape sequences does not automatically make it "Unicode" in Python parlance. You need to ensure the strings are properly Unicode.
This seems to work with the normal json.dumps() function (without any arguments); whereas the python client passes (by default) the ensure_ascii=False parameter to decrease the data size (JSON itself can be in UTF-8 encoding, and is not limited to ASCII).
Thus, a workaround may be to set your own encoding function for JSON which does not pass the ensure_ascii parameter; like so:
import json
import couchbase
couchbase.set_json_converters(json.dumps, json.loads)
Though this workaround is not recommended as it may inflate your document size slightly.

attribute error: list object has not attribute lstrip in sending an email with attachment

i am attaching a file from a particular path c:\important\log.txt
sender = 'poojagupta4112#gmail.com'
receiver = ['shubh4112#gmail.com']
message = """From: From Pooja Gupta <poojagupta4112#gmail.com>
To: To Shubha Goel <shubh4112#gmail.com>
Subject: SMTP e-mail test
This is a test e-mail message.
"""
file_name = 'C:\important\log.txt'
msg=MIMEMultipart()
msg['From'] = sender
msg['To'] = receiver
msg['Subject'] = message
msg['Date'] = email.Utils.formatdate(localtime=True)
# build the attachment
att = MIMEBase('application', 'base64')
att.set_payload(open(file_name, 'rb').read())
email.Encoders.encode_base64(att)
att.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(file_name))
msg.attach(att)
print 'successfully built attachment'
try:
session = smtplib.SMTP('smtp.gmail.com',587)
print 'Starting..'
session.ehlo()
print 'ehlo executed..'
session.starttls()
print 'starttls done'
session.login(sender,'snxzoumwhpybzvmo')
print 'logged in'
session.sendmail(sender,receiver,msg.as_string())
print 'sendmail executed..now quitting'
session.close()
except smtplib.SMTPRecipientsRefused:
print 'Recipient refused'
except smtplib.SMTPAuthenticationError:
print 'Auth error'
except smtplib.SMTPSenderRefused:
print 'Sender refused'
except smtplib.SMTPException:
print('Error')
It keeps on giving me the same error of Attribute error list object has no attribute lstrip
the following is the error, stack trace :
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
execfile('C:\important\secret_file.pyw')
File "C:\important\secret_file.pyw", line 45, in <module>
session.sendmail(sender,receiver,msg.as_string())
File "C:\Python27\lib\email\message.py", line 137, in as_string
g.flatten(self, unixfrom=unixfrom)
File "C:\Python27\lib\email\generator.py", line 83, in flatten
self._write(msg)
File "C:\Python27\lib\email\generator.py", line 115, in _write
self._write_headers(msg)
File "C:\Python27\lib\email\generator.py", line 164, in _write_headers
v, maxlinelen=self._maxheaderlen, header_name=h).encode()
File "C:\Python27\lib\email\header.py", line 410, in encode
value = self._encode_chunks(newchunks, maxlinelen)
File "C:\Python27\lib\email\header.py", line 370, in _encode_chunks
_max_append(chunks, s, maxlinelen, extra)
File "C:\Python27\lib\email\quoprimime.py", line 97, in _max_append
L.append(s.lstrip())
AttributeError: 'list' object has no attribute 'lstrip'
Please Help.
it was a small error. receiver parameter was list type. either it should be list converted to string using join method or if it is a single recipient, then pass it as a string only
receiver = ['shubh4112#gmail.com']
This is a list but msg['To'] is expecting a string and hence the error.
You can use ','.join(receiver) and that should solve your problem.
This appears to be a issue from smtplib. The documentation clearly says that it accepts a list
The arguments are:
- from_addr : The address sending this mail.
- **to_addrs : A list of addresses to send this mail to. A bare
string will be treated as a list with 1 address.**
- msg : The message to send.
Usage from documentation:
"Example:
>>> import smtplib
>>> s=smtplib.SMTP("localhost")
**>>> tolist=
["one#one.org","two#two.org","three#three.org","four#four.org"]**
>>> msg = '''\\
... From: Me#my.org
... Subject: testin'...
...
... This is a test '''
>>> s.sendmail("me#my.org",tolist,msg)"
Also as said in the documentation if recipients are passed as string, mail is being sent to first mailid only.
So actually the problem is that SMTP.sendmail and email.MIMEText need two different things.
email.MIMEText sets up the "To:" header for the body of the e-mail. It is
ONLY used for displaying a result to the human being at the other end, and
like all e-mail headers, must be a single string. (Note that it does not
actually have to have anything to do with the people who actually receive
the message.)
SMTP.sendmail, on the other hand, sets up the "envelope" of the message for
the SMTP protocol. It needs a Python list of strings, each of which has a
single address.
So, what you need to do is COMBINE the two replies you received. Set
msg['To'] to a single string, but pass the raw list to sendmail:
emails = ['a.com','b.com', 'c.com']**
**msg['To'] = ', '.join( emails )
....
s.sendmail( msg['From'], emails, msg.as_string() )****
I have the same problem, my solution:
msg['To'] = receiver
receive must be string like 'aa#bb.com,bb#cc.com', not list