I'm trying to encode a Python array into json using simplejson.dumps:
In [30]: s1 = ['test', '<script>']
In [31]: simplejson.dumps(s1)
Out[31]: '["test", "<script>"]'
Works fine.
But I want to escape the strings first (using escapejs from Django) before calling simplejson.dumps:
In [35]: s_esc
Out[35]: [u'test', u'\\u003Cscript\\u003E']
In [36]: print simplejson.dumps(s_esc)
["test", "\\u003Cscript\\u003E"]
My problem is: I want the escaped string to be: ["test", "\u003Cscript\u003E"] instead of ["test", "\\u003Cscript\\u003E"]
I can use replace:
In [37]: print simplejson.dumps(s_esc).replace('\\\\', '\\')
["test", "\u003Cscript\u003E"]
But is this a good approach? I just want to escape the strings first before encoding them to json. So there will be no syntax errors when I use them in template.
Thanks. :)
simplejson 2.1.0 and later include a JSONEncoderForHTML encoder that does exactly what you want. To use it in your example:
>>> s1 = ['test', '<script>']
>>> simplejson.dumps(s1, cls=simplejson.encoder.JSONEncoderForHTML)
'["test", "\\u003cscript\\u003e"]'
I ran into this recently where I didn't have control over the code that was generating the data structures, so I couldn't escape the strings as they were being assembled. JSONEncoderForHTML solved the problem neatly at the point of output.
Of course, you'll need to have simplejson 2.1.0 or later. (Django used to come with an older version, and Django 1.5 deprecated django.utils.simplejson entirely.) If you can't upgrade for some reason, the JSONEncoderForHTML code is relatively small and could probably be pulled into earlier code or used with Python 2.6+'s json package -- though I haven't tried this myself
You're doing the operations in the wrong order. You should dump your data to a JSON string, then escape that string. You can do the escaping with the addslashes Django filter.
Related
I'm testing with AjaxResponse with my request factory on a datetime. The problem is the string that Django gives is like this: 2020-08-27T22:46:07.354Z
But when I have datetime object, and I use the isoformat() method, I don't get the same string: 2020-08-27T22:46:07.354734+00:00
How am I going to be able to assert? I'm looking to assert by comparing the JSON with my own Python list (The list is what I can customize).
Thanks to Bast in Discord.py; the datetime conversion is not even iso8601 entirely... it's defined by ECMA-262 which is for JS:
from django.core.serializers.json import DjangoJSONEncoder
DjangoJSONEncoder().default(datetime_obj)
>>> '2020-08-28T03:41:59.194Z'
whereas using the .isoformat() method from standard lib would return 6 rather than 3 decimal points and use +00:00 rather than Z.
I have a api url.
http://10.236.220.130:5056/api/v1/viewentities?where={"EntityId":"srv-1499839282322"}
my search query variable/key is where={"EntityId":"srv-1499839282322"}
this is my code in python 2.7
tgt="srv-1499839282322"
baseurl="http://10.236.220.130:5056/api/v1/viewentities?"`enter code here`
values={"where": '{"EntityId": "srv-1499839282322"}'}
(if i hardcode this srv stuff above it works)
encode=urllib.urlencode(values)
url=baseurl + encode
print encode
Note the url which am getting is working but my condition is, i want that srv stuff to be a variable that will come as one of my function parameter.
================
I tried this these
tgt="srv-1499839282322"
baseurl="http://10.236.220.130:5056/api/v1/viewentities?"
value={"where": {}}
value["where"]["EntityId"]=tgt
encode=urllib.urlencode(values)
url=baseurl + encode
print encode
But this encode url is not working...what i found is the
the urlparse.urlencode takes the query as a dictionary and my value as itself is a dictionary. (nested)
if print values(working) and value(not working)
i find the single quotes are the one missing from value.
{'where': {'EntityId': 'srv-1499839282322'}}-----> not working
{'where': '{"EntityId": "srv-1499839282322"}'} ----> working
I have the following XML file :
<user-login-permission>true</user-login-permission>
<total-matched-record-number>15000</total-matched-record-number>
<total-returned-record-number>15000</total-returned-record-number>
<active-user-records>
<active-user-record>
<active-user-name>username</active-user-name>
<authentication-realm>realm</authentication-realm>
<user-roles>Role</user-roles>
<user-sign-in-time>date</user-sign-in-time>
<events>0</events>
<agent-type>text</agent-type>
<login-node>node</login-node>
</active-user-record>
There are many records
I'm trying to get values from tags and save them in a different text file using the following code :
soup = BeautifulSoup(open("path/to/xmlfile"), features="xml")
with open('path/to/outputfile', 'a') as f:
for i in range(len(soup.findall('active-user-name'))):
f.write ('%s\t%s\t%s\t%s\n' % (soup.findall('active-user-name')[i].text, soup.findall('authentication-realm')[i].text, soup.findall('user-roles')[i].text, soup.findall('login-node')[i].text))
I get the error TypeError : 'NoneType' object not callable Python with BeautifulSoup XML for line : for i in range(len(soup.findall('active-user-name'))):
Any idea what could be causing this?
Thanks!
There are a number of issues that need to be addressed with this, the first is that the XML file you provided is not valid XML - a root element is required.
Try something like this as the XML:
<root>
<user-login-permission>true</user-login-permission>
<total-matched-record-number>15000</total-matched-record-number>
<total-returned-record-number>15000</total-returned-record-number>
<active-user-records>
<active-user-record>
<active-user-name>username</active-user-name>
<authentication-realm>realm</authentication-realm>
<user-roles>Role</user-roles>
<user-sign-in-time>date</user-sign-in-time>
<events>0</events>
<agent-type>text</agent-type>
<login-node>node</login-node>
</active-user-record>
</active-user-records>
</root>
Now onto the python. First off there is not a findall method, it's either findAll or find_all. findAll and find_all are equivalent, as documented here
Next up I would suggest altering your code so you aren't making use of the find_all method quite so often - using find instead will improve the efficiency, especially for large XML files. Additionally the code below is easier to read and debug:
from bs4 import BeautifulSoup
xml_file = open('./path_to_file.xml', 'r')
soup = BeautifulSoup(xml_file, "xml")
with open('./path_to_output_f.txt', 'a') as f:
for s in soup.findAll('active-user-record'):
username = s.find('active-user-name').text
auth = s.find('authentication-realm').text
role = s.find('user-roles').text
node = s.find('login-node').text
f.write("{}\t{}\t{}\t{}\n".format(username, auth, role, node))
Hope this helps. Let me know if you require any further assistance!
The solution is simple: don't use findall method - use find_all.
Why? Because there is no findall method at all, there are findAll and find_all, which are equivalent. See docs for more information.
Though, I agree, error message is confusing.
Hope that helps.
The fix for my version of this problem is to coerce the BeautifulSoup instance into a type string. You do this following:
https://groups.google.com/forum/#!topic/comp.lang.python/ymrea29fMFI
you use the following pythonic:
From python manual
str( [object])
Return a string containing a nicely printable representation of an
object. For strings, this returns the string itself. The difference
with repr(object) is that str(object) does not always attempt to
return a string that is acceptable to eval(); its goal is to return a
printable string. If no argument is given, returns the empty string,
What i do...
___I have an upload form from where i upload .zip files with pictures. And everytime when there is a file title with some non-ascii character äüõ i get a unicode decode error.
title = ' '.join([filename[:filename.rfind('.')], str(count)])
Error:
This line generates the title of the picture , and that is exactly the line that gives me error: 'utf8' codec can't decode byte 0x82 in position 2: invalid start byte. You passed in 'cr\x82ations' (<type 'str'>)
What i tried to do:
I tried to .decode('utf-8') it too. But get the same result everytime no matter what i try.
I read about changing default djangos ascii to utf-8 in site.py , but am not sure it will help , and pretty sure that i don't want to do it.
ANy help is appreciated.
Django has some useful utility methods which you can use.
See: https://docs.djangoproject.com/en/dev/ref/unicode/#conversion-functions
I imagine the code might look something like this:
from django.utils.encoding import smart_str
title = ' '.join([smart_str(filename[:filename.rfind('.')]), str(count)])
I also believe firstly using .decode() is the right option, however, the code page ('utf-8')) you used might incorrect. Can you have a try '1252' or some others? Here are some standard encoding you might interest [Link]http://docs.python.org/library/codecs.html?highlight=arabic
The reason this fails, is because you try to join with a normal str object:
Instead of
' '.join(..)
use:
u' '.join(..)
Or make your life easier using:
from __future__ import unicode_literals
I have a json field in my database which is like
jsonfield = {'username':'chingo','reputation':'5'}
how can i write a query so that i can find if a user name exists. something like
username = 'chingo'
query = User.objects.get(jsonfield['username']=username)
I know the above query is a wrong but I wanted to know if there is a way to access it?
If you are using the django-jsonfield package, then this is simple. Say you have a model like this:
from jsonfield import JSONField
class User(models.Model):
jsonfield = JSONField()
Then to search for records with a specific username, you can just do this:
User.objects.get(jsonfield__contains={'username':username})
Since Django 1.9, you have been able to use PostgreSQL's native JSONField. This makes search JSON very simple. In your example, this query would work:
User.objects.get(jsonfield__username='chingo')
If you have an older version of Django, or you are using the Django JSONField library for compatibility with MySQL or something similar, you can still perform your query.
In the latter situation, jsonfield will be stored as a text field and mapped to a dict when brought into Django. In the database, your data will be stored like this
{"username":"chingo","reputation":"5"}
Therefore, you can simply search the text. Your query in this siutation would be:
User.objects.get(jsonfield__contains='"username":"chingo"')
2019: As #freethebees points out it's now as simple as:
User.objects.get(jsonfield__username='chingo')
But as the doc examples mention you can query deeply, and if the json is an array you can use an integer to index it:
https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#querying-jsonfield
>>> Dog.objects.create(name='Rufus', data={
... 'breed': 'labrador',
... 'owner': {
... 'name': 'Bob',
... 'other_pets': [{
... 'name': 'Fishy',
... }],
... },
... })
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': None})
>>> Dog.objects.filter(data__breed='collie')
<QuerySet [<Dog: Meg>]>
>>> Dog.objects.filter(data__owner__name='Bob')
<QuerySet [<Dog: Rufus>]>
>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
<QuerySet [<Dog: Rufus>]>
Although this is for postgres, I believe it works the same in other DBs like MySQL
Postgres: https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#querying-jsonfield
MySQL: https://django-mysql.readthedocs.io/en/latest/model_fields/json_field.html#querying-jsonfield
This usage is somewhat anti-pattern. Also, its implementation is not going to have regular performance, and perhaps is error-prone.
Normally don't use jsonfield when you need to look up through fields. Use the way the RDBMS provides or MongoDB(which internally operates on faster BSON), as Daniel pointed out.
Due to the deterministic of JSON format,
you could achieve it by using contains (regex has issue when dealing w/ multiple '\' and even slower), I don't think it's good to use username in this way, so use name instead:
def make_cond(name, value):
from django.utils import simplejson
cond = simplejson.dumps({name:value})[1:-1] # remove '{' and '}'
return ' ' + cond # avoid '\"'
User.objects.get(jsonfield__contains=make_cond(name, value))
It works as long as
the jsonfield using the same dump utility (the simplejson here)
name and value are not too special (I don't know any egde-case so far, maybe someone could point it out)
your jsonfield data is not corrupt (unlikely though)
Actually I'm working on a editable jsonfield and thinking about whether to support such operations. The negative proof is as said above, it feels like some black-magic, well.
If you use PostgreSQL you can use raw sql to solve problem.
username = 'chingo'
SQL_QUERY = "SELECT true FROM you_table WHERE jsonfield::json->>'username' = '%s'"
User.objects.extra(where=[SQL_EXCLUDE % username]).get()
where you_table is name of table in your database.
Any methods when you work with JSON like with plain text - looking like very bad way.
So, also I think that you need a better schema of database.
Here is the way I have found out that will solve your problem:
search_filter = '"username":{0}'.format(username)
query = User.objects.get(jsonfield__contains=search_filter)
Hope this helps.
You can't do that. Use normal database fields for structured data, not JSON blobs.
If you need to search on JSON data, consider using a noSQL database like MongoDB.