I have a Django app that runs a tool and displays the results from the tool back to the user using a Django template. Sometimes Django does not display the results. It doesn't complain about anything, it just doesn't display the results. I'm guessing this is something to do with one or more of the characters in the results being illegal as far as Django is concerned. How can I get more information about what it is that Django doesn't like? Also, is there some method I can use to filter out "bad" characters? The results are normally just lots of text. They contain company confidential stuff, so I can't give an example unfortunately. I have DEBUG set to True and TEMPLATE_DEBUG set to DEBUG.
UPDATE:
I added some code to filter out all chars with a decimal value greater than 127 and it now works.
If you are using the development server, put in a breakpoint with pdb and see what is going on. Or print out the string that you think has "bad" characters. If you aren't using the development server you could use the Python logging module to log the string you are getting from the tool.
You might be leaping to conclusions about the data containing bad characters. It may be something else, and without debugging further it is hard to speculate.
you could try using the built in django encoding methods to remove illegal characters.
from django.utils.encoding import smart_str
smart_str(your_string)
Related
I'm running the latest version of Django on postgres. I'm trying to store emojis in my postgres DB in a way that a React Native app can properly render it. Below I have the initial emojis variables setup that'll go into the table. I've copy and pasted the emojis from here. How do I store emojis in my postgres DB so that a React Native app can render it properly?
I tried following this blog, which suggests adding ’OPTIONS’: {’charset’: ’utf8mb4’} to DATABASES under settings.py, but I get this error django.db.utils.ProgrammingError: invalid dsn: invalid connection option "charset". Seems like this only works for MySQL DBs. How can I store emojis in a Django postgres DB?
Like in the comments suggested, you need to put quotes around the emojis since they're just chars. Though, something like flags is actually two chars. So that's something to be careful about. All your computer is doing is converting unicode to a rendered emoji that's platform dependent.
The emojis that you're using should be unicode supported. On your computer, they're definitely supported. For the most part, additional unicode support for new emojis is very quickly implemented once published on client machines. There should be no problem with emojis in strings. This is a nice video kinda explaining emojis by Tom Scott who keeps getting interviews about emojis: https://www.youtube.com/watch?v=sTzp76JXsoY
I'm not an expert so please correct me if I'm wrong.
In your models you need to use a CharField or a TextField to store emojis, that need to be passed as characters (for example "😄" and not directly 😄). Your database must use utf8 to support emojis, connect to your database with a SQL shell, to check the current encoding run:
SHOW CLIENT_ENCODING;
If the output is not UTF8 run:
SET CLIENT_ENCODING='UTF8';
Now remove ’OPTIONS’: {’charset’: ’utf8mb4’} from your Django settings.
I'm trying to find something that will return an exception upon finding anything that even remotely looks like HTML or Javascript. I've figured out how to do it for individual views, but it's not a scalable solution, and ultimately I need to prevent code from being saved to the database no matter what view gets targeted by the injection attack.
Here is the functionality I'm looking for.
ILLEGAL_CHARS = '<>[]{}():;,'.split()
# bunch of code in between
for value in [company_name, url, status, information, lt_type, company_source]:
if any(char in value for char in ILLEGAL_CHARS):
raise Exception(f"You passed one of several illegal characters: {ILLEGAL_CHARS}")
I'm using django rest framework so I have to handle it on the backend. Thanks.
actually you don't nead to sanitize any user input because when you show them int the template the jinja {{object}} will make sure that no html or java script will be executed until you mark them as safe {{object|safe}} but if you want want not to save them in database that might help Sanitizing HTML in submitted form data
I am building an app running on GAE which receives input from users via webform:
myUnicodeString = cgi.escape(self.request.get('myForm'))
It all works fine locally but after deployment unicode literals are converted into strings of the form: "E2=80=9C no problems with ASCII strings"
Having read Nick's comment here not to use cgi.escape I was wondering that it might be the culprit.
I have also tried adding
from __future__ import unicode_literals
after reading this post but then the program throws an error (TypeError: character mapping must return integer, None or unicode) which is apparently triggered by webapp2_extras session
Any ideas greatly appreciated!
UPDATE:
I have noticed that this decoding/encoding issue has something to do with the text input fields submitted in the same form as the file uploaded to the blobstore. No problems occur while I save the same non ASCII strings via separate forms or via ajax.
UPDATE2:
This is apparently the bug that causes the problem.
I just set up the environment for an existing Django project, on a new Mac. I know for certain there is nothing wrong with the code itself (just cloned the repo), but for some reason, Django can't seem to retrieve data from the database.
I know the correct tables and data is in the db.
I know the codebase is as it should be.
I can make queries using the Django shell.
Django doesn't throw any errors despite the data missing on the web page.
I realize that it's hard to debug this without further information, but I would really appreciate a finger pointing me to the right direction. I can't seem to find any useful logs.
EDIT:
I just realized the problem lies elsewhere. Unfortunately I can't delete this post with the bounty still open.
Without seeing any code, I can only suggest some general advice that might help you debug your problem. Please add a link to your repository if you can or some snippets of your database settings, the view which includes the database queries etc...
Debugging the view
The first thing I would recommend is using the python debugger inside the view which queries the database. If you've not used pdb before, it's a life saver which allows you to set breakpoints in your Python script and then interactively execute code inside the interpreter
>>> import pdb
>>> pdb.set_trace()
>>> # look at the results of your queries
If you are using the Django ORM, the QuerySet returned from the query should have all the data you expect.
If it doesn't then you need to look into your database configuration in settings.py.
If it does, then you must might not be returning that object to the template? Unlikely as you said the code was the same, but double check the objects you pass with your HttpResponse object.
Debugging the database settings
If you can query the database using the project settings inside settings.py from the django shell it sounds unlikley that there is a problem with this - but like everything double check.
You said that you've set up a new project on a mac. What is on a different operating system before? Maybe there is a problem with the paths now - to make your project platform independent remember to use the os.path.join() method when working with file paths.
And what about the username and password details....
Debugging the template
Maybe your template is referencing the wrong object variable name or object attribute.You mentioned that
Django doesn't throw any errors despite the data missing on the web
page.
This doesn't really tell us much - to quote the Django docs -
If you use a variable that doesn’t exist, the template system will
insert the value of the TEMPLATE_STRING_IF_INVALID setting, which is
set to '' (the empty string) by default.
So to check all the variables available to your template, you could use the debug template tag
{{ debug }}
Probably even better though is to use the django-debugging-toolbar - this will also let you examine the SQL queries your view is making.
Missing Modules
I would expect this to raise an exception if this were the problem, but have you checked that you have the psycopg module on your new machine?
I have stumbled on a strange issue that I can't resolve:
In my Django app there is a method which gets hit by a POST from a java applet, which sends it a JSON object. Django method parses it like so:
req = json.loads(request.raw_post_data)
and based on the results returns a value. I haven't written this code, but yesterday I was sent to investigate an error triggered in this method. It was saying there was "ValueError: Expecting property name: line 1 column 1 (char 1)".
What I discovered is that my raw post data looks like this:
{#012#011"ImmutableMachineFactors": #012#011{#012#011#011"machineName": "lukka",#012#011#011"osName": "MacOS"}}
The type of it was string, however, my attempts to replace these weird characters with spaces or nothing failed. It would just ignore the sub() command. I know that raw_post_data returns a bytestring, but when I tried to convert it to a regular string using:
mystring.decode('utf-8')
it did add the u'' notation, but didn't remove those weird characters. Stranger still, in many cases (on my personal machine), Django would happily convert this kind of data into JSON, it only fails sometimes, which led me to believe that the JSON which triggered the error was malformed, but when I would strip out all the #011 and #012 characters, it parsed perfectly.
My questions are:
1) What are those crazy things? (#011, #012). I tried to google around, but these are very common things to find in a search, so I couldn't find anything relevant.
2) How can I turn this bytestring into a regular string so that I can replace those characters? Or is it the wring way to approach this problem?
Thanks!
Luka
This may be way too late to help, but since QueryDict instances (request.POST or request.DATA) are immutable, it's reasonable to expect that request.raw_post_data is also immutable. You'd have to make a copy before changing it.