UnicodeEncodeError when saving ImageField containing non-ASCII characters in Django admin - django

I'm trying to upload an image file in django admin inlines and getting UnicodeEncodeError when trying to upload a file with a filename containing non-ascii characters:
File "/usr/local/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
self.name = self.storage.save(name, content)
File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 47, in save
name = self.get_available_name(name)
File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 73, in get_available_name
while self.exists(name):
File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 196, in exists
return os.path.exists(self.path(name))
File "/usr/local/lib/python2.6/genericpath.py", line 18, in exists
st = os.stat(path)
There is a paragraph about this issue in Django docs: http://docs.djangoproject.com/en/dev/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror - they say I must define LANG and LC_ALL env variables, plus defining them using os.env won't work. So I've defined them in my .htaccess file and I'm sure they are there:
META
Variable Value
CONTENT_LENGTH '27289'
...
LANG 'en_US.UTF-8'
LC_ALL 'en_US.UTF-8'
LC_LANG 'en_US.UTF-8'
The problem still exists. Django version is 1.2.3 (latest stable), sys.getfilesystemencoding() (which I believe is relevant to the issue) returns "ANSI_X3.4-1968".
The model/admin code is nothing special: an ArticleImage model with ImageField, and ArticleAdmin containing ArticleImage inlines.
UPDATE I couldn't fix this issue so I've given up using apache setup and started the application using runfcgi + nginx. Uploads work fine now but I'm not adding this as a solution because the question was about apache.

On Debian (Lenny) you simply add the following two lines to /etc/apache2/envvars:
export LANG='en_GB.UTF-8'
export LC_ALL='en_GB.UTF-8'
...that's for UK web servers. For US:
export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'
And restart Apache.

You should try defining the LANG and LC_ALL for the whole Apache 2 environment.
For my deployments I also make sure that the python default system encoding is set to utf-8 as well.
For the Python default encoding I usually create/edit sitecustomize.py, see http://blog.ianbicking.org/illusive-setdefaultencoding.html
As for Apache - there is line in init script /etc/init.d/apache2 (Ubuntu 8.04 LTS) that creates the environment. I added the correct LC_ALL, LANG there. Basically it should be in the server init scripts somewhere for all the OSes.

your can do like this.
in linux:
echo $LANG i got zh_CN.UTF-8
in apache2/envvars
export LANG='zh_CN.UTF-8' #keep this variable like echo $LANG.
export LC_ALL='zh_CN.UTF-8' #the same.
https://docs.djangoproject.com/en/1.4/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror
this doc is help me too.
i think it's the os and apache problem!

Related

Flask Security - TemplateAssertionError: no filter named 'urlencode'

I just added flask-security to my flask project. It works locally, but reports this error on OpenShift:
TemplateAssertionError: no filter named 'urlencode'
I don't know if it is some wrong library version, or how to debug this. This is my setup.py package list:
install_requires=['Flask==0.10.1',
'SQLAlchemy==0.9.8',
'Flask-SQLAlchemy==2.0',
'Flask-Security==1.7.4',
'Werkzeug==0.9.5',
'blinker==1.3',
'Flask-Login==0.2.11',
'Flask-Mail==0.9.1',
'Flask-Principal==0.4.0',
'Flask-Script==2.0.5',
'Flask-WTF==0.10.3',
'itsdangerous==0.24',
'passlib==1.6.2'
]
The urlencode filter was added to jinja in v2.7. But GAE only supports v2.6. Changing the version to "latest" in my app.yaml still runs with 2.6 (notice the python27_lib/versions/third_party/jinja2-2.6/jinja2/environment.py path):
...
File "/base/data/home/apps/s~healthier-staging/1.386037228785871893/lib/flask/templating.py", line 128, in render_template
context, ctx.app)
File "/base/data/home/apps/s~healthier-staging/1.386037228785871893/lib/flask/templating.py", line 110, in _render
rv = template.render(context)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/jinja2-2.6/jinja2/environment.py", line 894, in render
return self.environment.handle_exception(exc_info, True)
File "/base/data/home/apps/s~healthier-staging/1.386037228785871893/lib/flask_security/templates/security/_menu.html", line 4, in template
<li>Login</li>
TemplateAssertionError: no filter named 'urlencode'`
I fixed this by adding a simple filter (copying the code that was added to jinja) manually:
def do_urlescape(value):
"""Escape for use in URLs."""
return urllib.quote(value.encode('utf8'))
app.jinja_env.globals['urlencode'] = do_urlescape
I have solved this, by doing 'pip freeze' on my local machine, and copying libraries to setup.py one by one. Though I'm still not sure which one produced an error, probably wrong version of jinja2.

How to resolve "django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: foo" in Django 1.7?

On upgrading to Django 1.7 I'm getting the following error message from ./manage.py
$ ./manage.py
Traceback (most recent call last):
File "./manage.py", line 16, in <module>
execute_from_command_line(sys.argv)
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 427, in execute_from_command_line
utility.execute()
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 391, in execute
django.setup()
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/__init__.py", line 21, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/apps/registry.py", line 89, in populate
"duplicates: %s" % app_config.label)
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: foo
What's the problem and how do I resolve it?
The problem is that with the changes to apps in Django 1.7, apps are required to have a unique label.
By default the app label is the package name, so if you've got a package with the same name as one of your app modules (foo in this case), you'll hit this error.
The solution is to override the default label for your app, and force this config to be loaded by adding it to __init__.py.
# foo/apps.py
from django.apps import AppConfig
class FooConfig(AppConfig):
name = 'full.python.path.to.your.app.foo'
label = 'my.foo' # <-- this is the important line - change it to anything other than the default, which is the module name ('foo' in this case)
and
# foo/__init__.py
default_app_config = 'full.python.path.to.your.app.foo.apps.FooConfig'
See https://docs.djangoproject.com/en/1.7/ref/applications/#for-application-authors
I found simple solution for this. In my case following line is added twice under INSTALLED_APPS,
'django.contrib.foo',
Removed one line fixes the issue for me.
I had the same error - try this:
In INSTALLED_APPS, if you are including 'foo.apps.FooConfig', then Django already knows to include the foo app in the application, there is therefore no need to also include 'foo'.
Having both 'foo' and 'foo.apps.FooConfig' under INSTALLED_APPS could be the source of your problem.
Well, I created a auth app, and included it in INSTALLED_APP like src.auth (because it's in src folder) and got this error, because there is django.contrib.auth app also. So I renamed it like authentication and problem solved!
I got the same problem.
Here my app name was chat and in the settings.py , under installed apps i have written chat.apps.ChatConfig while i have already included the app name chat at the bottom. When i removed the chat.apps.ChatConfig mine problem was solved while migrations. This error may be due to the same instance that you might have defined you app name foo twice in the settings.py. I hope this works out!!
please check if anything is duplicated in INSTALLED_APPS of settings.py
This exception may also be raised if the name of the AppConfig class itself matches the name of another class in the project. For example:
class MessagesConfig(AppConfig):
name = 'mysite.messages'
and
class MessagesConfig(AppConfig):
name = 'django.contrib.messages'
will also clash even though the name attributes are different for each configuration.
In previous answer 'django.contrib.foo', was mentioned, but basically adding any app twice can cause this error just delete one (Django 3.0)
for me it was in settings.py
INSTALLED_APPS = [
...
'accounts.apps.AccountsConfig',
'accounts.apps.AccountsConfig',
...
]
just delete one of them
Basically this problem has been created due to duplication of name of installed app in the settings:
This is how I resolved the problem. In settings.py file:
Check the install app in the setting.py if the install app are duplicate
Error shown due to duplication of app name
Remove the duplicate name in the install file
After problem is resolved, you will see interface in your screen
For this I have created application name as polls instead of foo
As therefromhere said this is a new Django 1.7 feature which adds a kind of “app registry” where applications must be determined uniquely (and not only having different python paths).
The name attribute is the python path (unique), but the label also should be unique. For example if you have an app named 'admin', then you have to define the name (name='python.path') and a label which must be also unique (label='my admin' or as said put the full python path which is always unique).
Had same issue, read through the settings.py of the root folder, removed any INSTALLED APPS causing conflict... works fine. Will have to rename the apps names
Need to check in two file
1- apps.py
code something like
from django.apps import AppConfig
from django.utils.translation import ugettext_lazy as _
class ModuleConfig(AppConfig):
name = "ocb.module_name"
verbose_name = _("Module Name")
2 - init.py
code something like
default_app_config = "ocb.users.apps.ModuleConfig"
default_app_config is pointed to your apps.py's class name
in my case, in mysite settings.py , in INSTALLED_APPS array variable I put the name of the app twice by mistake.
I had almost the same issue.
```File "/Users/apples/.local/share/virtualenvs/ecommerce-pOPGWC06/lib/python3.7/site-packages/django/apps/registry.py", line 95, in populate
"duplicates: %s" % app_config.label)
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: auth```
I had installed Django.contrib.auth twice. I removed one and it worked well.
From my experience, this exception was masking the real error. To see the real error (which in my case was an uninstalled python package) comment out the following in django/apps/registry.py:
if app_config.label in self.app_configs:
# raise ImproperlyConfigured(
# "Application labels aren't unique, "
# "duplicates: %s" % app_config.label)
pass
Check for duplicates in INSTALLED_APPS inside the settings.py...If so remove one of it and rerun the command
I had Django==3.2.9 when tried to test my existing Django app on a new environment. I had this exact issue and fixed it by downgrading to Django==3.1.13.
There seems to be an update to applications, check the Django 3.2 documentation for detailed information.
This error occurs because of duplication in your INSTALLED_APPS in settings.py file which is inside your project.
For me, the problem was that I had copy-pasted entire app instead of creating it using command line. So, the app name in the apps.py file was same for 2 apps. After I corrected it, the problem was gone.
In case if you have added your app name in settings.py
example as shown in figure than IN settings.py Remove it and Try this worked for me.
give it a try .
This Worked Because settings.py assumes installing it twice and does not allow for migration
If you want to back older version, command
pip install django==1.6.7

autodoc failing on django ImageField

I'm attempting to use sphinx to document the django app I'm writing. So far I have my code.rst setup to look at models.py, and when I run make html I get the automatic documentation, however I get a warning:
WARNING: autodoc can't import/find attribute 'myapp.models.MyModel.image', it reported error: "image", please check your spelling and sys.path
The entire tracelog is:
Traceback (most recent call last):
File "C:\blah\lib\site-packages\sphinx\ext\autodoc.py", line 326, in import_object
obj = self.get_attr(obj, part)
File "C:\blah\lib\site-packages\sphinx\ext\autodoc.py", line 232, in get_attr
return safe_getattr(obj, name, *defargs)
File "C:\blah\lib\site-packages\sphinx\util\inspect.py", line 70, in safe_getattr
raise AttributeError(name)
AttributeError: image
MyModel.image is an image field, simply on the model defined as:
#: image file location for ``MyModel``
image = models.ImageField(upload_to="images/")
If I change image to models.Charfield for example it runs fine. Is there any reason why an ImageField would cause sphinx issues?
my code.rst:
.. automodule:: dynamicbanners.models
:members:
So after reading the comments it seems this is a quirk of Django that messes with sphinx.ext.autodoc. Hopefully a fix will be dropped into Django 1.6 soon, or failing that I think the upcoming 1.2 version of Sphinx might provide a way around it, but for now I'll need to find some sort of workaround.
Looks like this was fixed in Django 1.8 - https://code.djangoproject.com/ticket/12568

losing locale when running fcgi script

I'm facing an issue with encoding in running a django app.
I finally found out my django app has no locale set.
The weird thing is that I did set up the envvars file correctly. With this in envvars :
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2.pid
## The locale used by some modules like mod_dav
export LANG=C
## Uncomment the following line to use the system default locale instead:
. /etc/default/locale
export LANG
locale
When I restart apache the locale command gets executed and I get correct fr_FR.UTF-8 settings for LANG and LC_*.
Now I set up a little test.fcgi script :
#!/usr/bin/python
def myapp(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
from commands import getoutput
return ["%s"%getoutput("locale")]
from flup.server.fcgi import WSGIServer
WSGIServer(myapp).run()
when I run it with
sudo -u www-data test.fcgi
I get the correct locale settings as well.
But whenever I access the script through a web browser, I get no locale settings :
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
How come Apache has the right setting but my fcgi script hasn't?
I solved it by adding DefaultInitEnv LANG "en_US.UTF-8" in my sites-available/default. Now the fcgi script tells me UTF-8 !

Django makemessages ignore switch doesn't work for me

I have problems localizing a django-nonrel project, which is deployed to GAE. Because of GAE I have to put everything into my project folder, so it looks like something like this
project
+ django
+ dbindexer
+ registration
+ myapp
...
+ locale
+ templates
I have strings to localize in templates directory, and in the myapp directory.
When I run python manage.py makemessages -l en --ignore django\* from the project dir it crawl through all the directories of the project, including django, so I get a quite big po file. My strings from the templates are there, along with all of the strings from django directory.
after --ignore ( or just -i ) I tried to pu django django/* , but nothing changed.
Any ideas?
./manage.py help makemessages
-i PATTERN, --ignore=PATTERN
Ignore files or directories matching this glob-style
pattern. Use multiple times to ignore more.
I have just tested it, and this command successfully ignored my application:
./manage.py makemessages -l da -i "django*"
But beware that before you test it, you should delete the old .po file, as I think it will not automatically remove the translation lines from your previous makemessages execution.
The problem is with the pattern - maybe the shell was expanding it for you.
In general - it is good to avoid path separators (whether / or \) in the pattern.
If you need to always pass specific options to the makemessages command, you could consider your own wrapper, like this one, which I use myself:
from django.conf import settings
from django.core.management.base import BaseCommand
from django.core.management import call_command
class Command(BaseCommand):
help = "Scan i18n messages without going into externals."
def handle(self, *args, **options):
call_command('makemessages',
all=True,
extensions=['html', 'inc'],
ignore_patterns=['externals*'])
This saves you typing, and gives a common entry point for scanning messages across the project (your translator colleague will not destroy translations by missing out some parameter).
Don't delete the old .po file, once you have cleared it from the totally unwanted (i.e. - those from 'django' directory) messages. This allows gettext to recycle old unused messages, once they are used again (or simmilar ones, which will be marked as #, fuzzy.
Edit - as mt4x noted - the wrapper above doesn't allow for passing the options to the wrapped command. This is easy to fix:
from django.core.management import call_command
from django.core.management.commands.makemessages import (
Command as MakeMessagesCommand
)
class Command(MakeMessagesCommand):
help = "Scan i18n messages without going into externals."
def handle(self, *args, **options):
options['all'] = True
options['extensions'] = ['html', 'inc']
if 'ignore_patterns' not in options:
options['ignore_patterns'] = []
options['ignore_patterns'] += ['externals*']
call_command('makemessages', **options)
Thus - you can fix what needs to be fixed, and flex the rest.
And this needs not be blind override like above, but also some conditional edit of the parameters passed to the command - appending something to a list or only adding it when it's missing.