I have an object that uses \r\n line breaks in a CharField.
When I export using dumpdata it provides valid json:
{
"mykey": "example\r\nsentence\r\nhere"
}
However when I import again using loaddata the CharField is stripped of newlines.
How do I preserve them?
Related
I have a Unicode, which is read from a CSV file:
df.iloc[0,1]
Out[41]: u'EU-repr\xe6sentant udpeget'
In [42]: type(df_translated.iloc[0,1])
Out[42]: unicode
I would like to have it as EU-repræsentant udpeget. The final goal is to write this into a dictionary and then finally save that dict to a YAML file with PyYAML using safe_dump. However, I struggle with the encoding.
If you really need to use PyYAML you should provide the arguments
encoding='utf-8' and allow_unicode=True to the safe_dump()
routine.
If you ever intend to upgrade to YAML 1.2 and use ruamel.yaml
(disclaimer: I am the author of that package), those are the (much
more sensible) defaults:
import sys
import ruamel.yaml
yaml = ruamel.yaml.YAML()
data = [u'EU-repr\xe6sentant udpeget']
yaml.dump(data, sys.stdout)
which gives:
- EU-repræsentant udpeget
This piece of python code worked fine an hour ago, before I ran an apt-get upgrade on my raspberry pi.
This is now my python version: Python 2.7.9 (default, Sep 17 2016, 20:26:04)
import urllib, urllib2
from PIL import Image
URL="http://server.local/picture.jpg"
headers = {'Authorization': 'Basic ' + base64.encodestring('Guess:Thepassword')}
req = urllib2.Request(URL, None, headers)
img=Image.open(urllib2.urlopen(req,timeout=1))
But now I get this error:
File "/usr/lib/python2.7/httplib.py", line 1017, in putheader
raise ValueError('Invalid header value %r' % (one_value,))
ValueError: Invalid header value 'Basic TGlvbjpSdW5SYWJiaXRSdW4=\n'
I assume something has changed, but can't figure out what..
You can't have a new line character \n at the end of your header. Instead of using base64.encodestring, use base64.b64encode.
I don't think this has anything to do with an update to Python, since this behaviour has been there since the base64 module was included back in Python 2.4 (see the bolded text):
Encode the string s, which can contain arbitrary binary data, and
return a string containing one or more lines of base64-encoded data.
encodestring() returns a string containing one or more lines of
base64-encoded data always including an extra trailing newline ('\n').
FYI - I believe the change in functionality can be traced to this security update:
https://launchpad.net/ubuntu/+source/python2.7/2.7.6-8ubuntu0.3:
SECURITY UPDATE: CRLF injection vulnerability in the
HTTPConnection.putheader
- debian/patches/CVE-2016-5699.patch: disallow newlines in
putheader() arguments when not followed by spaces or tabs in
Lib/httplib.py, add tests in Lib/test/test_httplib.py
- CVE-2016-5699
How to get all my data from model to JSON? I need export and import my data
Is there any command?
You can use dumpdata command to dump data in your table. By default it gives in JSON format.
You can do in command line, to print data on screen.
$ python manage.py dumpdata
As suggested by Rohan, you need the dumpdata command.
I generally do an app at a time, output to file, and add an indent to the output to make it more readable -
$ python manage.py dumpdata --indent 2 myapp > /path/to/myapp/fixtures/my_data.json
First create fixtures folder inside your app.
Dump all models data of the app:
python manage.py dumpdata --format=json --indent=4 [app_name] > [app_name]/fixtures/initial_data.json
I'm trying to upload an image file in django admin inlines and getting UnicodeEncodeError when trying to upload a file with a filename containing non-ascii characters:
File "/usr/local/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
self.name = self.storage.save(name, content)
File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 47, in save
name = self.get_available_name(name)
File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 73, in get_available_name
while self.exists(name):
File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 196, in exists
return os.path.exists(self.path(name))
File "/usr/local/lib/python2.6/genericpath.py", line 18, in exists
st = os.stat(path)
There is a paragraph about this issue in Django docs: http://docs.djangoproject.com/en/dev/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror - they say I must define LANG and LC_ALL env variables, plus defining them using os.env won't work. So I've defined them in my .htaccess file and I'm sure they are there:
META
Variable Value
CONTENT_LENGTH '27289'
...
LANG 'en_US.UTF-8'
LC_ALL 'en_US.UTF-8'
LC_LANG 'en_US.UTF-8'
The problem still exists. Django version is 1.2.3 (latest stable), sys.getfilesystemencoding() (which I believe is relevant to the issue) returns "ANSI_X3.4-1968".
The model/admin code is nothing special: an ArticleImage model with ImageField, and ArticleAdmin containing ArticleImage inlines.
UPDATE I couldn't fix this issue so I've given up using apache setup and started the application using runfcgi + nginx. Uploads work fine now but I'm not adding this as a solution because the question was about apache.
On Debian (Lenny) you simply add the following two lines to /etc/apache2/envvars:
export LANG='en_GB.UTF-8'
export LC_ALL='en_GB.UTF-8'
...that's for UK web servers. For US:
export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'
And restart Apache.
You should try defining the LANG and LC_ALL for the whole Apache 2 environment.
For my deployments I also make sure that the python default system encoding is set to utf-8 as well.
For the Python default encoding I usually create/edit sitecustomize.py, see http://blog.ianbicking.org/illusive-setdefaultencoding.html
As for Apache - there is line in init script /etc/init.d/apache2 (Ubuntu 8.04 LTS) that creates the environment. I added the correct LC_ALL, LANG there. Basically it should be in the server init scripts somewhere for all the OSes.
your can do like this.
in linux:
echo $LANG i got zh_CN.UTF-8
in apache2/envvars
export LANG='zh_CN.UTF-8' #keep this variable like echo $LANG.
export LC_ALL='zh_CN.UTF-8' #the same.
https://docs.djangoproject.com/en/1.4/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror
this doc is help me too.
i think it's the os and apache problem!
I'd like to call the equivalent of manage.py loaddata from a Django view. I'd like to be able to specify where to load the data from and which application to load it into.
Any ideas?
Each django-admin.py (manage.py) command, as seen in the documentation, you can call from your code with:
from django.core.management import call_command
call_command('loaddata', 'myapp')
Where first param is the command name, all other position params are the same as command line position params and all keyword params are options.