wkhtmltopdf subprocess error using django wrapper, but not in shell - django

I'm using wkhtmltopdf together with the django-wkhtmltopdf wrapper to create a .pdf of a template.
I'm using the example from the django-wkhtmltopdf documentation (though I eventually want more than just a static template):
url(r'^pipeline/snapshot/$', PDFTemplateView.as_view(
template_name='pdf/pipeline_snapshot.html',
filename='my_pdf.pdf'), name='pdf'),
And I'm getting the error:
Traceback:
File "/home/pluscitizen/webapps/odin/lib/python2.7/django/core/handlers/base.py" in get_response
140. response = response.render()
File "/home/pluscitizen/webapps/odin/lib/python2.7/django/template/response.py" in render
105. self.content = self.rendered_content
File "/home/pluscitizen/webapps/odin/lib/python2.7/django_wkhtmltopdf-1.2.2-py2.7.egg/wkhtmltopdf/views.py" in rendered_content
144. footer_filename=footer_filename)
File "/home/pluscitizen/webapps/odin/lib/python2.7/django_wkhtmltopdf-1.2.2-py2.7.egg/wkhtmltopdf/views.py" in convert_to_pdf
103. return wkhtmltopdf(pages=[filename], **cmd_options)
File "/home/pluscitizen/webapps/odin/lib/python2.7/django_wkhtmltopdf-1.2.2-py2.7.egg/wkhtmltopdf/utils.py" in wkhtmltopdf
92. return check_output(ck_args, **ck_kwargs)
File "/usr/local/lib/python2.7/subprocess.py" in check_output
575. raise CalledProcessError(retcode, cmd, output=output)
Exception Type: CalledProcessError at /pipeline/snapshot/
Exception Value: Command '['wkhtmltopdf', '--encoding', u'utf8', '--quiet', '/tmp/wkhtmltopdfVaAKrX.html', '-']' returned non-zero exit status 127
However, when I run the same command, with the same file, from the Django shell
>>> subprocess.check_output(['wkhtmltopdf', '--encoding', u'utf8', '--quiet', '/tmp/wkhtmltopdfSGFfYh.html', '-'])
everything runs fine. Ditto for:
>>> wkhtmltopdf(['/tmp/wkhtmltopdfSGFfYh.html'], **{})
So, figuring that the difference must be in the whole shell situation, I've tried adding a shell=True to the call to subprocess within django-wkhtmltopdf (I know it's a security problem), but no luck there. Probably because I have no idea what's really going on.
I saw somewhere saying that the problem might have to do with PATHs not being set up properly, but then I don't understand why Django's shell is not having issues with this.
This entire process has been incredibly taxing, and now that I'm so close, I'm finally turning to SO for an answer.
EDIT: I have tried running the subprocess directly in a view, and it returns the same error, which I guess means that the shell and the server aren't necessarily both running from the same environment?
FIXED: Solved, and the answer by sparks led me on the way. I decided to look at the logs (duh) output by the django app on the server, and noticed that before the traceback, I'm getting the actual output of the command:
wkhtmltopdf: error while loading shared libraries: libwkhtmltox.so.0: cannot open shared object file: No such file or directory
Which seems to imply that it's not finding the libraries I have in /home/user/lib. When I explicitly added these to the WKHTMLTOPDF_ENV variable in settings.py, everything worked swimmingly.

exit status 127
This means that the command you're trying to run can't be found. It is an error in the environment of your process.
I would suggest looking at the contents of sys.path in python and comparing that to the PATH environment variable, or compared to sys.path in the django console.

More specifically, add this to settings.py:
WKHTMLTOPDF_ENV = {'FONTCONFIG_PATH': '/etc/fonts'}

Related

Pytest does not see files in folders in Django

I have a following question:
I have a test function written in Pytest for Django project:
#pytest.mark.django_db
def test_class():
path = Path(r'ascertain\tests\csv')
handler = DatabaseCSVUpload(path, delimiter=',')
handler()
ascertain\tests\test_upload_csv.py:19 (TestUploadCSVtoDatabaseNegative.test_class)
Traceback (most recent call last):
File "C:\Users\hardcase1\PycharmProjects\telephone_numbers\ascertain\tests\test_upload_csv.py", line 27, in test_class
handler()
File "C:\Users\hardcase1\PycharmProjects\telephone_numbers\ascertain\handle_csv.py", line 129, in __call__
for file in self.get_csv_files():
File "C:\Users\hardcase1\PycharmProjects\telephone_numbers\ascertain\handle_csv.py", line 48, in get_csv_files
raise EmptyFolder()
telephone_numbers.custom_exceptions.EmptyFolder:
It basically expects *.csv files be inside directory ‘path’ and handle them to upload to DB.
Problem is that Pytest can’t see any files in this directory as well as in other directories. I have tried multiple times with different folders.
In fact file is there:
list(Path(r'ascertain\tests\csv').glob('*.csv'))
[WindowsPath('ascertain/tests/csv/test.csv')]
Same test function written in unitests works correctly:
from rest_framework.test import APITestCase
class TestCase(APITestCase):
def test_open(self):
path = Path(r'ascertain\tests\csv')
handler = DatabaseCSVUpload(path, delimiter=',')
handler()
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...
Process finished with exit code 0
Question is -what I need to do to make Pytest see this file?
Thank you!
I'm not sure this problem relates to Pytest vs Unittest, rather I think it has to do with where you executing these files from. I think you can resolve this by using an absolute path instead of the relative path you included in the example.
An absolute path will start at your systems root directory instead of starting at the directory you are executing the file from. I'm not a Windows user, but from what I can see online the path to this data should look something like this C:/users/path/to/data.

neovim: no module named __future__

When I try to open certain python files in neovim, I get an error:
"pool.py" 667L, 25276C
function provider#python#Call[9]..remote#host#Require[10]..provider#pythonx#Require, line 15
Vim(if):ch 1 was closed by the client
Traceback (most recent call last):
File "/home/user/.pyenv/versions/neovim2/lib/python2.7/site.py", line 67, in <module>
import os
File "./os.py", line 44, in <module>
from __future__ import absolute_import
ImportError: No module named __future__
Failed to load python host. You can try to see what happened by starting nvim with $NVIM_PYTHON_LOG_FILE set and opening the generated log file. Also
, the host stderr is available in messages.
Press ENTER or type command to continue
This happens any time I open a python file in a directory that contains an os.py or os.pyc file. It looks like neovim is trying to import the local os.py file instead of the one in the virtualenv.
What can I do about this?
EDIT: turns out it's not when I open a file in the same directory as an os.py file, it's when I open a file anywhere while the current working directory has an os.py file. Basically, it looks like python is checking the local directory for imports before checking the python libs.
I figured it out. The problem was with my $PYTHONPATH. I had in my .bashrc file this:
export PYTHONPATH="$PYTHONPATH:~/.local/lib/python"
The problem was that, when that line is executed, $PYTHONPATH is empty, leading to the string starting with a :. I'm not 100% sure why, but that resulted in python checking the local directory for a module BEFORE checking the python libraries.
I changed it to
if [ -z "$PYTHONPATH" ]; then
export PYTHONPATH="~/.local/lib/python"
else
export PYTHONPATH="$PYTHONPATH:~/.local/lib/python"
fi
And now it works.

Windows executable file generated by PyGame + PyInstaller gives me "No such file or directory" error

I have first tried to create my original game sample using Pygame.
I have traced the instruction written in the Web site below:
https://irwinkwan.com/2013/04/29/python-executables-pyinstaller-and-a-48-hour-game-design-compo/
Specifically,
I created "myFile" class something like following and read every file (.txt, .png, .mp3, etc...) using this class
myFile Class
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import sys
class myFile():
def resource_path(self, relative):
if hasattr(sys, "_MEIPASS"):
return os.path.join(sys._MEIPASS, relative)
return os.path.join(relative)
### code where I read something ###
myfile = myFIle()
filename = myfile.resource_path(os.path.join("some dir", "somefile"
+ "extension")
I typed command below to create .spec file (myRPG.py contains main)
pyinstaller --onefile myRPG.py
I modified .spec file so exe object include Trees (I store data files in separate
directories such as data, image, and so on)
myRPG.spec file
# -*- mode: python -*-
block_cipher = None
a = Analysis(['myRPG.py'],
pathex=['C:\\mygame\\mygame'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
Tree('bgm', prefix='bgm'),
Tree('charachip', prefix='charachip'),
Tree('data', prefix='data'),
Tree('image', prefix='image'),
Tree('mapchip', prefix='mapchip'),
Tree('se', prefix='se'),
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='myRPG',
debug=False,
strip=False,
upx=True,
console=True )
I did rebuild my package using modified .spec file
pyinstaller myRPG.spec
When I execute myRPG.exe file, I got the error below
C:\mygame\mygame\dist>myRPG.exe
[Errno 2] No such file or directory:
'C:\\Users\\bggfr\\AppData\\Local\\Temp\\_MEI64~1\\item.data' IO Errorが発生しました。
Traceback (most recent call last):
File "myRPG.py", line 606, in <module>
File "myRPG.py", line 37, in __init__
File "myItemList.py", line 11, in __init__
File "myItemList.py", line 33, in load
UnboundLocalError: local variable 'fp' referenced before assignment
Failed to execute script myRPG
I believe that I properly specify the directories where data is expanded since I use the function that checks _MEIPASS but it does not work.
I have also tried to use "added_files" instead of Tree did not help for me at all. ”a.datas" may work for small number of files but I do not want to specify all the files I'm going to use, because it will be over hundreds of thousands of files.
This is a very helpful page that I used when converting my pygame program into an executable.
In short:
Here is the place where you can download pyInstaller. Click on the download button under "Installation." Wait for it to download, then run it.
Type this into the popup where it says command: venv -c -i pyi-env-name. The reason for this is stated in the page mentioned at the top.
If you have pyInstaller installed already, type pyinstaller --version to make sure. If you don't, type pip install PyInstaller. If this doesn't work, look at step 7 in the guide up top.
Make sure that you put a copy of your program in the folder that it says on the command line.
If you want a zip with the exe and all necessary files in it type pyinstaller and then the name of your program, complete with the .py extension. However, if you want an exe that doesn't need all of the extra files in the zip that you can just run by itself, type pyinstaller --onefile --windowed and then the name of your program, complete with the extension of .py.
Once you have done this, the exe will appear in a dist folder inside of the one where you placed your program in step 3.
This is how it worked for me, and I apologize if I have misinformed you. Be sure to upvote/mark as answered if this works.

GeoDjango GDAL_ERROR when uplading .shp

Try to parse a .shp in django shell:
from django.contrib.gis.gdal import DataSource
ds = DataSource('/Users/.../Downloads/Iceland.shp')
get:
GDAL_ERROR 4: Unable to open /Users/.../Downloads/Iceland.shx or /Users/.../Downloads/Iceland.SHX.
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Users/.../lib/python2.7/site-packages/django/contrib/gis/gdal/datasource.py", line 78, in __init__
raise GDALException('Could not open the datasource at "%s"' % ds_input)
GDALException: Could not open the datasource at "/Users/.../Downloads/Iceland.shp"
File exists, chmod is 755, .shx file is correct (tested in online services).
Then I try to test .kml file and it works
OS: Mac OS X 10.10.5
You are missing "Iceland.shx" file. It should be in the same archive as Iceland.shp. Just put it in the same directory. I've tried this files: http://www.eea.europa.eu/data-and-maps/data/eea-reference-grids-2/gis-files/iceland-shapefile and catch same error.
If it will not help, there some other options how to debug:
Check the mandatory .shx, .shp and .dbf files are in the same directory.
Check no other programs have the shapefile open (and locked).
Open the shapefile in QGIS/ArcGIS and validate geometry.
Try another shapefile
I need to add that I have the same error when I tried to run the command:
python manage.py ogrinspect /map/data/points.shp BuildingFootprints --srid=4326 --mapping --multi
And the error was generated because of a backslash before the map, so it passed when I run:
python manage.py ogrinspect map/data/points.shp BuildingFootprints --srid=4326 --mapping --multi
When I was looking for my solution I found this question, which is answered. Anyway, I wanted to post this answer for those people who came here like I do a few min ago. Maybe someone help.
python manage.py ogrinspect map/data/points.shp BuildingFootprints --srid=4326 --mapping --multi
Ensure your data is located in the folder... map/data/points.shp BuildingFootprints

UnicodeDecodeError in Django 1.6 static file server

I've recently upgraded my project environment to django 1.6. After the upgrade, I found out that all the static files in the existing projects do not work anymore, even if I create a new project following the tutorial, the static file still does not work. Can any one help me out?
Here is my project structure
-mysite
manage.py
-mysite
urls.py
views.py
settings.py
__init__.py
wsgi.py
-static
jqury.js
settings.py looks like this
INSTALLED_APPS = (
...
'django.contrib.staticfiles',
)
TEMPLATE_DIRS = (
os.path.join(BASE_DIR, 'WorldCup\\templates'),
)
TEMPLATE_CONTEXT_PROCESSORS = [
'django.core.context_processors.static',
'django.contrib.auth.context_processors.auth',
]
STATIC_URL = '/static/'
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'WorldCup\\static'),
)
print STATICFILES_DIRS
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
#'django.contrib.staticfiles.finders.DefaultStorageFinder',
)
I print the staticfiles_dirs, the result is ('G:\django_project\mysite\mysite\static',), which correctly points to my static file folder.
The html file looks like:
{% load staticfiles %}
<script type="text/javascript" src="{% static "WorldCup/jquery-1.10.2.js" %}"
<body>
hello world======{% static "WorldCup/jquery-1.10.2.js" %}=======
</body>
the page displays "hello world======/static/jquery-1.10.2.js=======", which seems correct.
But the Console complained with:
Traceback (most recent call last):
File "D:\Python27\lib\wsgiref\handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "D:\Python27\lib\site-packages\django\contrib\staticfiles\handlers.py", l
ine 68, in __call__
return super(StaticFilesHandler, self).__call__(environ, start_response)
File "D:\Python27\lib\site-packages\django\core\handlers\wsgi.py", line 206, i
n __call__
response = self.get_response(request)
File "D:\Python27\lib\site-packages\django\contrib\staticfiles\handlers.py", l
ine 58, in get_response
return self.serve(request)
File "D:\Python27\lib\site-packages\django\contrib\staticfiles\handlers.py", l
ine 51, in serve
return serve(request, self.file_path(request.path), insecure=True)
File "D:\Python27\lib\site-packages\django\contrib\staticfiles\views.py", line
41, in serve
return static.serve(request, path, document_root=document_root, **kwargs)
File "D:\Python27\lib\site-packages\django\views\static.py", line 61, in serve
content_type, encoding = mimetypes.guess_type(fullpath)
File "D:\Python27\lib\mimetypes.py", line 297, in guess_type
init()
File "D:\Python27\lib\mimetypes.py", line 358, in init
db.read_windows_registry()
File "D:\Python27\lib\mimetypes.py", line 258, in read_windows_registry
for subkeyname in enum_types(hkcr):
File "D:\Python27\lib\mimetypes.py", line 249, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 0: ordinal
not in range(128)
[01/Dec/2013 07:42:33] "GET /static/jquery-1.10.2.js HTTP/1.1" 500 59
Even if I change to the image file instead of js file, I still get those errors....
My best guess based on your error message is that you have non-ASCII characters in your Windows registry, and something in the mimetypes module (part of the Python standard library) is choking on that. I assume either the Django statics server, or the Django staticfiles app itself, is calling mimetypes.
This is not a problem with your system per se; if I'm correct in my diagnosis it's clearly a bug in mimetypes.
Here are some options:
You could upgrade to Python 3.3 or 3.4 beta. Django 1.6 is fully compatible with Python 3 and I would be very surprised to see that this bug still exists in Py3 because one of the explicit goals of Py3 was to make unicode encode/decode errors less common.
You could downgrade back to Django 1.5.
You could use a different operating system for development work. Relatively few Python programmers do their dev work in Windows, although the number is not zero either. You are less likely to run into extremely obscure bugs if you use OS X or Linux for development work. That having been said, Windows is an officially supported platform, so if there is a problem with the interaction between CPython and Windows, then that is a CPython bug.
You could try to find exactly what change between 1.5 and 1.6 caused this problem, but that is likely a fool's errand and, assuming it's not an outright bug in Django, will at best result in you having to maintain a fork with your patch for your personal use (not recommended).
You could try to find the offending line in your registry and scrub it, but I don't know how you would go about doing this. You may have a combination of locale settings, the localized version of Windows, non-English installed software or custom MIME types that is causing this problem. Since it seems likely that it is a bug in the mimetypes library and not the fault of your system, you could easily find that the problems in your registry are simply a consequence of some normal function of your computer.
If the problem is with the Django statics server in a way that wouldn't affect your application in production (where the Django statics server, which is only suitable for development, would presumably be disabled) then you could disable the Django statics server and use some other statics serving solution for development. This is inconvenient but would probably work, as long as the problem is with the Django statics server only. To test, set DEBUG to False or otherwise turn off the statics server and run your application -- if it still crashes, then it's not a Django statics server issue; otherwise, it likely is.
Finally, you could try to track down the actual bug and fix it. This is a complex process -- you would need to isolate the bug, see whether it needs to be fixed in the CPython standard library or in Django (I suspect the former), file a report with either of those projects, then write a patch and submit it or get someone else to do likewise. Until it is submitted and accepted, you could theoretically maintain a fork of CPython with the fix applied, but that is a maintenance nightmare.
There is still the possibility that I am outright wrong about the cause and it's not a bug in mimetypes -- in that case you might have more options.
very unfortunetly I have this problem as well. I was using python 3.3, everything works fine. But since my production side is running python 2.7, so I created an a virtualenv with python 2.7.6, and then installed South, debug toolbar, which is the same as my Python3.3 environment. But it just simply not work with python2.7 virtualenv. My Django version is 1.6. So my simple way around this is to switch back to my python3.3.
For more details, my guess would be something related models/database goes wrong. I recall that after a database migration in my virtualenv python2.7 version, this problem appears, while before that everything goes well.
So I would suggest you to run a virtualenv with Python3.3, and debug for the old problem whenever you have more time. Anyway, to save time first. I find Django is really a nice framework, but Python with its multiple packages just have too much buggy stuff that you just do not want to waste your time to go over every one of them.
I have same problem.
Today I have access only to machine, with windows XP.
I've debugged this error and found, that key in windows registry has bad value.
In my case it was: 'BDATuner.���������'.
I hadn't enouth time to edit registry so I temporary add exception to File "D:\Python27\lib\mimetypes.py", line 249, in enum_types
try:
ctype = ctype.encode(default_encoding) # omit in 3.x!
except UnicodeEncodeError, UnicodeDecodeError:
pass
else:
yield ctype
I solved my problems.
I know that patching django code that way is bad, so it's only workaround.
The correct way is to get rid of bad registry values (regedit - next search for this values).