Pytest does not see files in folders in Django - django

I have a following question:
I have a test function written in Pytest for Django project:
#pytest.mark.django_db
def test_class():
path = Path(r'ascertain\tests\csv')
handler = DatabaseCSVUpload(path, delimiter=',')
handler()
ascertain\tests\test_upload_csv.py:19 (TestUploadCSVtoDatabaseNegative.test_class)
Traceback (most recent call last):
File "C:\Users\hardcase1\PycharmProjects\telephone_numbers\ascertain\tests\test_upload_csv.py", line 27, in test_class
handler()
File "C:\Users\hardcase1\PycharmProjects\telephone_numbers\ascertain\handle_csv.py", line 129, in __call__
for file in self.get_csv_files():
File "C:\Users\hardcase1\PycharmProjects\telephone_numbers\ascertain\handle_csv.py", line 48, in get_csv_files
raise EmptyFolder()
telephone_numbers.custom_exceptions.EmptyFolder:
It basically expects *.csv files be inside directory ‘path’ and handle them to upload to DB.
Problem is that Pytest can’t see any files in this directory as well as in other directories. I have tried multiple times with different folders.
In fact file is there:
list(Path(r'ascertain\tests\csv').glob('*.csv'))
[WindowsPath('ascertain/tests/csv/test.csv')]
Same test function written in unitests works correctly:
from rest_framework.test import APITestCase
class TestCase(APITestCase):
def test_open(self):
path = Path(r'ascertain\tests\csv')
handler = DatabaseCSVUpload(path, delimiter=',')
handler()
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...
Process finished with exit code 0
Question is -what I need to do to make Pytest see this file?
Thank you!

I'm not sure this problem relates to Pytest vs Unittest, rather I think it has to do with where you executing these files from. I think you can resolve this by using an absolute path instead of the relative path you included in the example.
An absolute path will start at your systems root directory instead of starting at the directory you are executing the file from. I'm not a Windows user, but from what I can see online the path to this data should look something like this C:/users/path/to/data.

Related

neovim: no module named __future__

When I try to open certain python files in neovim, I get an error:
"pool.py" 667L, 25276C
function provider#python#Call[9]..remote#host#Require[10]..provider#pythonx#Require, line 15
Vim(if):ch 1 was closed by the client
Traceback (most recent call last):
File "/home/user/.pyenv/versions/neovim2/lib/python2.7/site.py", line 67, in <module>
import os
File "./os.py", line 44, in <module>
from __future__ import absolute_import
ImportError: No module named __future__
Failed to load python host. You can try to see what happened by starting nvim with $NVIM_PYTHON_LOG_FILE set and opening the generated log file. Also
, the host stderr is available in messages.
Press ENTER or type command to continue
This happens any time I open a python file in a directory that contains an os.py or os.pyc file. It looks like neovim is trying to import the local os.py file instead of the one in the virtualenv.
What can I do about this?
EDIT: turns out it's not when I open a file in the same directory as an os.py file, it's when I open a file anywhere while the current working directory has an os.py file. Basically, it looks like python is checking the local directory for imports before checking the python libs.
I figured it out. The problem was with my $PYTHONPATH. I had in my .bashrc file this:
export PYTHONPATH="$PYTHONPATH:~/.local/lib/python"
The problem was that, when that line is executed, $PYTHONPATH is empty, leading to the string starting with a :. I'm not 100% sure why, but that resulted in python checking the local directory for a module BEFORE checking the python libraries.
I changed it to
if [ -z "$PYTHONPATH" ]; then
export PYTHONPATH="~/.local/lib/python"
else
export PYTHONPATH="$PYTHONPATH:~/.local/lib/python"
fi
And now it works.

Getting no module named _internals error while using Bloomberg API

I am currently using Python 2.7 and my OS is Windows 7. While attempting to use the Bloomberg API I am getting this error:
Traceback (most recent call last):
File "datagrab.py", line 1, in <module>
import blpapi, time, json
File "C:\Python27\lib\blpapi\__init__.py", line 5, in <module>
from .internals import CorrelationId
File "C:\Python27\lib\blpapi\internals.py", line 50, in <module>
_internals = swig_import_helper()
File "C:\Python27\lib\blpapi\internals.py", line 42, in swig_import_helper
import _internals
ImportError: No module named _internals
I have set my path variable to point to blpapi3_64.dll and also updated my bloomberg terminal. I have also moved the local blpapi API to a different directory but still the problem exists.
I am kind of new to this API in general. So can someone please guide me?
Thank you in advance!
From your question is sounds like maybe you have tried this, but just outlining one possible solution from the README in the Python Supported Release release available here.
Note that many Python installations add the current directory to the
module search path. If the Python interpreter is invoked from the
installer directory, such a configuration will attempt to use the
(incomplete) local blpapi directory as a module. If the above
import line fails with the message Import Error: No module named
_internals, move to a different directory before invoking python.
I know this question is a bit stale, but in case people end up here like me. Do you have the C++ version of blpapi? it is a requirement for the python api as mentioned here: https://www.bloomberg.com/professional/support/api-library/
so download the C++ zip installer, extract somewhere, and then add it as an environment variable so that the python api can find it:
Environment variable name: BLPAPI_ROOT
Value: C:\blp\blpapi_cpp_3.8.18.1 (THIS IS WHERE MINE IS INSTALLED, YOUR VALUE HERE MAY BE DIFFERENT)
Hope that helps!

Windows executable file generated by PyGame + PyInstaller gives me "No such file or directory" error

I have first tried to create my original game sample using Pygame.
I have traced the instruction written in the Web site below:
https://irwinkwan.com/2013/04/29/python-executables-pyinstaller-and-a-48-hour-game-design-compo/
Specifically,
I created "myFile" class something like following and read every file (.txt, .png, .mp3, etc...) using this class
myFile Class
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import sys
class myFile():
def resource_path(self, relative):
if hasattr(sys, "_MEIPASS"):
return os.path.join(sys._MEIPASS, relative)
return os.path.join(relative)
### code where I read something ###
myfile = myFIle()
filename = myfile.resource_path(os.path.join("some dir", "somefile"
+ "extension")
I typed command below to create .spec file (myRPG.py contains main)
pyinstaller --onefile myRPG.py
I modified .spec file so exe object include Trees (I store data files in separate
directories such as data, image, and so on)
myRPG.spec file
# -*- mode: python -*-
block_cipher = None
a = Analysis(['myRPG.py'],
pathex=['C:\\mygame\\mygame'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
Tree('bgm', prefix='bgm'),
Tree('charachip', prefix='charachip'),
Tree('data', prefix='data'),
Tree('image', prefix='image'),
Tree('mapchip', prefix='mapchip'),
Tree('se', prefix='se'),
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='myRPG',
debug=False,
strip=False,
upx=True,
console=True )
I did rebuild my package using modified .spec file
pyinstaller myRPG.spec
When I execute myRPG.exe file, I got the error below
C:\mygame\mygame\dist>myRPG.exe
[Errno 2] No such file or directory:
'C:\\Users\\bggfr\\AppData\\Local\\Temp\\_MEI64~1\\item.data' IO Errorが発生しました。
Traceback (most recent call last):
File "myRPG.py", line 606, in <module>
File "myRPG.py", line 37, in __init__
File "myItemList.py", line 11, in __init__
File "myItemList.py", line 33, in load
UnboundLocalError: local variable 'fp' referenced before assignment
Failed to execute script myRPG
I believe that I properly specify the directories where data is expanded since I use the function that checks _MEIPASS but it does not work.
I have also tried to use "added_files" instead of Tree did not help for me at all. ”a.datas" may work for small number of files but I do not want to specify all the files I'm going to use, because it will be over hundreds of thousands of files.
This is a very helpful page that I used when converting my pygame program into an executable.
In short:
Here is the place where you can download pyInstaller. Click on the download button under "Installation." Wait for it to download, then run it.
Type this into the popup where it says command: venv -c -i pyi-env-name. The reason for this is stated in the page mentioned at the top.
If you have pyInstaller installed already, type pyinstaller --version to make sure. If you don't, type pip install PyInstaller. If this doesn't work, look at step 7 in the guide up top.
Make sure that you put a copy of your program in the folder that it says on the command line.
If you want a zip with the exe and all necessary files in it type pyinstaller and then the name of your program, complete with the .py extension. However, if you want an exe that doesn't need all of the extra files in the zip that you can just run by itself, type pyinstaller --onefile --windowed and then the name of your program, complete with the extension of .py.
Once you have done this, the exe will appear in a dist folder inside of the one where you placed your program in step 3.
This is how it worked for me, and I apologize if I have misinformed you. Be sure to upvote/mark as answered if this works.

GeoDjango GDAL_ERROR when uplading .shp

Try to parse a .shp in django shell:
from django.contrib.gis.gdal import DataSource
ds = DataSource('/Users/.../Downloads/Iceland.shp')
get:
GDAL_ERROR 4: Unable to open /Users/.../Downloads/Iceland.shx or /Users/.../Downloads/Iceland.SHX.
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Users/.../lib/python2.7/site-packages/django/contrib/gis/gdal/datasource.py", line 78, in __init__
raise GDALException('Could not open the datasource at "%s"' % ds_input)
GDALException: Could not open the datasource at "/Users/.../Downloads/Iceland.shp"
File exists, chmod is 755, .shx file is correct (tested in online services).
Then I try to test .kml file and it works
OS: Mac OS X 10.10.5
You are missing "Iceland.shx" file. It should be in the same archive as Iceland.shp. Just put it in the same directory. I've tried this files: http://www.eea.europa.eu/data-and-maps/data/eea-reference-grids-2/gis-files/iceland-shapefile and catch same error.
If it will not help, there some other options how to debug:
Check the mandatory .shx, .shp and .dbf files are in the same directory.
Check no other programs have the shapefile open (and locked).
Open the shapefile in QGIS/ArcGIS and validate geometry.
Try another shapefile
I need to add that I have the same error when I tried to run the command:
python manage.py ogrinspect /map/data/points.shp BuildingFootprints --srid=4326 --mapping --multi
And the error was generated because of a backslash before the map, so it passed when I run:
python manage.py ogrinspect map/data/points.shp BuildingFootprints --srid=4326 --mapping --multi
When I was looking for my solution I found this question, which is answered. Anyway, I wanted to post this answer for those people who came here like I do a few min ago. Maybe someone help.
python manage.py ogrinspect map/data/points.shp BuildingFootprints --srid=4326 --mapping --multi
Ensure your data is located in the folder... map/data/points.shp BuildingFootprints

wkhtmltopdf subprocess error using django wrapper, but not in shell

I'm using wkhtmltopdf together with the django-wkhtmltopdf wrapper to create a .pdf of a template.
I'm using the example from the django-wkhtmltopdf documentation (though I eventually want more than just a static template):
url(r'^pipeline/snapshot/$', PDFTemplateView.as_view(
template_name='pdf/pipeline_snapshot.html',
filename='my_pdf.pdf'), name='pdf'),
And I'm getting the error:
Traceback:
File "/home/pluscitizen/webapps/odin/lib/python2.7/django/core/handlers/base.py" in get_response
140. response = response.render()
File "/home/pluscitizen/webapps/odin/lib/python2.7/django/template/response.py" in render
105. self.content = self.rendered_content
File "/home/pluscitizen/webapps/odin/lib/python2.7/django_wkhtmltopdf-1.2.2-py2.7.egg/wkhtmltopdf/views.py" in rendered_content
144. footer_filename=footer_filename)
File "/home/pluscitizen/webapps/odin/lib/python2.7/django_wkhtmltopdf-1.2.2-py2.7.egg/wkhtmltopdf/views.py" in convert_to_pdf
103. return wkhtmltopdf(pages=[filename], **cmd_options)
File "/home/pluscitizen/webapps/odin/lib/python2.7/django_wkhtmltopdf-1.2.2-py2.7.egg/wkhtmltopdf/utils.py" in wkhtmltopdf
92. return check_output(ck_args, **ck_kwargs)
File "/usr/local/lib/python2.7/subprocess.py" in check_output
575. raise CalledProcessError(retcode, cmd, output=output)
Exception Type: CalledProcessError at /pipeline/snapshot/
Exception Value: Command '['wkhtmltopdf', '--encoding', u'utf8', '--quiet', '/tmp/wkhtmltopdfVaAKrX.html', '-']' returned non-zero exit status 127
However, when I run the same command, with the same file, from the Django shell
>>> subprocess.check_output(['wkhtmltopdf', '--encoding', u'utf8', '--quiet', '/tmp/wkhtmltopdfSGFfYh.html', '-'])
everything runs fine. Ditto for:
>>> wkhtmltopdf(['/tmp/wkhtmltopdfSGFfYh.html'], **{})
So, figuring that the difference must be in the whole shell situation, I've tried adding a shell=True to the call to subprocess within django-wkhtmltopdf (I know it's a security problem), but no luck there. Probably because I have no idea what's really going on.
I saw somewhere saying that the problem might have to do with PATHs not being set up properly, but then I don't understand why Django's shell is not having issues with this.
This entire process has been incredibly taxing, and now that I'm so close, I'm finally turning to SO for an answer.
EDIT: I have tried running the subprocess directly in a view, and it returns the same error, which I guess means that the shell and the server aren't necessarily both running from the same environment?
FIXED: Solved, and the answer by sparks led me on the way. I decided to look at the logs (duh) output by the django app on the server, and noticed that before the traceback, I'm getting the actual output of the command:
wkhtmltopdf: error while loading shared libraries: libwkhtmltox.so.0: cannot open shared object file: No such file or directory
Which seems to imply that it's not finding the libraries I have in /home/user/lib. When I explicitly added these to the WKHTMLTOPDF_ENV variable in settings.py, everything worked swimmingly.
exit status 127
This means that the command you're trying to run can't be found. It is an error in the environment of your process.
I would suggest looking at the contents of sys.path in python and comparing that to the PATH environment variable, or compared to sys.path in the django console.
More specifically, add this to settings.py:
WKHTMLTOPDF_ENV = {'FONTCONFIG_PATH': '/etc/fonts'}