Installing pyproj in Anaconda python (Ubuntu) - python-2.7

I am trying to install pyproj on a remote server running anaconda python 2.7.15 on Ubuntu 16.04. After running conda install -c conda-forge pyproj (first listed option at https://anaconda.org/conda-forge/pyproj), the package installs successfully (pyproj 2.0.2). However, upon running my program I get the following traceback:
Traceback (most recent call last):
File "run_qc_worker.py", line 1288, in <module>
GC.make_metadata_caches() # For percentiles check
File "/home/ubuntu/qc-mem/qc/lib/global_class.py", line 384, in make_metadata_caches
mercator_arr = self.proj_arr(lat_lon_arr) # project to x/y
File "/home/ubuntu/qc-mem/qc/lib/global_class.py", line 421, in proj_arr
inproj = Proj(init='epsg:4326') # WGS 84
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/pyproj/__init__.py", line 322, in __init__
self.crs = CRS.from_user_input(projparams if projparams is not None else kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/pyproj/crs.py", line 224, in from_user_input
return cls(**value)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/pyproj/crs.py", line 146, in __init__
super(CRS, self).__init__(projstring)
File "pyproj/_crs.pyx", line 307, in pyproj._crs._CRS.__init__
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/pyproj/datadir.py", line 62, in get_data_dir
"Valid PROJ.4 data directory not found."
pyproj.exceptions.DataDirError: Valid PROJ.4 data directory not found.Either set the path using the environmental variable PROJ_LIB or with `pyproj.datadir.set_data_dir`.
In my python script, I import with from pyproj import Proj, transform, and the relevant lines of code are:
inproj = Proj(init='epsg:4326') # WGS 84
outproj = Proj(init='epsg:3857') # WGS 84 / Pseudo-Mercator
Apparently, pyproj is unable to find a data directory. Searching the directory at
~/anaconda2/lib/python2.7/site-packages/pyproj, I confirmed there is no data directory. I have another server with an installation of pyproj on anaconda3 that does contain a data directory (with an epsg file). I copied this directory to the anaconda2 server, and declared the path using the environment variable as instructed in the traceback:
export PROJ_LIB=~/anaconda2/lib/python2.7/site-packages/pyproj/data
I still get the same error traceback.
I cannot find a data directory on the github repo for pyproj (https://github.com/pyproj4/pyproj/).
How can I resolve this error?

I was able to resolve this by installing a prior version of pyproj. I replaced the pyproj-2.0.2 installation with pyproj-1.9.6, by running conda install -c conda-forge/label/gcc7 pyproj. (proj4 was also downgraded from 6.0.0 to 5.2.0 as part of this install). This is the second available conda install option listed at https://anaconda.org/conda-forge/pyproj.
As described above, I also manually copied (scp) an existing data directory from an anaconda3 install on another server, and then exported the PROJ_LIB environment variable with the path to the data directory.
Note that when running pyproj-1.9.6 before exporting the path to data, the error traceback instead indicates RuntimeError: 'no arguments in initialization list'. This issue is well-described by other users here: https://github.com/pyproj4/pyproj/issues/134.

The current version of pyproj 2.2.0 works for me:
(base) $ conda create -c conda-forge -n proj27 python=2.7 pyproj
...
proj4 conda-forge/linux-64::proj4-6.1.0-he751ad9_2
pyproj conda-forge/linux-64::pyproj-2.2.0-py27hc44880f_0
...
(base) snowal#snowal-lx2:~$ conda activate proj27
(proj27) snowal#snowal-lx2:~$ python
Python 2.7.15 | packaged by conda-forge | (default, Feb 28 2019, 04:00:11)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyproj import Proj, transform
>>> inproj = Proj(init='epsg:4326') # WGS 84
>>> outproj = Proj(init='epsg:3857') # WGS 84 / Pseudo-Mercator
>>> import pyproj
>>> pyproj.__version__
'2.2.0'

Related

How to use an already built Caffe when running py-faster-rcnn?

I'm trying to build and run py-faster-rcnn model on my Ubuntu 16.04.
However, when I run ./tools/demo.py (as stated in the installation guide), I get the following error:
➜ py-faster-rcnn git:(master) ✗ ./tools/demo.py
Traceback (most recent call last):
File "./tools/demo.py", line 18, in <module>
from fast_rcnn.test import im_detect
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 16, in <module>
import caffe
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/pycaffe.py", line 13, in <module>
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: No module named _caffe
Before attempting to install py-faster-rcnn, I've installed Caffe in my ~/code/caffe folder and it seems to work fine:
➜ ~ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> print caffe.__version__
1.0.0-rc3
So, if I can import caffe module in python environment, I assume I've installed it successfully.
Here're the commands I've used (from the official guide):
sudo make all
sudo make test
sudo make runtest
sudo make pycaffe
sudo make distribute
Then I've cloned the py-faster-rcnn repository in my ~/WEB/DeepLearning folder.
After that I've followed the installation instructions from the repo:
Clone the repo
cd $FRCN_ROOT/lib && make
cd $FRCN_ROOT/caffe-fast-rcnn
make -j8 && make pycaffe (I didn't run this)
cd $FRCN_ROOT && ./data/scripts/fetch_faster_rcnn_models.sh
cd $FRCN_ROOT && ./tools/demo.py
So, step 4 in the installation guide says I have to build caffe and pycaffe in $FRCN_ROOT/caffe-fast-rcnn folder. The contents of caffe-fast-rcnn folder seem to be identical with the original caffe repository from which I've built Caffe.
So, it seems that I don't need to build caffe again, right? When trying to run the demo, I've skipped the step of building caffe and got the error stated above.
After googling for a while, I've found out that my issue is connected with path environment variables, so below are my path variables in .bashrc:
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:~/code/caffe/distribute/lib:$LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=/usr/include/python2.7
export PYTHONPATH=~/code/caffe/python:$PYTHONPATH
Am I doing something wrong and I have to change my path variables somehow?
Or I really need to build caffe again, but in a caffe-fast-rcnn folder?
And what about this distribute folder I've generated in ~/code/caffe/distribute by running sudo make distribute? Is it of any use? If so, how should I use it? The official documentation is not very clear about it.
A simple, clear and detailed explanation on how to use an already built Caffe framework with other projects like Faster-RCNN would be really helpful.
I struggled with this for a while and then got it working as below.
First, check PYTHONPATH env variable. It should have python path like, for eg. based on your python version and installation
PYTHONPATH = /usr/lib/python2.7
If its empty, you can set it with python path captured in your python shell. To check python path information, open python shell and type below
>>import sys
>>for p in sys.path
... print(p)
It will list you all python path info, for eg
...
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
...
If you have installed caffe already and want it to configure to be used by python, you just need to update your PYTHONPATH env variable by adding path to your /caffe-installation-path/python folder to it, like
export PYTHONPATH = /home/mypc/caffe-master/python:$PYTHONPATH
Note:- You don't need to rebuild caffe but configure caffe and python in PYTHONPATH env variable correctly.

AttributeError with pip install matplotlib

I am attempting to install matplotlib with pip inside a virtualenv on Ubuntu Linux. I already have a bunch of packages installed successfully - this is the one that's failing. As the output indicates, This is python 2.7.3 and all of the dependencies are already present.
Is this specific to v1.4.3 of matplotlib? Should I perhaps install an earlier version? I've even manually installed mock (v1.3.0), since the install seems to fail on trying to import it, but that didn't help either.
Here is the output:
Collecting matplotlib
Using cached matplotlib-1.4.3.tar.gz
Complete output from command python setup.py egg_info:
============================================================================
Edit setup.cfg to change the build options
BUILDING MATPLOTLIB
matplotlib: yes [1.4.3]
python: yes [2.7.3 (default, Jun 22 2015, 19:33:41) [GCC
4.6.3]]
platform: yes [linux2]
REQUIRED DEPENDENCIES AND EXTENSIONS
numpy: yes [version 1.9.2]
six: yes [using six version 1.9.0]
dateutil: yes [using dateutil version 2.4.2]
pytz: yes [using pytz version 2015.4]
tornado: yes [tornado was not found. It is required for the
WebAgg backend. pip/easy_install may attempt to
install it after matplotlib.]
pyparsing: yes [using pyparsing version 2.0.3]
pycxx: yes [Couldn't import. Using local copy.]
libagg: yes [pkg-config information for 'libagg' could not
be found. Using local copy.]
freetype: yes [version 2.4.8]
png: yes [version 1.2.46]
qhull: yes [pkg-config information for 'qhull' could not be
found. Using local copy.]
OPTIONAL SUBPACKAGES
sample_data: yes [installing]
toolkits: yes [installing]
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "/tmp/pip-build-mHyI6G/matplotlib/setup.py", line 155, in <module>
result = package.check()
File "setupext.py", line 666, in check
import mock
File "/home/awarnock/PyProjects/OMTO3dIMG/local/lib/python2.7/site-packages/mock/__init__.py", line 2, in <module>
import mock.mock as _mock
File "/home/awarnock/PyProjects/OMTO3dIMG/local/lib/python2.7/site-packages/mock/mock.py", line 71, in <module>
_v = VersionInfo('mock').semantic_version()
AttributeError: 'VersionInfo' object has no attribute 'semantic_version'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip- build-mHyI6G/matplotlib
Thanks in advance for any suggestions.
You can try with:
pip install oslo.utils
Or:
pip install testrepository
pip install oslo.utils
You can also install it with packages:
Debian / Ubuntu:
apt-get install python-matplotlib
Redhat / CentOS:
yum install python-matplotlib

python-magic WindowsError: access violation writing 0x00000000

I installed python-magic (0.4.6) on my Win 7 64bit using pip.
I then installed cygwin 1.7.33-2 to provide the needed dlls and created a copy of cygmagic-1.dll named magic1.dll (see
When I run the Python 2.7.6 32bit shell, the "import magic" works fine.
However, a
magic.from_file('c:\user\username\sample.txt')
gives me a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\Python27\lib\site-packages\magic.py", line 119, in from_file
m = _get_magic_type(mime)
File "c:\Python27\lib\site-packages\magic.py", line 107, in _get_magic_type
i = instances.__dict__[mime] = Magic(mime=mime)
File "c:\Python27\lib\site-packages\magic.py", line 55, in __init__
self.cookie = magic_open(flags)
WindowsError: exception: access violation writing 0x00000000
Any ideas what causes the this error and how I can fix it?
Thank you for your help!
The GnuWin32 file package has a 32-bit magic1.dll, along with its dependencies regex2.dll and zlib1.dll. I know from testing that this version works with python-magic. Here's an overview of the steps that I took to test this in 32-bit Python 3.3.
Extract the files to GnuWin32's installation directory:
C:\Temp>set "GNU=C:\Program Files (x86)\GNU"
C:\Temp>7z x -y -o"%GNU%" file-5.03-bin.zip > nul
C:\Temp>7z x -y -o"%GNU%" file-5.03-dep.zip > nul
Set up the environment to find the DLLs and magic file:
C:\Temp>set PATH=%PATH%;%GNU%\bin
C:\Temp>set MAGIC=%GNU%\share\misc\magic
Install python-magic:
C:\Temp>py -3.3-32 -m pip install python-magic
Collecting python-magic
Downloading python-magic-0.4.6.tar.gz
Installing collected packages: python-magic
Running setup.py install for python-magic
Successfully installed python-magic-0.4.6
Verify that it works:
C:\Temp>py -3.3-32
Python 3.3.5 (v3.3.5:62cf4e77f785, Mar 9 2014, 10:37:12)
[MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, magic
>>> magic.from_file(sys.executable)
b'PE32 executable for MS Windows (console) Intel 80386 32-bit'

Differences in sys.path when python2.7 is invoked normally or via subprocess

Question
Why is it that python2.7 when called using a subprocess via python3 does
not have the same sys.path as python2.7 called normally? Specifically,
python2.7 via subprocess does not have the "/path/to/site-packages/"
directory in sys.path.
Context
I'd like to use fabric to deploy a Django app I'm writing. My problem is that
I've written the app in python3, but fabric doesn't have explicit python3
support yet. My workaround, until fabric is fully compatible with python3,
is to call the fab script using subprocess.
For some reason when I call python2.7 using subprocess via python3, I
don't have access to any modules in site-packages.
python2.7 checks
I've got python2.7 and fabric==1.10.0 installed via Enthought.
$ which python
/Users/.../Library/Enthought/Canopy_32bit/User/bin/python
$ python --version
Python 2.7.6 -- 32-bit
$ which fab
/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab
$ fab --version
Fabric 1.10.0
Paramiko 1.15.1
subprocess checks
I have no problem calling fab from within python2.7 using subprocess.
$ python
Enthought Canopy Python 2.7.6 | 32-bit | (default, Apr 11 2014, 12:06:39)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('fab --version', shell=True)
'Fabric 1.10.0\nParamiko 1.15.1\n'
I also have no problem calling python2.7 from within python3 using subprocess.
$ python3
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('which python', shell=True)
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/python\n'
>>> subprocess.check_output('python --version', shell=True)
Python 2.7.6 -- 32-bit
b''
DistributionNotFound: Fabric==1.10.0
However, even though my subprocess of python2.7 can "find" the fab script, I
can't call it.
# python3
>>> subprocess.check_output(['which', 'fab'])
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab\n'
>>> subprocess.check_output(['fab', '--version'])
Traceback (most recent call last):
File "/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab", line 5, in <module>
from pkg_resources import load_entry_point
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 2877, in <module>
working_set.require(__requires__)
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 698, in require
needed = self.resolve(parse_requirements(requirements))
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 596, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Fabric==1.10.0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['fab', '--version']' returned non-zero exit status 1
site-packages not in sys.path
It appears that python2.7 when called using subprocess via python3 does
not have the same sys.path as python2.7 called normally.
As expected, sys.path did not have the Enthought "site-packages" directory,
which contains the fabric module.
# python3
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## does not contain '/path/to/Enthought/python2.7/site-packages'
Manually add site-packages to sys.path
To confirm that it's possible: when I manually add the correct
"site-packages" directory, I can successfully import fabric.
# python3
>>> subprocess.check_output('python -c\
"import sys; sys.path.append(\'/path/to/Enthought/site-packages\');\
from fabric import version; print version.get_version()"',\
shell = True)
b'1.10.0\n'
Other options?
There's got to be a better way to make sure that python2.7, when
invoked via subprocess from python3, has the same sys.path as python2.7
invoked normally. Can someone more familiar with subprocess weigh in?
Additional thoughts
It's really interesting that python2.7 can spawn another python2.7
via subprocess and that subprocess has the correct site-packages dir
in sys.path.
$ python
>>> import subprocess
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## contains "/path/to/Enthought/python2.7/site-packages"
I also compared the sys.path's from python3, python3 subprocessed by python3,
and python3 subprocessed by python2.7, and was a bit surprised to find that
all three resulted in the same sys.path.
subprocess supports an env parameter that, if given, will be the environment for the called command -- so make a copy of it, remove any troublesome variables, and pass that copy to subprocess:
my_env = os.environ.copy()
del my_env['__PYENV_LAUNCHER__']
subprocess.check_output(..., env=my_env)

Error Psycopg2 Mac OS X Mountain Lion Symbol not found: _GSS_C_NT_HOSTBASED_SERVICE

I'm getting a psycopg2 error after updating to Mac OS X Mountain Lion, previously it was working fine.
This is the error if I try to import the module:
Python 2.7.3 (default, Sep 27 2012, 21:57:16)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.65))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psycopg2/__init__.py", line 67, in <module>
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: dlopen(/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Symbol not found: _GSS_C_NT_HOSTBASED_SERVICE
Referenced from: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psycopg2/_psycopg.so
Expected in: flat namespace
in /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psycopg2/_psycopg.so
I've used brew to install python and psycopg2.
Any idea would be helpful.
Thanks
Alex
Try brewing postgresql too. The error is probably in the way the libpq (the PostgreSQL client-side library, of which psycopg is a wrapper) is linked.
Mixing native and brew app is not that good... I would rather uninstall psycopg2 and python from brew and install the "Command Line Tools", like explained in pip install fails with /usr/bin/clang: No such file or directory. Then retry pip install, it will work, at least it did the job for me !