PYTHONPATH affecting BASH search path - python-2.7

Using Anaconda 2.5.0 (python 2.7) on a relatively clean Ubuntu 15.04 desktop system, I have the following unusual behavior:
If I package my application with cx_Freeze 4.3.4 and add the bin directory to the PYTHONPATH environment variable:
export PYTHONPATH=~/frozen-app/bin
Then, when I mistype a bash command:
sl
Instead of the expected:
The program 'sl' is currently not installed. You can install it by typing:
sudo apt-get install sl
I get a python stack trace:
Traceback (most recent call last):
File "/usr/lib/command-not-found", line 23, in <module>
import gettext
File "/usr/lib/python3.4/gettext.py", line 49, in <module>
import locale, copy, io, os, re, struct, sys
File "/usr/lib/python3.4/locale.py", line 18, in <module>
import collections
File "/usr/lib/python3.4/collections/__init__.py", line 11, in <module>
from operator import itemgetter as _itemgetter, eq as _eq
ImportError: dynamic module does not define init function (PyInit_operator)
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 62, in apport_excepthook
import re, traceback
File "/usr/lib/python3.4/traceback.py", line 3, in <module>
import linecache
File "/usr/lib/python3.4/linecache.py", line 10, in <module>
import tokenize
File "/usr/lib/python3.4/tokenize.py", line 29, in <module>
import collections
File "/usr/lib/python3.4/collections/__init__.py", line 11, in <module>
from operator import itemgetter as _itemgetter, eq as _eq
ImportError: dynamic module does not define init function (PyInit_operator)
Original exception was:
Traceback (most recent call last):
File "/usr/lib/command-not-found", line 23, in <module>
import gettext
File "/usr/lib/python3.4/gettext.py", line 49, in <module>
import locale, copy, io, os, re, struct, sys
File "/usr/lib/python3.4/locale.py", line 18, in <module>
import collections
File "/usr/lib/python3.4/collections/__init__.py", line 11, in <module>
from operator import itemgetter as _itemgetter, eq as _eq
ImportError: dynamic module does not define init function (PyInit_operator)
It seems that the "command-not-found" command gets confused by my PYTHONPATH environment, but I can't figure out exactly why. I don't think any of the mentioned modules are defined in my application (I don't define an "re" module in my app for example).
If I run "strace sl", then I just get "can't stat sl". Is there another way to figure out exactly which file in my application is causing this?

Using the suggestion by Charles Duffy above, and looking more closely at the python stack trace, I was able to reproduce the error with:
python3.4 -m operator
Just this replicates the stack trace. Since my PYTHONPATH points to a bin directory from from an Anaconda2.5 (python 2.7), it seems that the python3.4 tries to load operator.so from the the anaconda distribution which has been frozen in my bin directory. This particular import fails. Deleting operator.so from my bin fixes this issue. It seems that using the system python3.4 (which is used by command-not-found) and the Anaconda 2.7 combined with explicitly setting PYTHONPATH can cause trouble.

Related

Theano ImportError: cannot import name inplace_increment

I've installed Theano as such:
(python2) bash-3.2$ sudo pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git
I'm using conda with Python 2.7. The python2 in parentheses is my conda environment. I'm trying to run the code from iGAN: https://github.com/junyanz/iGAN. I'm on OS X 10.11.6.
The error I get is ImportError: cannot import name inplace_increment. I've tried manually removing the Theano cache (as described in https://github.com/Theano/Theano/issues/5564) and the same error still arises. (I can't use theano-cache purge because it requires installing a version of scipy that clashes with the version required by OpenCV.) I've also tried uninstalling and reinstalling Theano via pip and still get the same error. Any suggestions?
Here's the context:
(python2) bash-3.2$ python iGAN_main.py
//anaconda/envs/python2/lib/python2.7/site-packages/numpy/lib/utils.py:254: FutureWarning: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a record
array. This code will likely break in the next numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
ai = a.__array_interface__
Traceback (most recent call last):
File "iGAN_main.py", line 9, in <module>
import constrained_opt
File "/Users/.../Desktop/Code/iGAN/constrained_opt.py", line 3, in <module>
from lib.rng import np_rng
File "/Users/.../Desktop/Code/iGAN/lib/rng.py", line 2, in <module>
from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
File "//anaconda/envs/python2/lib/python2.7/site-packages/theano/__init__.py", line 102, in <module>
from theano.scan_module import (scan, map, reduce, foldl, foldr, clone,
File "//anaconda/envs/python2/lib/python2.7/site-packages/theano/scan_module/__init__.py", line 41, in <module>
from theano.scan_module import scan_opt
File "//anaconda/envs/python2/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 60, in <module>
from theano import tensor, scalar
File "//anaconda/envs/python2/lib/python2.7/site-packages/theano/tensor/__init__.py", line 9, in <module>
from theano.tensor.subtensor import *
File "//anaconda/envs/python2/lib/python2.7/site-packages/theano/tensor/subtensor.py", line 27, in <module>
from cutils_ext.cutils_ext import inplace_increment
ImportError: cannot import name inplace_increment
I had the same problem.
Deleting the cache manually worked.
rm -rf ~/.theano
If that doesn't help try installing from source.
python setup.py install

Python Scipy.stats operations do not work when copied to another machine (where there is no python installed) with the source folder

I am using Scipy.stats to perform distribution fits on my data. It works perfectly on my machine. But when I package my script with Python source folder, python27.dll and try to run it on another machine (where python is not installed and I do not want to also) by invoking my script like below
Path/to/Python.exe Path/to/script.py
I get the follwing error
Traceback (most recent call last):
File "test2.py", line 2, in <module>
import scipy.stats
File "C:\Users\User\Desktop\Python27\lib\site-packages\scipy\stats\__init__.py
", line 344, in <module>
from .stats import *
File "C:\Users\User\Desktop\Python27\lib\site-packages\scipy\stats\stats.py",
line 173, in <module>
import scipy.special as special
File "C:\Users\User\Desktop\Python27\lib\site-packages\scipy\special\__init__.
py", line 636, in <module>
from ._ufuncs import *
ImportError: DLL load failed: The specified module could not be found.
My script is
import sys
import scipy.stats
import numpy
dataarray= [2.45, 3.67, 1.90, 2.56, 1.78, 2.67]
desc = scipy.stats.describe(dataarray)
print desc
This occurs only on Scipy import and scipy operations, all other imports and operations work fine. My purpose is to run this script in different machines without having to actually install python

ImportError from greenlet with Python 2.7.3 in virtualenv

I installed locustio
pip install locustio
Once installed, I tried running
locust --help
But I get the following error:
Traceback (most recent call last):
File "/home/vagrant/.virtualenvs/chalktalk-legacy/bin/locust", line 7, in <module>
from locust.main import main
File "/home/vagrant/.virtualenvs/chalktalk-legacy/local/lib/python2.7/site-packages/locust/__init__.py", line 1, in <module>
from core import HttpLocust, Locust, TaskSet, task
File "/home/vagrant/.virtualenvs/chalktalk-legacy/local/lib/python2.7/site-packages/locust/core.py", line 1, in <module>
import gevent
File "/home/vagrant/.virtualenvs/chalktalk-legacy/local/lib/python2.7/site-packages/gevent/__init__.py", line 49, in <module>
from gevent.hub import get_hub, iwait, wait, PYPY
File "/home/vagrant/.virtualenvs/chalktalk-legacy/local/lib/python2.7/site-packages/gevent/hub.py", line 11, in <module>
from greenlet import greenlet, getcurrent, GreenletExit
ImportError: /home/vagrant/.virtualenvs/chalktalk-legacy/local/lib/python2.7/site-packages/greenlet.so: undefined symbol: _PyTrash_thread_deposit_object
Is the solution to rebuild gevent? If so, how does one go about rebuilding greenlet in debian virtualenv with python 2.7.3? If not, how do I solve this issue without upgrading python?

Failing to import Shapely / GEOS in Python on OSX

I am running a brewed Python 2.7.11 on OS X El Capitan 10.11.3. I fail to import Shapely / GEOS:
Failed `CDLL(/usr/local/lib/libgeos_c.dylib)`
Failed `CDLL(/Library/Frameworks/GEOS.framework/Versions/Current/GEOS)`
Failed `CDLL(/opt/local/lib/libgeos_c.dylib)`
Traceback (most recent call last):
File "example.py", line 2, in <module>
from shapely.geometry import Point
File "/Library/Frameworks/GEOS.framework/Versions/3/Python/2.7/site-packages/shapely/geometry/__init__.py", line 4, in <module>
from .base import CAP_STYLE, JOIN_STYLE
File "/Library/Frameworks/GEOS.framework/Versions/3/Python/2.7/site-packages/shapely/geometry/base.py", line 9, in <module>
from shapely.coords import CoordinateSequence
File "/Library/Frameworks/GEOS.framework/Versions/3/Python/2.7/site-packages/shapely/coords.py", line 8, in <module>
from shapely.geos import lgeos
File "/Library/Frameworks/GEOS.framework/Versions/3/Python/2.7/site-packages/shapely/geos.py", line 92, in <module>
mode=(DEFAULT_MODE | 16))
File "/Library/Frameworks/GEOS.framework/Versions/3/Python/2.7/site-packages/shapely/geos.py", line 61, in load_dll
libname, fallbacks or []))
OSError: Could not find lib geos_c or load any of its variants ['/Library/Frameworks/GEOS.framework/Versions/Current/GEOS', '/opt/local/lib/libgeos_c.dylib'].
This error is strikingly similar to those reported here:
Python can't import shapely and
OSError geos_c could not be found when Installing Shapely. After implementing all suggestions that "did the trick" for others, I continue to get this error.
I installed geos with brew install geos today. It installed in /usr/local/Cellar/geos/3.5.0.
If I run find / -name "libgeos_c.dylib", I find three versions:
/Library/Frameworks/GEOS.framework/Versions/3/unix/lib/libgeos_c.dylib
/usr/local/Cellar/geos/3.5.0/lib/libgeos_c.dylib
/usr/local/lib/libgeos_c.dylib
Note that the last line is exactly the same path as in the first line of the error. So Python fails to import an existing file? I know it is an alias (to the file in Cellar), but would be surprised if that's the reason.
Also, $DYLD_LIBRARY_PATH and $GEOS_LIBRARY_PATH are empty at startup, but setting them to any of the following does not resolve the problem:
export DYLD_LIBRARY_PATH=/usr/local/lib
export GEOS_LIBRARY_PATH=/usr/local/Cellar/geos/3.5.0
export GEOS_LIBRARY_PATH=/usr/local/Cellar/geos/3.5.0/lib
Any insight will be tremendously welcome! This has taken me more than half of the day already...

conda update --all = "ImportError: DLL load failed: %1 is not a valid Win32 application." when trying to import matplotlib.pyplot

I use the Python distribution. Python 2.7 x64 with Windows 7 SP1 x64 Ultimate.
After a conda update --all, whenever I try to import matplotlib.pyplot I get ImportError: DLL load failed: %1 is not a valid Win32 application.. Why?
Full error stack:
Traceback (most recent call last):
File "C:\svn\hw4\code\test_con.py", line 1, in <module>
import matplotlib.pyplot as plt
File "C:\Anaconda\lib\site-packages\matplotlib\pyplot.py", line 109, in <module>
_backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
File "C:\Anaconda\lib\site-packages\matplotlib\backends\__init__.py", line 32, in pylab_setup
globals(),locals(),[backend_name],0)
File "C:\Anaconda\lib\site-packages\matplotlib\backends\backend_qt4agg.py", line 17, in <module>
from .backend_qt5agg import NavigationToolbar2QTAgg
File "C:\Anaconda\lib\site-packages\matplotlib\backends\backend_qt5agg.py", line 18, in <module>
from .backend_qt5 import QtCore
File "C:\Anaconda\lib\site-packages\matplotlib\backends\backend_qt5.py", line 31, in <module>
from .qt_compat import QtCore, QtGui, QtWidgets, _getSaveFileName, __version__
File "C:\Anaconda\lib\site-packages\matplotlib\backends\qt_compat.py", line 91, in <module>
from PyQt4 import QtCore, QtGui
ImportError: DLL load failed: %1 is not a valid Win32 application.
I had the same issue after running conda update anaconda. The solution that worked for me was to simply to download the latest windows installer, then uninstall and reinstall.
I suspect this is related to the pathname of your conda environment, as if you do a clean install anaconda now wants to live in C:\Users\yourname\AppData\Local\Continuum\Anaconda2, where previously it was just Anaconda (no 2). I suspect after running the upgrade scripts some things are pointing to the nonexistant (for you) "new" path.
I found that if I explicitly activate a conda environment with activate myenvname before running anything it works fine. Likewise if you you run from the "anaconda prompt" (which activates your default environment for you) it works fine. But to get my default environment to stay active from anywhere like its supposed to I had to reinstall.
Here is what worked for me:
Run conda update -f matplotlib (from the Anaconda command prompt)
This gave me a new error:
RuntimeError: module compiled against API version a but this version of numpy is 9
Traceback (most recent call last):
File "C:\svn\hw4\code\test_con.py", line 1, in <module>
import matplotlib.pyplot
File "C:\Anaconda\lib\site-packages\matplotlib\pyplot.py", line 27, in <module>
import matplotlib.colorbar
File "C:\Anaconda\lib\site-packages\matplotlib\colorbar.py", line 32, in <module>
import matplotlib.artist as martist
File "C:\Anaconda\lib\site-packages\matplotlib\artist.py", line 12, in <module>
from .transforms import Bbox, IdentityTransform, TransformedBbox, \
File "C:\Anaconda\lib\site-packages\matplotlib\transforms.py", line 39, in <module>
from matplotlib._path import (affine_transform, count_bboxes_overlapping_bbox,
ImportError: numpy.core.multiarray failed to import
Run conda update -f numpy
This gave me a new error:
Traceback (most recent call last):
File "C:\svn\hw4\code\test_con.py", line 1, in <module>
import matplotlib.pyplot
File "C:\Anaconda\lib\site-packages\matplotlib\pyplot.py", line 27, in <module>
import matplotlib.colorbar
File "C:\Anaconda\lib\site-packages\matplotlib\colorbar.py", line 34, in <module>
import matplotlib.collections as collections
File "C:\Anaconda\lib\site-packages\matplotlib\collections.py", line 27, in <module>
import matplotlib.backend_bases as backend_bases
File "C:\Anaconda\lib\site-packages\matplotlib\backend_bases.py", line 56, in <module>
import matplotlib.textpath as textpath
File "C:\Anaconda\lib\site-packages\matplotlib\textpath.py", line 22, in <module>
from matplotlib.mathtext import MathTextParser
File "C:\Anaconda\lib\site-packages\matplotlib\mathtext.py", line 63, in <module>
import matplotlib._png as _png
ImportError: DLL load failed: The specified module could not be found.
Run conda install anaconda. This rolled your system back to a stable anaconda distribtution, and solved the issue. (I guess you could skip steps 1 and 2, but I don't feel like going through it again to confirm…)