Unable To Run IPython nbconvert From Python2.7 Virtual Environment - python-2.7

I have a virtual environment of Python 2.7 with ipython installed (Ubuntu 16.04.2 (Xenial) LTS.)
When I’m working in the virtual environment (after running source venv/bin/activate in bash shell while being in the parent directory of the virtual environment) I have no problem executing conversion of my jupiter’s notebook from bash shell like so:
ipython nbconvert --to html --execute my_notes.ipynb --stdout > /tmp/report.html
But when I’m trying to call that command from fabric’s task using subprocess:
command = ['ipython', 'nbconvert', '--to', 'html', '--execute', notebook_path, '--stdout']
output = subprocess.check_output(command,
cwd=os.environ['PYTHONPATH'],
env=os.environ.copy())
It always fails with this exception I cannot find a reason for it:
Traceback (most recent call last):
File "/opt/backend/venv/bin/ipython", line 7, in <module>
from IPython import start_ipython
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
from .core.application import Application
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
from IPython.core import release, crashhandler
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
from IPython.core import ultratb
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/ultratb.py", line 119, in <module>
from IPython.core import debugger
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/debugger.py", line 46, in <module>
from pdb import Pdb as OldPdb
File "/usr/lib/python2.7/pdb.py", line 59, in <module>
class Pdb(bdb.Bdb, cmd.Cmd):
AttributeError: 'module' object has no attribute 'Cmd'
More info to save your time.
I’ve tried:
Using same paths for PYTHONPATH as I got from PyCharm run/debug configuration.
Using nbconvert as python library from this documentation.
Tried os.system("ipython nbconvert…").
Wrapped working command (ipython nbconvert…) with a shell script and used it in subprocess.check_output and os.system.
Get drunk and bang my head on a brick wall.
And always end-up with that cursed exception.

Reposting as an answer for completeness:
There was a file called cmd.py somewhere where Python was finding it as an importable module. This was shadowing the cmd module in the standard library, which is used by pdb, which IPython imports. When pdb tried to subclass a class from cmd, that class wasn't there. Moving cmd.py out of the way lets it find the cmd module it needs.
This is an unfortunate annoyance with Python - lots of short words are already used as module names, and using them yourself produces crashes, with a wide range of different errors.

Related

Problem with migrating a django server (syntax error)

I am trying to make a django server for a sociometric badge (https://github.com/HumanDynamics/openbadge-server) for our university's project. The code (and the whole badge) has been done by someone else and I have not myself changed anything, I am simply trying to get it to work. I am able to build the server but when trying to migrate or create a superuser, I get a syntax error. I've been trying to troubleshoot it by myself but I have very limited knowledge of python, django and Ubuntu so I'm probably missing something.
The error implies an error in the line 44 of /usr/local/lib/python2.7/site-packages/pkgconf/init.py but I cannot find the file so I cannot check it. In fact, the whole site-packages folder is empty so I wonder if I have installed modules in a wrong way? The code is written in python2.7 (Which I cannot change as it is not my code) so I also wonder if the python2.7 being EOL could cause issues? It has already broken some parts, mainly how to get some of the dependencies.
The code and docker files used in this project can be found here: https://github.com/HumanDynamics/openbadge-server
The dependency versions should be fine, the Django version should be compatible with Python2.7 and same for other modules. I've tried changing the versions around but to no avail. Down here is the requirement texts
Django==1.8.4
Fabric==1.10.2
django-grappelli==2.7.1
simplejson==3.8.0
MarkupSafe==0.23
django-pipeline==1.5.4
djangorestframework==3.2.3
djangorestframework-expiring-authtoken==0.1.1
pytz==2015.7
python-dateutil==2.5.3
jsonfield==1.0.3
django-controlcenter===0.2.6
# Configuration
django-environ==0.4.1
# Python-PostgreSQL Database Adapter
psycopg2==2.7.3.2
# Unicode slugification
awesome-slugify==1.6.5
# Import and export using the admin tool
# Using tablib 0.12.1. Newer versions break the import-export add-on
tablib==0.12.1
django-import-export==1.0.0
coverage==4.3.1
django-coverage-plugin==1.3.1
Sphinx==1.5.1
django-extensions==1.7.5
Werkzeug==0.11.15
django-test-plus==1.0.16
factory-boy==2.8.1
django-debug-toolbar==1.6
# improved REPL
ipdb==0.10.1
pytest-django==3.1.2
pytest-sugar==0.8.0
This is the error. From my limited knowledge, I'd gather that it doesn't find the files from /usr/local/lib/python2.7/site-packages/ but it is completely empty, the dependencies are either installed locally or to dist-packages. Someone earlier said that it was a python3 problem but nothing should be Python3. Could it also be a docker version problem if the build somehow installs wrong things?
Starting openbadge-server_postgres_1 ... done
Postgres is up - continuing...
Traceback (most recent call last):
File "manage.py", line 23, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
django.setup()
File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 86, in create
module = import_module(entry)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/site-packages/controlcenter/__init__.py", line 1, in <module>
from .dashboards import Dashboard # NOQA
File "/usr/local/lib/python2.7/site-packages/controlcenter/dashboards.py", line 10, in <module>
from . import app_settings
File "/usr/local/lib/python2.7/site-packages/controlcenter/app_settings.py", line 1, in <module>
from pkgconf import Conf
File "/usr/local/lib/python2.7/site-packages/pkgconf/__init__.py", line 44
class Conf(metaclass=ConfMeta):
^
SyntaxError: invalid syntax
I will provide information to best of my abilities.
Fixed the issue by adding django-pkgconf==0.3.0 to the requirements. While the requirements did not have the package at all, it was still installed (and used) through other packages and it installed version 0.4.0 which does not support Python 2.7.
Weird thing is that I could not find a trace where it was installed if I installed it without having it on requirements so even when I manually installed 0.3.0, it would still use 0.4.0 (despite seemingly not having it installed) so in order to get it to work it had to be installed through the docker build.

Problems with CPLEX Python API on a mac

I have read many posts about problems but none of them can solve mine. Although I have been following this blog exactly I still get this error when I try to run one of the example src python files:
Traceback (most recent call last):
File "facility.py", line 25, in <module>
import cplex
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/__init__.py", line 43, in <module>
import callbacks
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/callbacks.py", line 48, in <module>
from _internal._aux_functions import apply_freeform_two_args, apply_freeform_one_arg
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/__init__.py", line 22, in <module>
import _list_array_utils
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_list_array_utils.py", line 13, in <module>
import _pycplex as CPX
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_pycplex.py", line 19, in <module>
_pycplex_platform = swig_import_helper()
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_pycplex.py", line 15, in swig_import_helper
_mod = imp.load_module('_pycplex_platform', fp, pathname, description)
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_pycplex_platform.py", line 23, in <module>
from cplex._internal.py1013_cplex1251 import *
ImportError: dlopen(/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/py1013_cplex1251.so, 2): no suitable image found. Did find:
/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/py1013_cplex1251.so: mach-o, but wrong architecture
Unfortunately I am not familiar with the /.bash_profile but what is posted in the link I added at the end.
Can please someone help me out here?
A possible solution to this would be to check whether you can copy the cplex directory manually over towards the site-packages that are installed (you may need to use sudo).
From your stacktrace I see that you have installed cplex into
/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/
First run (I assume you python 2.7) in the interactive shell:
import site; site.getsitepackages()
See How do I find the location of my Python site-packages directory? for details about this step.
This will give you the directory of the site-packages where you need to copy the "cplex" directory to. I assume it is /Library/Python/2.7/site-packages from here
on a mac then run:
sudo cp -r ./cplex /Library/Python/2.7/site-packages/
This sets up the cplex manually as an importable package for your python installation. You should therefore be able to import cplex within the python interactive shell.

Pyside Import Error "No Module named Ctypes"

Im pretty new to working with this side of python so pardon me if I'm a bit off kilter.
I am trying to incorporate PySide into Cryengine's Python 2.7.5 Integration; I used PIP to install the Site - Package to Cryengines Python Directory using the pip install PySide.whl -t CRYENGINE/Editor/Python/Lib/Site - Package command through command prompt.
Here is the script I am trying to run in the Python Shell Of CryEngine:
import sys
import site
site.ENABLE_USER_SITE
print site.USER_BASE
site.addsitedir("F:\SteamLibrary\steamapps\common\CRYENGINE\Editor\Python\Lib\site-packages\PySide")
from PySide import QtGui
app = QtGui.QApplication(sys.argv)
wid = QtGui.QWidget()
wid.resize(250, 150)
wid.setWindowTitle('Simple')
wid.show()
sys.exit(app.exec_())
I get the following error in the Python script terminal:
] from PySide import QtGui
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\site-packages\PySide\__init__.py", line 41, in <module>
_setupQtDirectories()
File "C:\Python27\lib\site-packages\PySide\__init__.py", line 9, in _setupQtDirectories
from . import _utils
File "C:\Python27\lib\site-packages\PySide\_utils.py", line 31, in <module>
import ctypes
ImportError: No module named ctypes
I used pip to install the PySide wheel files. So its really weird that I am getting this error. I double checked the files that ctypes uses are there, and in the Python 2.7 installation. And all of them are there. I tested this using Python Shell, and it works great! But for some reason when I run it through CryEngine's python 2.7.5 integration even though the file path is to C:\Python27 where all of those file Ctypes files are!
I am almost sure that I am oversimplifying the problem, but I'm not sure where to start getting this problem solved. I want to get this working so I can build an integration that will work for the whole CryEngine Community. I've looked all over the place for a solution, but am really at the end of my rope here.
Any help would be greatly appreciated!!

What should I modify to solve the "No module named _sqlite3" error message?

I installed ATpy-0.9.7 on my pc successfully and I also have the Python version of "2.7.5".
But when I import atpy I get the following error message:
>>> import atpy
ERROR: ImportError: No module named _sqlite3 [unknown]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "atpy/__init__.py", line 1, in <module>
from .basetable import Table, TableSet, VectorException
File "atpy/basetable.py", line 15, in <module>
from . import registry
File "atpy/registry.py", line 186, in <module>
from . import sqltable
File "atpy/sqltable.py", line 10, in <module>
from . import sqlhelper as sql
File "atpy/sqlhelper.py", line 11, in <module>
import sqlite3
File "/export/aibn84_2/zahra/lib/Python-2.7.5/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/export/aibn84_2/zahra/lib/Python-2.7.5/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named _sqlite3
I also installed db_sqlite3.egg-info. I don't know why this error message occurs!
I installed again th python2.7.5 with the following command :
./configure --prefix=$PYTHONPATH
but I also get this error after executing make:
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _sqlite3 bsddb185
dbm dl gdbm
imageop sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
How could I run configure in order to install required C libraries?
If you are using a self built version of Python you need to ensure that both the base and the development sqllite3 packages are installed on your system before building Python.
If they are not and, as you said, you do not have superuser privileges, you can download and build sqlite locally, and get your Python build to use that version. This blog post describes how.
According to this question
How can I install sqlite3 to Python?
...you shouldn't have to install anything to get sqlite3 for python. Before I could import atpy I did have to install astropy (which was quite involved). After I did that, everything worked.

Pydoop: No module named _hdfs_*

I was able to build and install Pydoop without errors, so, for example, I can do the following:
>>> import pydoop
>>> pydoop.__version__
'0.10.0'
However, when I try to import main Pydoop modules such as pipes or hdfs I'm getting ImportError:
>>> import pydoop.hdfs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pydoop/hdfs/__init__.py", line 79, in <module>
from fs import hdfs, default_is_local
File "pydoop/hdfs/fs.py", line 28, in <module>
hdfs_ext = pydoop.import_version_specific_module("_hdfs")
File "pydoop/__init__.py", line 111, in import_version_specific_module
return import_module(complete_mod_name(name))
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named _hdfs_2_0_0_cdh_4_3_0
In addition, when I try to use pydoop script I'm getting such a hint:
...
ImportError: /usr/local/lib/python2.7/dist-packages/pydoop/_pipes_2_0_0_cdh_4_3_0.so: undefined symbol: BIO_s_mem
BIO_s_mem is a symbol from libssl (OpenSSL), so I guess Pydoop can't find this shared library. I made sure it is available, ends with .so (as opposed to, say, .so.1) and is in LD_LIBRARY_PATH.
So what may be the reason for this error? How can I fix it (build options? environment variables?)
Any help is appreciated.
What OS version are you using? Try setting LD_PRELOAD to the path of your libssl, e.g.:
export LD_PRELOAD=/lib/x86_64-linux-gnu/libssl.so.1.0.0
Not sure about the pipes error, but I ran into your issue with _hdfs_2_0_0_cdh_4_3_0 (mine was a different version of hadoop, but I believe the problem is similar).
The setup.py script seems to want to make an egg file in /usr/local/lib/python2.7/dist-packages for pydoop, but the setup requires that it just be a folder (which will have that _hdfs_2_0_0_cdh_4_3_0.so file in it).
The solution is pretty simple: just delete /usr/local/lib/python2.7/dist-packages/pydoop-0.11.1.egg-info or the equivalent for you version.