I was able to build and install Pydoop without errors, so, for example, I can do the following:
>>> import pydoop
>>> pydoop.__version__
'0.10.0'
However, when I try to import main Pydoop modules such as pipes or hdfs I'm getting ImportError:
>>> import pydoop.hdfs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pydoop/hdfs/__init__.py", line 79, in <module>
from fs import hdfs, default_is_local
File "pydoop/hdfs/fs.py", line 28, in <module>
hdfs_ext = pydoop.import_version_specific_module("_hdfs")
File "pydoop/__init__.py", line 111, in import_version_specific_module
return import_module(complete_mod_name(name))
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named _hdfs_2_0_0_cdh_4_3_0
In addition, when I try to use pydoop script I'm getting such a hint:
...
ImportError: /usr/local/lib/python2.7/dist-packages/pydoop/_pipes_2_0_0_cdh_4_3_0.so: undefined symbol: BIO_s_mem
BIO_s_mem is a symbol from libssl (OpenSSL), so I guess Pydoop can't find this shared library. I made sure it is available, ends with .so (as opposed to, say, .so.1) and is in LD_LIBRARY_PATH.
So what may be the reason for this error? How can I fix it (build options? environment variables?)
Any help is appreciated.
What OS version are you using? Try setting LD_PRELOAD to the path of your libssl, e.g.:
export LD_PRELOAD=/lib/x86_64-linux-gnu/libssl.so.1.0.0
Not sure about the pipes error, but I ran into your issue with _hdfs_2_0_0_cdh_4_3_0 (mine was a different version of hadoop, but I believe the problem is similar).
The setup.py script seems to want to make an egg file in /usr/local/lib/python2.7/dist-packages for pydoop, but the setup requires that it just be a folder (which will have that _hdfs_2_0_0_cdh_4_3_0.so file in it).
The solution is pretty simple: just delete /usr/local/lib/python2.7/dist-packages/pydoop-0.11.1.egg-info or the equivalent for you version.
Related
I have a script named requests.py that needs to use the third-party requests package. The script either can't import the package, or can't access its functionality.
Why isn't this working, and how do I fix it?
Trying a plain import and then using the functionality results in an AttributeError:
import requests
res = requests.get('http://www.google.ca')
print(res)
Traceback (most recent call last):
File "/Users/me/dev/rough/requests.py", line 1, in <module>
import requests
File "/Users/me/dev/rough/requests.py", line 3, in <module>
requests.get('http://www.google.ca')
AttributeError: module 'requests' has no attribute 'get'
In more recent versions of Python, the error message instead reads AttributeError: partially initialized module 'requests' has no attribute 'get' (most likely due to a circular import).
Using from-import of a specific name results in an ImportError:
from requests import get
res = get('http://www.google.ca')
print(res)
Traceback (most recent call last):
File "requests.py", line 1, in <module>
from requests import get
File "/Users/me/dev/rough/requests.py", line 1, in <module>
from requests import get
ImportError: cannot import name 'get'
In more recent versions of Python, the error message instead reads ImportError: cannot import name 'get' from partially initialized module 'requests' (most likely due to a circular import) (/Users/me/dev/rough/requests.py).
Using from-import for a module inside the package results in a different ImportError:
from requests.auth import AuthBase
Traceback (most recent call last):
File "requests.py", line 1, in <module>
from requests.auth import AuthBase
File "/Users/me/dev/rough/requests.py", line 1, in <module>
from requests.auth import AuthBase
ImportError: No module named 'requests.auth'; 'requests' is not a package
Using a star-import and then using the functionality raises a NameError:
from requests import *
res = get('http://www.google.ca')
print(res)
Traceback (most recent call last):
File "requests.py", line 1, in <module>
from requests import *
File "/Users/me/dev/rough/requests.py", line 3, in <module>
res = get('http://www.google.ca')
NameError: name 'get' is not defined
This happens because your local module named requests.py shadows the installed requests module you are trying to use. The current directory is prepended to sys.path, so the local name takes precedence over the installed name.
An extra debugging tip when this comes up is to look at the Traceback carefully, and realize that the name of your script in question is matching the module you are trying to import:
Notice the name you used in your script:
File "/Users/me/dev/rough/requests.py", line 1, in <module>
The module you are trying to import: requests
Rename your module to something else to avoid the name collision.
Python may generate a requests.pyc file next to your requests.py file (in the __pycache__ directory in Python 3). Remove that as well after your rename, as the interpreter will still reference that file, re-producing the error. However, the pyc file in __pycache__ should not affect your code if the py file has been removed.
In the example, renaming the file to my_requests.py, removing requests.pyc, and running again successfully prints <Response [200]>.
The error occurs because a user-created script has a name-clash with a library filename. Note, however, that the problem can be caused indirectly. It might take a little detective work to figure out which file is causing the problem.
For example: suppose that you have a script mydecimal.py that includes import decimal, intending to use the standard library decimal library for accurate floating-point calculations with decimal numbers. That doesn't cause a problem, because there is no standard library mydecimal. However, it so happens that decimal imports numbers (another standard library module) for internal use, so a script called numbers.py in your project would cause the problem.
In one especially pernicious case, having a file named token.py in a project (or the current working directory, when starting up Python in interactive mode) causes the interactive help to break:
$ touch token.py
$ python
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help
Type help() for interactive help, or help(object) for help about object.
>>> help()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/_sitebuiltins.py", line 102, in __call__
import pydoc
File "/usr/lib/python3.8/pydoc.py", line 66, in <module>
import inspect
File "/usr/lib/python3.8/inspect.py", line 40, in <module>
import linecache
File "/usr/lib/python3.8/linecache.py", line 11, in <module>
import tokenize
File "/usr/lib/python3.8/tokenize.py", line 35, in <module>
from token import EXACT_TOKEN_TYPES
ImportError: cannot import name 'EXACT_TOKEN_TYPES' from 'token' (/current/working/directory/token.py)
The traceback tells us all we need to know: calling help triggers a deferred import of the standard library pydoc, which indirectly attempts to import the standard library token, but finds our token.py which doesn't contain the appropriate name. In older versions of Python, it was even worse: tokenize would do a star-import from token, and then its top-level code would try to use a name defined there, resulting in NameError - and a stack trace not mentioning the file name token.py.
If you still encounter problems like this after tracking own and renaming or removing the appropriate .py files in your project, also check for .pyc files that Python uses to cache bytecode compilation when importing modules. In 3.x, these will be stored in folders with the special name __pycache__; it is safe to delete such folders and files, and possible to suppress them (but you normally won't want to).
I use windows 7 and python 2.7
When I used py2exe to make an .exe file I get the error;
Traceback (most recent call last):
File "mainpy", line 17, in
File "main.py", line 17, in
File "zipextimporter.pyc", line 82, in load_module
File "zipextimporter.pyc", line 82, in load_module
File "logging_init_.pyc", line 26, in
File "zipextimporter.pyc", line 82, in load_module
File "weakref.pyc", line 14, in
ImportError: cannot import name _remove_dead_weakref
The same code could be used to make an .exe file in another computer so there is nothing wrong with the code in main.py. The minor environmental difference may cause this problem. I used pycharm, python 2.7.10 and py2exe 0.6.9. On another computer all other config are the same except using sublimetext instead of pycharm.
Could anyone please tell me how to fix that?
Another tricky thing is that
It is possible that the library does not exists for the other computer.Please check whether the library exists or not.
I have a virtual environment of Python 2.7 with ipython installed (Ubuntu 16.04.2 (Xenial) LTS.)
When I’m working in the virtual environment (after running source venv/bin/activate in bash shell while being in the parent directory of the virtual environment) I have no problem executing conversion of my jupiter’s notebook from bash shell like so:
ipython nbconvert --to html --execute my_notes.ipynb --stdout > /tmp/report.html
But when I’m trying to call that command from fabric’s task using subprocess:
command = ['ipython', 'nbconvert', '--to', 'html', '--execute', notebook_path, '--stdout']
output = subprocess.check_output(command,
cwd=os.environ['PYTHONPATH'],
env=os.environ.copy())
It always fails with this exception I cannot find a reason for it:
Traceback (most recent call last):
File "/opt/backend/venv/bin/ipython", line 7, in <module>
from IPython import start_ipython
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
from .core.application import Application
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
from IPython.core import release, crashhandler
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
from IPython.core import ultratb
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/ultratb.py", line 119, in <module>
from IPython.core import debugger
File "/opt/backend/venv/local/lib/python2.7/site-packages/IPython/core/debugger.py", line 46, in <module>
from pdb import Pdb as OldPdb
File "/usr/lib/python2.7/pdb.py", line 59, in <module>
class Pdb(bdb.Bdb, cmd.Cmd):
AttributeError: 'module' object has no attribute 'Cmd'
More info to save your time.
I’ve tried:
Using same paths for PYTHONPATH as I got from PyCharm run/debug configuration.
Using nbconvert as python library from this documentation.
Tried os.system("ipython nbconvert…").
Wrapped working command (ipython nbconvert…) with a shell script and used it in subprocess.check_output and os.system.
Get drunk and bang my head on a brick wall.
And always end-up with that cursed exception.
Reposting as an answer for completeness:
There was a file called cmd.py somewhere where Python was finding it as an importable module. This was shadowing the cmd module in the standard library, which is used by pdb, which IPython imports. When pdb tried to subclass a class from cmd, that class wasn't there. Moving cmd.py out of the way lets it find the cmd module it needs.
This is an unfortunate annoyance with Python - lots of short words are already used as module names, and using them yourself produces crashes, with a wide range of different errors.
I have read many posts about problems but none of them can solve mine. Although I have been following this blog exactly I still get this error when I try to run one of the example src python files:
Traceback (most recent call last):
File "facility.py", line 25, in <module>
import cplex
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/__init__.py", line 43, in <module>
import callbacks
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/callbacks.py", line 48, in <module>
from _internal._aux_functions import apply_freeform_two_args, apply_freeform_one_arg
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/__init__.py", line 22, in <module>
import _list_array_utils
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_list_array_utils.py", line 13, in <module>
import _pycplex as CPX
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_pycplex.py", line 19, in <module>
_pycplex_platform = swig_import_helper()
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_pycplex.py", line 15, in swig_import_helper
_mod = imp.load_module('_pycplex_platform', fp, pathname, description)
File "/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/_pycplex_platform.py", line 23, in <module>
from cplex._internal.py1013_cplex1251 import *
ImportError: dlopen(/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/py1013_cplex1251.so, 2): no suitable image found. Did find:
/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/cplex/_internal/py1013_cplex1251.so: mach-o, but wrong architecture
Unfortunately I am not familiar with the /.bash_profile but what is posted in the link I added at the end.
Can please someone help me out here?
A possible solution to this would be to check whether you can copy the cplex directory manually over towards the site-packages that are installed (you may need to use sudo).
From your stacktrace I see that you have installed cplex into
/Users/sb/Applications/IBM/ILOG/CPLEX_Studio1251/cplex/python/x86_darwin/
First run (I assume you python 2.7) in the interactive shell:
import site; site.getsitepackages()
See How do I find the location of my Python site-packages directory? for details about this step.
This will give you the directory of the site-packages where you need to copy the "cplex" directory to. I assume it is /Library/Python/2.7/site-packages from here
on a mac then run:
sudo cp -r ./cplex /Library/Python/2.7/site-packages/
This sets up the cplex manually as an importable package for your python installation. You should therefore be able to import cplex within the python interactive shell.
I installed ATpy-0.9.7 on my pc successfully and I also have the Python version of "2.7.5".
But when I import atpy I get the following error message:
>>> import atpy
ERROR: ImportError: No module named _sqlite3 [unknown]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "atpy/__init__.py", line 1, in <module>
from .basetable import Table, TableSet, VectorException
File "atpy/basetable.py", line 15, in <module>
from . import registry
File "atpy/registry.py", line 186, in <module>
from . import sqltable
File "atpy/sqltable.py", line 10, in <module>
from . import sqlhelper as sql
File "atpy/sqlhelper.py", line 11, in <module>
import sqlite3
File "/export/aibn84_2/zahra/lib/Python-2.7.5/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/export/aibn84_2/zahra/lib/Python-2.7.5/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named _sqlite3
I also installed db_sqlite3.egg-info. I don't know why this error message occurs!
I installed again th python2.7.5 with the following command :
./configure --prefix=$PYTHONPATH
but I also get this error after executing make:
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _sqlite3 bsddb185
dbm dl gdbm
imageop sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
How could I run configure in order to install required C libraries?
If you are using a self built version of Python you need to ensure that both the base and the development sqllite3 packages are installed on your system before building Python.
If they are not and, as you said, you do not have superuser privileges, you can download and build sqlite locally, and get your Python build to use that version. This blog post describes how.
According to this question
How can I install sqlite3 to Python?
...you shouldn't have to install anything to get sqlite3 for python. Before I could import atpy I did have to install astropy (which was quite involved). After I did that, everything worked.