I understand that sys.path refers to
OS paths that have your system libraries. I take it that these refer to /lib in *nix or Windows on Windows.
current directory python started from - I take it if Python is started from C:\Python, this would be the current path
environmental variable $PYTHONPATH or %PYTHONPATH% - This refers to the path where I can call the Python binary from the command line
you can add paths at runtime - Which I understand to be when I run IDLE
I am able to add paths by running the command sys.path.append however when I run the command sys.path.remove to 'delete' the path I appended, I am unable to do so. Is there a way to do so without having to close IDLE each time?
I am running Python 2.7 on Windows 7 as well as Ubuntu
Everything works as intended on my machine :)
Python 2.7.3 (default, Sep 26 2012, 21:51:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/home/sergey')
>>> sys.path
['', ..., '/home/sergey']
>>> sys.path.remove('/home/sergey')
>>> sys.path
['', ...]
>>>
What exactly have you tried?
Regarding your understanding of things - I'm afraid there are some mis-understandings:
sys.path is a list of directories which contain Python modules, not system libraries. So, simplifying, when you have something like import blah in your script, Python interpreter checks those directories one by one to check if there is a file called blah.py (or a subdirectory named blah with __init__.py file inside)
Current directory is where the script is located, not where Python interpreter is. So if you have foo.py and bar.py in a directory, you can use import bar in foo.py and the module will be found because it's in the same directory.
$PYTHONPATH is an environment variable which is getting appended to sys.path on interpreter startup. So, again, it is related to module search path and has nothing to do with starting Python from command line.
Correct, you can modify sys.path at runtime - either when running a python script on in IDLE
See sys.path and site for more details.
We can try below to insert, append or remove from sys.path
>>> import sys
>>>
>>> sys.path.insert(1, '/home/log')
>>> sys.path.append('/home/log')
>>> sys.path
['', '/home/log']
>>> sys.path.remove('/home/log')
>>> sys.path
>>> ['']
>>>
Use
sys.path.append('path/to/file')
instead of
sys.path.append('path/to/file/')
Same with sys.path.remove().
Related
I'm trying to build and run py-faster-rcnn model on my Ubuntu 16.04.
However, when I run ./tools/demo.py (as stated in the installation guide), I get the following error:
➜ py-faster-rcnn git:(master) ✗ ./tools/demo.py
Traceback (most recent call last):
File "./tools/demo.py", line 18, in <module>
from fast_rcnn.test import im_detect
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 16, in <module>
import caffe
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/pycaffe.py", line 13, in <module>
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: No module named _caffe
Before attempting to install py-faster-rcnn, I've installed Caffe in my ~/code/caffe folder and it seems to work fine:
➜ ~ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> print caffe.__version__
1.0.0-rc3
So, if I can import caffe module in python environment, I assume I've installed it successfully.
Here're the commands I've used (from the official guide):
sudo make all
sudo make test
sudo make runtest
sudo make pycaffe
sudo make distribute
Then I've cloned the py-faster-rcnn repository in my ~/WEB/DeepLearning folder.
After that I've followed the installation instructions from the repo:
Clone the repo
cd $FRCN_ROOT/lib && make
cd $FRCN_ROOT/caffe-fast-rcnn
make -j8 && make pycaffe (I didn't run this)
cd $FRCN_ROOT && ./data/scripts/fetch_faster_rcnn_models.sh
cd $FRCN_ROOT && ./tools/demo.py
So, step 4 in the installation guide says I have to build caffe and pycaffe in $FRCN_ROOT/caffe-fast-rcnn folder. The contents of caffe-fast-rcnn folder seem to be identical with the original caffe repository from which I've built Caffe.
So, it seems that I don't need to build caffe again, right? When trying to run the demo, I've skipped the step of building caffe and got the error stated above.
After googling for a while, I've found out that my issue is connected with path environment variables, so below are my path variables in .bashrc:
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:~/code/caffe/distribute/lib:$LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=/usr/include/python2.7
export PYTHONPATH=~/code/caffe/python:$PYTHONPATH
Am I doing something wrong and I have to change my path variables somehow?
Or I really need to build caffe again, but in a caffe-fast-rcnn folder?
And what about this distribute folder I've generated in ~/code/caffe/distribute by running sudo make distribute? Is it of any use? If so, how should I use it? The official documentation is not very clear about it.
A simple, clear and detailed explanation on how to use an already built Caffe framework with other projects like Faster-RCNN would be really helpful.
I struggled with this for a while and then got it working as below.
First, check PYTHONPATH env variable. It should have python path like, for eg. based on your python version and installation
PYTHONPATH = /usr/lib/python2.7
If its empty, you can set it with python path captured in your python shell. To check python path information, open python shell and type below
>>import sys
>>for p in sys.path
... print(p)
It will list you all python path info, for eg
...
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
...
If you have installed caffe already and want it to configure to be used by python, you just need to update your PYTHONPATH env variable by adding path to your /caffe-installation-path/python folder to it, like
export PYTHONPATH = /home/mypc/caffe-master/python:$PYTHONPATH
Note:- You don't need to rebuild caffe but configure caffe and python in PYTHONPATH env variable correctly.
I am a python newbie and am using the book Learn Python the Hard Way. Exercise 25 from the book asks us to import the python module into the python interpreter.
I used "/usr/bin/python" to invoke the interpreter:
S-MacBook-Air:~ s$ /usr/bin/python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
But i get an error when importing the file:
>import ex26_test
>>Traceback (most recent call last):
>>File "< stdin >", line 1, in < module >
>ImportError: No module named ex26_test
the file sits on the following folder
(~/documents/1Webdev/LPTHW/ex26_test.py)
How can i import the module "ex26_test" into the interpreter please?
thank you in advance!
SD
When a module named spam is imported, the interpreter first searches
for a built-in module with that name. If not found, it then searches
for a file named spam.py in a list of directories given by the
variable sys.path. sys.path is initialized from these locations:
1) The directory containing the input script (or the current directory
when no file is specified).
2) PYTHONPATH (a list of directory names,
with the same syntax as the shell variable PATH).
3) The installation-dependent default.
python module import documentation
so you can cd to ~/documents/1Webdev/LPTHW and run python from there.
Added path to your module in sys.path
import sys
sys.path.append(r'~/documents/1Webdev/LPTHW')
import ex26_test
you need to open python in the same folder where the file is located because python can't see folders that aren't in the sys.path. There is of course way to add that folder temporary or permanently to sys.path that I will show in a moment
solution 1: move to that folder, use de cd (change directory) command to do it, that should look something similar to this:
$ cd ~/documents/1Webdev/LPTHW
$ ~/documents/1Webdev/LPTHW: python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ex26_test
>>>
note that if your python is correctly configured you should be able to call it be just typing python
solution 2: add that folder to sys.path to do it, open python and do the follow
>>> import sys
>>> sys.path.append("~/documents/1Webdev/LPTHW")
>>> import ex26_test
>>>
this is the temporary solution, but it will work regardless where you open python
solution 3: permanently add that folder to sys.path, to do this you need to modify/create the environ variable PYTHONPATH, I don't know how to do that in mac, but it should not be that difficult, in that regard some of those should be helpful: https://stackoverflow.com/search?q=pythonpath+on+mac in particular this looks like the correct one: How can I edit PYTHONPATH on a Mac?
Ok, it should be easy, but previous answers don't give all the solution. You can see where python find its modules using the sys.path function, but before you can use it, you have to import the module sys :
>>> import sys
>>> print(sys.path)
Next you have to add the folder where your module is with the commande path.append
>>> sys.path.append("~/documents/1Webdev/LPTHW")
And to load your module :
>>> from ex26_test import *
Now you can access all functions defined in your ex26_test.py file.
OS: CentOS 6.6
Python 2.7
So, I've (re)installed Canopy after it suddenly stopped working after an abrupt shutdown. It worked fine immediately after the install (I installed as my default Python). But after one reboot, when I try to open it with /root/Canopy/canopy (the icon under applications no longer works, either), I get the following error:
(Canopy 64bit) [xxuser#xxlinux ~]$ /root/Canopy/canopy Traceback (most recent call last): File "/home/xxuser/qiime_software/sphinx-1.0.4-release/lib/python2.7/site-packages/site.py", line 73, in <module>
__boot() File "/home/xxuser/qiime_software/sphinx-1.0.4-release/lib/python2.7/site-packages/site.py", line 2, in __boot
import sys, imp, os, os.path ImportError: No module named path
I found this link: Python - os.path doesn't exist: AttributeError: 'module' object has no attribute 'path', but both of my os.py and os.pyc were 250 and 700 bytes, respectively. There was another file called site.py which was 0 bytes and site.pyc was about 100 bytes. What are these files? And would deleting them hurt anything (which is what they did)? And why is this happening after reboot? (using reboot command).
I also found this: https://groups.google.com/forum/#!topic/spyderlib/hKB15JYyLqM , which could be relevant. I've updated my python path before with sys.path.append('/..')
My guess is that for some reason os.path isn't in sys.path? and __boot can't find it? But I'm new to Python and Linux and want to know what I'm doing before I go modifying any boot files, paths, etc.
Thanks in advance.
More information (saw that I'm supposed to update new info in an edit to original question. New to this.)
From one of the comments:
This is what I got:
import os.path
import posixpath
os.path
module 'posixpath' from '/home/xxuser/qiime_software/python-2.7.3-release/lib/python2.7/posixpath.pyc'
posixpath
module 'posixpath' from '/home/xxuser/qiime_software/python-2.7.3-release/lib/python2.7/posixpath.pyc'
Looks like os.path is there.
Could this have to do with a permissions error? I have it installed to /root/Canopy/canopy and I found this: docs.python.org/2/library/os.html#module-os (section 15.1.4). Does that make sense?
I'm also not sure if the following is related, but it might possibly. I can no longer seem to update my path with sys.path.append('/file/path/here'). It works until I close the terminal, then I have to re-append the next time I want to call a module from the new directory. Are sys.path and os.path related in any way?
Just fixed this on OSX with:
brew uninstall python
brew install python
No idea why, never seen it in 5 years of working with Python :S
It turns out that I was onto something with my last comment.
I'd downloaded a bunch of biology modules that depend on python, and so many of them came with their own install. When I added the modules to ~/.bashrc, my bash began calling them in advance of my original CentOS install. Resetting ~/.bashrc and restarting (for some reason source ~/.bashrc didn't work) eliminated all of the extra stuff I'd added to my $PATH and Canopy began working again. I'm going to go through and remove the extra installations of python and hopefully the issue will be behind me. Thanks to everyone who posted answers, especially A.J., because that's what got me thinking about .bashrc .
Edit: With some better understanding, this was all because of using python in a virtual environment. Canopy was resetting my path every time I opened it. I'm using a self-installed virtual environment now and have configured my path.
Try seeing if you have posixpath by typing import posixpath:
>>> import os.path
>>> os.path
<module 'posixpath' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>
>>> ^D
bash-3.2$ python
>>> import posixpath
>>> posixpath
<module 'posixpath' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>
>>>
Question
Why is it that python2.7 when called using a subprocess via python3 does
not have the same sys.path as python2.7 called normally? Specifically,
python2.7 via subprocess does not have the "/path/to/site-packages/"
directory in sys.path.
Context
I'd like to use fabric to deploy a Django app I'm writing. My problem is that
I've written the app in python3, but fabric doesn't have explicit python3
support yet. My workaround, until fabric is fully compatible with python3,
is to call the fab script using subprocess.
For some reason when I call python2.7 using subprocess via python3, I
don't have access to any modules in site-packages.
python2.7 checks
I've got python2.7 and fabric==1.10.0 installed via Enthought.
$ which python
/Users/.../Library/Enthought/Canopy_32bit/User/bin/python
$ python --version
Python 2.7.6 -- 32-bit
$ which fab
/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab
$ fab --version
Fabric 1.10.0
Paramiko 1.15.1
subprocess checks
I have no problem calling fab from within python2.7 using subprocess.
$ python
Enthought Canopy Python 2.7.6 | 32-bit | (default, Apr 11 2014, 12:06:39)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('fab --version', shell=True)
'Fabric 1.10.0\nParamiko 1.15.1\n'
I also have no problem calling python2.7 from within python3 using subprocess.
$ python3
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('which python', shell=True)
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/python\n'
>>> subprocess.check_output('python --version', shell=True)
Python 2.7.6 -- 32-bit
b''
DistributionNotFound: Fabric==1.10.0
However, even though my subprocess of python2.7 can "find" the fab script, I
can't call it.
# python3
>>> subprocess.check_output(['which', 'fab'])
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab\n'
>>> subprocess.check_output(['fab', '--version'])
Traceback (most recent call last):
File "/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab", line 5, in <module>
from pkg_resources import load_entry_point
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 2877, in <module>
working_set.require(__requires__)
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 698, in require
needed = self.resolve(parse_requirements(requirements))
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 596, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Fabric==1.10.0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['fab', '--version']' returned non-zero exit status 1
site-packages not in sys.path
It appears that python2.7 when called using subprocess via python3 does
not have the same sys.path as python2.7 called normally.
As expected, sys.path did not have the Enthought "site-packages" directory,
which contains the fabric module.
# python3
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## does not contain '/path/to/Enthought/python2.7/site-packages'
Manually add site-packages to sys.path
To confirm that it's possible: when I manually add the correct
"site-packages" directory, I can successfully import fabric.
# python3
>>> subprocess.check_output('python -c\
"import sys; sys.path.append(\'/path/to/Enthought/site-packages\');\
from fabric import version; print version.get_version()"',\
shell = True)
b'1.10.0\n'
Other options?
There's got to be a better way to make sure that python2.7, when
invoked via subprocess from python3, has the same sys.path as python2.7
invoked normally. Can someone more familiar with subprocess weigh in?
Additional thoughts
It's really interesting that python2.7 can spawn another python2.7
via subprocess and that subprocess has the correct site-packages dir
in sys.path.
$ python
>>> import subprocess
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## contains "/path/to/Enthought/python2.7/site-packages"
I also compared the sys.path's from python3, python3 subprocessed by python3,
and python3 subprocessed by python2.7, and was a bit surprised to find that
all three resulted in the same sys.path.
subprocess supports an env parameter that, if given, will be the environment for the called command -- so make a copy of it, remove any troublesome variables, and pass that copy to subprocess:
my_env = os.environ.copy()
del my_env['__PYENV_LAUNCHER__']
subprocess.check_output(..., env=my_env)
I am running python code where I want to write some output to a particular folder (which different from the location where I execute the script).
Therefore I was planning to change the path of Python to this particular folder using the os module:
os.chdir("myLocation.../Folder")
However, the script still writes to the folder where I executed the script, and when I invoke the command
os.curdir
it returns ".".
I am a little bit lost here and would appreciate any hint.
os.chdir should do the correct thing. Here is some code used for testing on python REPL, assuming you have a ./test dir in working dir.
>>> import os
>>> os.chdir('test')
>>> f = open('testfile', 'w')
>>> print>>f, 'hello world'
>>> f.close()
test/testfile is now present with the right contents.