importing a python module into the python interpreter - python-2.7

I am a python newbie and am using the book Learn Python the Hard Way. Exercise 25 from the book asks us to import the python module into the python interpreter.
I used "/usr/bin/python" to invoke the interpreter:
S-MacBook-Air:~ s$ /usr/bin/python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
But i get an error when importing the file:
>import ex26_test
>>Traceback (most recent call last):
>>File "< stdin >", line 1, in < module >
>ImportError: No module named ex26_test
the file sits on the following folder
(~/documents/1Webdev/LPTHW/ex26_test.py)
How can i import the module "ex26_test" into the interpreter please?
thank you in advance!
SD

When a module named spam is imported, the interpreter first searches
for a built-in module with that name. If not found, it then searches
for a file named spam.py in a list of directories given by the
variable sys.path. sys.path is initialized from these locations:
1) The directory containing the input script (or the current directory
when no file is specified).
2) PYTHONPATH (a list of directory names,
with the same syntax as the shell variable PATH).
3) The installation-dependent default.
python module import documentation
so you can cd to ~/documents/1Webdev/LPTHW and run python from there.

Added path to your module in sys.path
import sys
sys.path.append(r'~/documents/1Webdev/LPTHW')
import ex26_test

you need to open python in the same folder where the file is located because python can't see folders that aren't in the sys.path. There is of course way to add that folder temporary or permanently to sys.path that I will show in a moment
solution 1: move to that folder, use de cd (change directory) command to do it, that should look something similar to this:
$ cd ~/documents/1Webdev/LPTHW
$ ~/documents/1Webdev/LPTHW: python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ex26_test
>>>
note that if your python is correctly configured you should be able to call it be just typing python
solution 2: add that folder to sys.path to do it, open python and do the follow
>>> import sys
>>> sys.path.append("~/documents/1Webdev/LPTHW")
>>> import ex26_test
>>>
this is the temporary solution, but it will work regardless where you open python
solution 3: permanently add that folder to sys.path, to do this you need to modify/create the environ variable PYTHONPATH, I don't know how to do that in mac, but it should not be that difficult, in that regard some of those should be helpful: https://stackoverflow.com/search?q=pythonpath+on+mac in particular this looks like the correct one: How can I edit PYTHONPATH on a Mac?

Ok, it should be easy, but previous answers don't give all the solution. You can see where python find its modules using the sys.path function, but before you can use it, you have to import the module sys :
>>> import sys
>>> print(sys.path)
Next you have to add the folder where your module is with the commande path.append
>>> sys.path.append("~/documents/1Webdev/LPTHW")
And to load your module :
>>> from ex26_test import *
Now you can access all functions defined in your ex26_test.py file.

Related

python related (numpy and scipy)

when i run command import numpy as np or import scipy as sp, it gives me error like:
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
import numpy as np
ImportError: No module named numpy
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
import scipy as sp
ImportError: No module named scipy
(Disclaimer: there are already tons of well established way / tutorials on the internet. I'm merely posting this to hopefully help out quickly)
 What you want to achieve
To install a library (e.g. numpy, scipy) locally on a machine (e.g. laptop, server, etc.) and import that library from a Python code.
 One of the solutions: Anaconda
One of the popular / quick ways in the Python scientific community is to do this via Anaconda (disclaimer 2: I personally prefer Anaconda due to its ease of enabling me to switch / play with different Python environments). Here is the step by step instructions:
Download and install the Anaconda distribution onto the machine locally.
Create a file environment.yml and store it anywhere you like (e.g. a subdirectory within your home directory). The file looks like this gist file - tweak it to your taste (e.g. choose Python version 2.x vs 3.x, add/remove/edit dependencies, etc.)
Within the same directory where you've created the environment.yml, create a conda environment by: conda env create -f environment.yml. For this particular gist file, it will create a conda environment (called "helloworld") with the specific Python version (2.7) and the anaconda package (which includes the popular numpy and scipy libraries).
Activate the environment (i.e. "go inside that environment") by: source activate helloworld (replace "helloworld" with whatever name you specified in environment.yml).
Now you are in the "helloworld" conda environment, start up a python console: jupyter console.
Now try importing stuff within this console:
Python 2.7.13 |Anaconda 4.4.0 (x86_64)| (default, Dec 20 2016, 23:05:08)
Type "copyright", "credits" or "license" for more information.
IPython 5.3.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import numpy as np
In [2]: import scipy as sp
In [3]: np.version.version
Out[3]: '1.12.1'
In [4]: sp.version.version
Out[4]: '0.19.0'
(to quit console just do a Ctrl + D to go back to command line)
For step 5 above, also try out:
jupyter notebook
jupyter qtconsole
And play with the python commands.
When you are done with the conda environment, just "deactivate" (i.e. get out of) it by doing this in the command line: source deactivate.
Top tip: don't forget step 4 - this defines which conda environment you are in (i.e. which python version and libraries available etc.). I occasionally ommited step 4 by accident and get that error "no module named numpy", etc.)
See this Anaconda get started guide for more info.
The non Anaconda way
If you would like to avoid Anaconda all together, just simply do this in the command line:
Install the numpy and scipy libraries:
pip install numpy
pip install scipy
Start a Python interpreter:
python
Do the library import stuff within the Python interpreter:
>>> import numpy as np
>>> import scipy as sp
>>> np.version.version
'1.11.3'
>>> sp.version.version
'1.11.3'
 Additional alternatives
You could try out the Python VirtualEnv - though I've never really used it since I've started using Anaconda.

How to use an already built Caffe when running py-faster-rcnn?

I'm trying to build and run py-faster-rcnn model on my Ubuntu 16.04.
However, when I run ./tools/demo.py (as stated in the installation guide), I get the following error:
➜ py-faster-rcnn git:(master) ✗ ./tools/demo.py
Traceback (most recent call last):
File "./tools/demo.py", line 18, in <module>
from fast_rcnn.test import im_detect
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 16, in <module>
import caffe
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/pycaffe.py", line 13, in <module>
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: No module named _caffe
Before attempting to install py-faster-rcnn, I've installed Caffe in my ~/code/caffe folder and it seems to work fine:
➜ ~ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> print caffe.__version__
1.0.0-rc3
So, if I can import caffe module in python environment, I assume I've installed it successfully.
Here're the commands I've used (from the official guide):
sudo make all
sudo make test
sudo make runtest
sudo make pycaffe
sudo make distribute
Then I've cloned the py-faster-rcnn repository in my ~/WEB/DeepLearning folder.
After that I've followed the installation instructions from the repo:
Clone the repo
cd $FRCN_ROOT/lib && make
cd $FRCN_ROOT/caffe-fast-rcnn
make -j8 && make pycaffe (I didn't run this)
cd $FRCN_ROOT && ./data/scripts/fetch_faster_rcnn_models.sh
cd $FRCN_ROOT && ./tools/demo.py
So, step 4 in the installation guide says I have to build caffe and pycaffe in $FRCN_ROOT/caffe-fast-rcnn folder. The contents of caffe-fast-rcnn folder seem to be identical with the original caffe repository from which I've built Caffe.
So, it seems that I don't need to build caffe again, right? When trying to run the demo, I've skipped the step of building caffe and got the error stated above.
After googling for a while, I've found out that my issue is connected with path environment variables, so below are my path variables in .bashrc:
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:~/code/caffe/distribute/lib:$LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=/usr/include/python2.7
export PYTHONPATH=~/code/caffe/python:$PYTHONPATH
Am I doing something wrong and I have to change my path variables somehow?
Or I really need to build caffe again, but in a caffe-fast-rcnn folder?
And what about this distribute folder I've generated in ~/code/caffe/distribute by running sudo make distribute? Is it of any use? If so, how should I use it? The official documentation is not very clear about it.
A simple, clear and detailed explanation on how to use an already built Caffe framework with other projects like Faster-RCNN would be really helpful.
I struggled with this for a while and then got it working as below.
First, check PYTHONPATH env variable. It should have python path like, for eg. based on your python version and installation
PYTHONPATH = /usr/lib/python2.7
If its empty, you can set it with python path captured in your python shell. To check python path information, open python shell and type below
>>import sys
>>for p in sys.path
... print(p)
It will list you all python path info, for eg
...
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
...
If you have installed caffe already and want it to configure to be used by python, you just need to update your PYTHONPATH env variable by adding path to your /caffe-installation-path/python folder to it, like
export PYTHONPATH = /home/mypc/caffe-master/python:$PYTHONPATH
Note:- You don't need to rebuild caffe but configure caffe and python in PYTHONPATH env variable correctly.

Differences in sys.path when python2.7 is invoked normally or via subprocess

Question
Why is it that python2.7 when called using a subprocess via python3 does
not have the same sys.path as python2.7 called normally? Specifically,
python2.7 via subprocess does not have the "/path/to/site-packages/"
directory in sys.path.
Context
I'd like to use fabric to deploy a Django app I'm writing. My problem is that
I've written the app in python3, but fabric doesn't have explicit python3
support yet. My workaround, until fabric is fully compatible with python3,
is to call the fab script using subprocess.
For some reason when I call python2.7 using subprocess via python3, I
don't have access to any modules in site-packages.
python2.7 checks
I've got python2.7 and fabric==1.10.0 installed via Enthought.
$ which python
/Users/.../Library/Enthought/Canopy_32bit/User/bin/python
$ python --version
Python 2.7.6 -- 32-bit
$ which fab
/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab
$ fab --version
Fabric 1.10.0
Paramiko 1.15.1
subprocess checks
I have no problem calling fab from within python2.7 using subprocess.
$ python
Enthought Canopy Python 2.7.6 | 32-bit | (default, Apr 11 2014, 12:06:39)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('fab --version', shell=True)
'Fabric 1.10.0\nParamiko 1.15.1\n'
I also have no problem calling python2.7 from within python3 using subprocess.
$ python3
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('which python', shell=True)
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/python\n'
>>> subprocess.check_output('python --version', shell=True)
Python 2.7.6 -- 32-bit
b''
DistributionNotFound: Fabric==1.10.0
However, even though my subprocess of python2.7 can "find" the fab script, I
can't call it.
# python3
>>> subprocess.check_output(['which', 'fab'])
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab\n'
>>> subprocess.check_output(['fab', '--version'])
Traceback (most recent call last):
File "/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab", line 5, in <module>
from pkg_resources import load_entry_point
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 2877, in <module>
working_set.require(__requires__)
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 698, in require
needed = self.resolve(parse_requirements(requirements))
File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 596, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Fabric==1.10.0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['fab', '--version']' returned non-zero exit status 1
site-packages not in sys.path
It appears that python2.7 when called using subprocess via python3 does
not have the same sys.path as python2.7 called normally.
As expected, sys.path did not have the Enthought "site-packages" directory,
which contains the fabric module.
# python3
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## does not contain '/path/to/Enthought/python2.7/site-packages'
Manually add site-packages to sys.path
To confirm that it's possible: when I manually add the correct
"site-packages" directory, I can successfully import fabric.
# python3
>>> subprocess.check_output('python -c\
"import sys; sys.path.append(\'/path/to/Enthought/site-packages\');\
from fabric import version; print version.get_version()"',\
shell = True)
b'1.10.0\n'
Other options?
There's got to be a better way to make sure that python2.7, when
invoked via subprocess from python3, has the same sys.path as python2.7
invoked normally. Can someone more familiar with subprocess weigh in?
Additional thoughts
It's really interesting that python2.7 can spawn another python2.7
via subprocess and that subprocess has the correct site-packages dir
in sys.path.
$ python
>>> import subprocess
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## contains "/path/to/Enthought/python2.7/site-packages"
I also compared the sys.path's from python3, python3 subprocessed by python3,
and python3 subprocessed by python2.7, and was a bit surprised to find that
all three resulted in the same sys.path.
subprocess supports an env parameter that, if given, will be the environment for the called command -- so make a copy of it, remove any troublesome variables, and pass that copy to subprocess:
my_env = os.environ.copy()
del my_env['__PYENV_LAUNCHER__']
subprocess.check_output(..., env=my_env)

matplotlib plots do not show on Anaconda for Linux

I've read these and tried the solutions but still not getting anything to show:
matplotlib show nothing although i called show
Why doesn't pyplot.show() work? [duplicate]
matplotlib does not show my drawings although I call pyplot.show()
I'm using Python 2.7.6 |Anaconda 1.9.0 (64-bit) on Ubuntu 12.04.
How can I go about diagnosing the problem and trying to solve it?
EDIT:
-First I edited matplotlibrc and change the backend to GtkAgg (was set to QtAgg).
-Then I tried to change the backend via code: matplotlib.rcParams['backend'] = "GtkAgg"
EDIT2 - adding detail from the Spyder console (no plot is shown at the end):
Python 2.7.6 |Anaconda 1.9.0 (64-bit)| (default, Jan 17 2014, 10:13:17)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Imported NumPy 1.8.0, SciPy 0.13.3, Matplotlib 1.3.1
Type "scientific" for more details.
>>> scientific
This is a standard Python interpreter with preloaded tools for scientific
computing and visualization. It tries to import the following modules:
>>> import numpy as np # NumPy (multidimensional arrays, linear algebra, ...)
>>> import scipy as sp # SciPy (signal and image processing library)
>>> import matplotlib as mpl # Matplotlib (2D/3D plotting library)
>>> import matplotlib.pyplot as plt # Matplotlib's pyplot: MATLAB-like syntax
>>> from pylab import * # Matplotlib's pylab interface
>>> ion() # Turned on Matplotlib's interactive mode
Within Spyder, this interpreter also provides:
* special commands (e.g. %ls, %pwd, %clear)
* system commands, i.e. all commands starting with '!' are subprocessed
(e.g. !dir on Windows or !ls on Linux, and so on)
>>> plot(range(3))
[<matplotlib.lines.Line2D object at 0x3f3cd50>]
>>> show()
I ended up solving by doing this in Spyder:
Preferences -> Console -> External modules -> check Install Spyder's input hook for Qt
In order for your matplotlib backend choice to work, you need to have the Python bindings for that backend installed. You may be running a GTK-based windows manager like GNOME, for example, but still not have the Python bindings around. According to this, the proper Ubuntu package is python-gtk. So, open up your favorite Terminal emulator and enter
sudo apt-get install python-gtk
(you'll need admin privileges to do it). Restart your python/matplotlib session, and you should be good to go.
I'm on Mavericks, solved the problem by changing backend to Qt4Agg in matplotlibrc. (as instructed in another answer here: Why doesn't pyplot.show() work?)

Removing path from Python search module path

I understand that sys.path refers to
OS paths that have your system libraries. I take it that these refer to /lib in *nix or Windows on Windows.
current directory python started from - I take it if Python is started from C:\Python, this would be the current path
environmental variable $PYTHONPATH or %PYTHONPATH% - This refers to the path where I can call the Python binary from the command line
you can add paths at runtime - Which I understand to be when I run IDLE
I am able to add paths by running the command sys.path.append however when I run the command sys.path.remove to 'delete' the path I appended, I am unable to do so. Is there a way to do so without having to close IDLE each time?
I am running Python 2.7 on Windows 7 as well as Ubuntu
Everything works as intended on my machine :)
Python 2.7.3 (default, Sep 26 2012, 21:51:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/home/sergey')
>>> sys.path
['', ..., '/home/sergey']
>>> sys.path.remove('/home/sergey')
>>> sys.path
['', ...]
>>>
What exactly have you tried?
Regarding your understanding of things - I'm afraid there are some mis-understandings:
sys.path is a list of directories which contain Python modules, not system libraries. So, simplifying, when you have something like import blah in your script, Python interpreter checks those directories one by one to check if there is a file called blah.py (or a subdirectory named blah with __init__.py file inside)
Current directory is where the script is located, not where Python interpreter is. So if you have foo.py and bar.py in a directory, you can use import bar in foo.py and the module will be found because it's in the same directory.
$PYTHONPATH is an environment variable which is getting appended to sys.path on interpreter startup. So, again, it is related to module search path and has nothing to do with starting Python from command line.
Correct, you can modify sys.path at runtime - either when running a python script on in IDLE
See sys.path and site for more details.
We can try below to insert, append or remove from sys.path
>>> import sys
>>>
>>> sys.path.insert(1, '/home/log')
>>> sys.path.append('/home/log')
>>> sys.path
['', '/home/log']
>>> sys.path.remove('/home/log')
>>> sys.path
>>> ['']
>>>
Use
sys.path.append('path/to/file')
instead of
sys.path.append('path/to/file/')
Same with sys.path.remove().