I'm trying to build and run py-faster-rcnn model on my Ubuntu 16.04.
However, when I run ./tools/demo.py (as stated in the installation guide), I get the following error:
➜ py-faster-rcnn git:(master) ✗ ./tools/demo.py
Traceback (most recent call last):
File "./tools/demo.py", line 18, in <module>
from fast_rcnn.test import im_detect
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 16, in <module>
import caffe
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/pycaffe.py", line 13, in <module>
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: No module named _caffe
Before attempting to install py-faster-rcnn, I've installed Caffe in my ~/code/caffe folder and it seems to work fine:
➜ ~ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> print caffe.__version__
1.0.0-rc3
So, if I can import caffe module in python environment, I assume I've installed it successfully.
Here're the commands I've used (from the official guide):
sudo make all
sudo make test
sudo make runtest
sudo make pycaffe
sudo make distribute
Then I've cloned the py-faster-rcnn repository in my ~/WEB/DeepLearning folder.
After that I've followed the installation instructions from the repo:
Clone the repo
cd $FRCN_ROOT/lib && make
cd $FRCN_ROOT/caffe-fast-rcnn
make -j8 && make pycaffe (I didn't run this)
cd $FRCN_ROOT && ./data/scripts/fetch_faster_rcnn_models.sh
cd $FRCN_ROOT && ./tools/demo.py
So, step 4 in the installation guide says I have to build caffe and pycaffe in $FRCN_ROOT/caffe-fast-rcnn folder. The contents of caffe-fast-rcnn folder seem to be identical with the original caffe repository from which I've built Caffe.
So, it seems that I don't need to build caffe again, right? When trying to run the demo, I've skipped the step of building caffe and got the error stated above.
After googling for a while, I've found out that my issue is connected with path environment variables, so below are my path variables in .bashrc:
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:~/code/caffe/distribute/lib:$LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=/usr/include/python2.7
export PYTHONPATH=~/code/caffe/python:$PYTHONPATH
Am I doing something wrong and I have to change my path variables somehow?
Or I really need to build caffe again, but in a caffe-fast-rcnn folder?
And what about this distribute folder I've generated in ~/code/caffe/distribute by running sudo make distribute? Is it of any use? If so, how should I use it? The official documentation is not very clear about it.
A simple, clear and detailed explanation on how to use an already built Caffe framework with other projects like Faster-RCNN would be really helpful.
I struggled with this for a while and then got it working as below.
First, check PYTHONPATH env variable. It should have python path like, for eg. based on your python version and installation
PYTHONPATH = /usr/lib/python2.7
If its empty, you can set it with python path captured in your python shell. To check python path information, open python shell and type below
>>import sys
>>for p in sys.path
... print(p)
It will list you all python path info, for eg
...
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
...
If you have installed caffe already and want it to configure to be used by python, you just need to update your PYTHONPATH env variable by adding path to your /caffe-installation-path/python folder to it, like
export PYTHONPATH = /home/mypc/caffe-master/python:$PYTHONPATH
Note:- You don't need to rebuild caffe but configure caffe and python in PYTHONPATH env variable correctly.
Related
I have added sys.path
sys.path.append("C:\\Program Files\\FME\\fmeobjects\\python27")
in python script which works well when I run the script. I am not trying to dockerize the script. My docker script is
FROM python:2.7-alpine
ADD test1.py /
CMD [ "python", "./test1.py" ]
it builds the image but while running the image it gives error
Traceback (most recent call last):
File "./test1.py", line 17, in <module>
import fmeobjects
ImportError: No module named fmeobjects
It seems like your script cannot import fmeobjects because it is outside the container. Try adding the import for fmeobjects in the directory you ADD.
What does test1.py do?
If fmeobjects is a package / module, you need to add as mentioned above to the environment of the image.
You can also set up a distutils for it and you can pip install it in the image.
Effectively, as currently constructed, you're trying to import a package in your script that does not exist because it has not been installed.
Even for small standalone applications, using the standard distribution tools streamlines this process significantly. This is doubly true if you have colleagues that might have different usernames, directory layouts, or even operating systems. Don't manually edit sys.path in your script.
You should write a setup.py file that uses the setuptools library. Complete documentation is here but a minimal example might look like:
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(
name="fmeobjects",
version="0.1",
packages=find_packages(),
entry_points={
'console_scripts': [
'fmeobjects = fmeobjects.main:main'
]
}
)
For development use, create a virtual environment and install your package in it.
virtualenv vpy
. vpy/bin/activate
pip install -e .
The . activate line sets some additional environment variables for you, including adding the virtual environment to your $PATH. (source is an equivalent vendor extension that works in some shells; . is part of the standard and works even in minimal shells like what you get in Alpine or Busybox installations.) You can now run fmeobjects at the shell prompt, which will call the main() function in fmeobjects/main.py (see the entry_points declaration).
You have a couple of options of how to install this in Docker. Probably the most straightforward is to simply import your source tree and install it. Since Docker containers provide isolated filesystems and generally do only one thing, there's not much point in supporting an isolated Python installation within that; just install your package into the global Python.
FROM python:2.7
WORKDIR /usr/src/app
COPY . .
RUN pip install .
CMD ["fmeobjects"]
(If your virtual environment is in your source tree, you can add vpy to a .dockerignore file to cause it to not be copied, saving time and space.)
When trying to import gurobipy i get the following ImportError:
D:\Development\Anaconda3\envs\hiwi\python.exe D:/Development/Hiwi/Project_Code_Source/experiment/demo/run.py
Traceback (most recent call last):
File "D:/Development/Hiwi/Project_Code_Source/experiment/demo/run.py", line 2, in <module>
import experiment.demo.scenarios as scenarios
File "D:\Development\Hiwi\Project_Code_Source\experiment\demo\scenarios.py", line 15, in <module>
import framework.simu.ins_gen_v3 as insgen
File "D:\Development\Hiwi\Project_Code_Source\framework\simu\ins_gen_v3.py", line 10, in <module>
from gurobipy import *
File "D:\Development\Anaconda3\envs\hiwi\lib\site-packages\gurobipy\__init__.py", line 1, in <module>
from .gurobipy import *
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden. (English: Module not found.)
Process finished with exit code 1
Since searching google resulted in errors related to environment variables and 64 vs 32 bit Versions i checkt those.
I am running the 64 bit Versions of Annaconda (3.6 using a 2.7 env) on Windows 10 and Gurobi
Gurobi is installed according to this instruction (including the license): http://www.gurobi.com/downloads/get-anaconda
The GUROBI_HOME variable is correctly set to C:\gurobi702\win64 and the Path variable to C:\gurobi702\win64\bin
Python Path variables are set.
Following packages are installed: gurobi 7.5.1, mkl 2017.0.3, numpy 1.13.1, pip 9.0.1, python 2.7.13, setuptools 27.2.0, vs2008_runtime 9.00.30729.5054, wheel 0.29.0, wsgiref 0.1.2
Starting the gurobi console via cmd gurobi works fine.
I reinstalled my complete python environment multiple times and tried restarting after installation.
I got it running with a new conda environment and installing via python setup.py install, as mentioned by Greg Glockner and sascha.
This problem [from .gurobipy import] is caused from the IDE Pycharm on Ubuntu. I am not sure if you may face it with the other IDEs .
First You can make sure from the shell the python has gurobipy installed.
after you installed from the file Gurobi ~\gurobi752\win64
python setup.py install
then just run python from the shell as normal python command
python yourFileName.py and gurobi will works perfectly
In windows 10 i added the path of gurobi to the IDE pycharm
after python setup.py install and pip tool for gurobipy https://sites.google.com/site/pydatalog/python/pip-for-windows, then i added the path
Settings->Project Interpreter-> show all -> interpreter paths -> add your gurobi path
I hope it works cheers ;)
when i run command import numpy as np or import scipy as sp, it gives me error like:
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
import numpy as np
ImportError: No module named numpy
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
import scipy as sp
ImportError: No module named scipy
(Disclaimer: there are already tons of well established way / tutorials on the internet. I'm merely posting this to hopefully help out quickly)
What you want to achieve
To install a library (e.g. numpy, scipy) locally on a machine (e.g. laptop, server, etc.) and import that library from a Python code.
One of the solutions: Anaconda
One of the popular / quick ways in the Python scientific community is to do this via Anaconda (disclaimer 2: I personally prefer Anaconda due to its ease of enabling me to switch / play with different Python environments). Here is the step by step instructions:
Download and install the Anaconda distribution onto the machine locally.
Create a file environment.yml and store it anywhere you like (e.g. a subdirectory within your home directory). The file looks like this gist file - tweak it to your taste (e.g. choose Python version 2.x vs 3.x, add/remove/edit dependencies, etc.)
Within the same directory where you've created the environment.yml, create a conda environment by: conda env create -f environment.yml. For this particular gist file, it will create a conda environment (called "helloworld") with the specific Python version (2.7) and the anaconda package (which includes the popular numpy and scipy libraries).
Activate the environment (i.e. "go inside that environment") by: source activate helloworld (replace "helloworld" with whatever name you specified in environment.yml).
Now you are in the "helloworld" conda environment, start up a python console: jupyter console.
Now try importing stuff within this console:
Python 2.7.13 |Anaconda 4.4.0 (x86_64)| (default, Dec 20 2016, 23:05:08)
Type "copyright", "credits" or "license" for more information.
IPython 5.3.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import numpy as np
In [2]: import scipy as sp
In [3]: np.version.version
Out[3]: '1.12.1'
In [4]: sp.version.version
Out[4]: '0.19.0'
(to quit console just do a Ctrl + D to go back to command line)
For step 5 above, also try out:
jupyter notebook
jupyter qtconsole
And play with the python commands.
When you are done with the conda environment, just "deactivate" (i.e. get out of) it by doing this in the command line: source deactivate.
Top tip: don't forget step 4 - this defines which conda environment you are in (i.e. which python version and libraries available etc.). I occasionally ommited step 4 by accident and get that error "no module named numpy", etc.)
See this Anaconda get started guide for more info.
The non Anaconda way
If you would like to avoid Anaconda all together, just simply do this in the command line:
Install the numpy and scipy libraries:
pip install numpy
pip install scipy
Start a Python interpreter:
python
Do the library import stuff within the Python interpreter:
>>> import numpy as np
>>> import scipy as sp
>>> np.version.version
'1.11.3'
>>> sp.version.version
'1.11.3'
Additional alternatives
You could try out the Python VirtualEnv - though I've never really used it since I've started using Anaconda.
I've found a lot of posts on Macports/Django install issues but none seem to quite address my situation.
Installed Django using Macports from the command line using
sudo port install py27-django
This seemed to work fine. I opened up IDLE and was able to import django. The following bit of code
import django
from django.core import management
print django.VERSION
returns
(1, 5, 1, 'final', 0)
Which I take to mean I have the final version of Django 1.5.1 installed. So, all seems to be well.
However, now I switch over to my Django tutorial to get learning and I am asked to type this into the command line to confirm that Django is installed:
python -c "import django; print(django.get_version())"
and get the following error:
> File "<string>", line 1, in <module> ImportError: No module named
> django
Some other bits of data. Here's my $PATH:
/Library/Frameworks/Python.framework/Versions/2.7/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin
Finally, the command
django-admin.py startproject mysite
returns
-bash: django-admin.py: command not found
But the command
django-admin-2.7.py startproject mysite
works (creates a new directory called mysite in the working directory). I read somehwere that Macports renames the django-admin file.
I've only been working with Unix for a few weeks so this is a bit confusing. It seems like I have Django on my system, but perhaps the python I am trying to access from my command line is a different python from that which Macports uses.
Any help appreciated!
Dennis
You will need to set your PYTHONPATH environment variable from your terminal to reference your django install. IDLE probably did this for you, but in another shell you'll need to set it. You can set it in your terminal permanently by setting it in ~/.bashrc or ~/.bash_profile (or whatever your terminal shell is).
Your PATH starts with /Library/Frameworks/Python.framework/Versions/2.7/bin so when you type python you get the python in that directory which is the Apple supplied one nut you have installed Django for the Macports one so python does not see it.
To fix you need to have /opt/local/bin earlier on the path than the /Library one. even better remove that /Library entry and use port select python ... to choose which python to run the Macports 2.7 or Apple's or others
As for the django-admin-2.7.py issue, Macports allows you to have several versions of python at once so code depending on eth version has the version add into the script names e.g. you could have a django-admin-2.6.py as well if you installed the py26-django package. This also applies to python itself however python also adds into the port select mechanism so you set a default version.
I installed python 2.7.5 which is working fine.
I then install scrapy (which, I think, uses twisted internally). My scrapy spider is also working fine.
I installed twisted:
sudo apt-get install python-twisted
Then, I created a sample program using Echo Server code shown here
Here is the code
from twisted.internet import protocol, reactor
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
reactor.listenTCP(1234, EchoFactory())
reactor.run()
I try to run this code using this command:
$ python twistedTester.py
Traceback (most recent call last):
File "twistedTester.py", line 1, in <module>
from twisted.internet import protocol, reactor
ImportError: No module named twisted.internet
Can anyone help me with how I can debug why my twisted package is not being picked up by Python installation?
If you use pip just try:
pip install twisted
The same works with w3lib and lxml.
On some *nix systems this might give you a permission error. If that happens, try:
sudo -H pip install twisted
I figured out why this error was happening. For some reason, using apt-get to install a python package was not installing it right.
So, I had to download a tar ball and install the package from them.
I downloaded Twisted tar from here.
I did a tar xjf Twisted-13.1.0.tar.bz2 - this created a directory called Twisted-13.1.0
next, cd Twisted-13.1.0
Finally, python setup.py install
This gave me an error. Twisted required another package called zope.interface. So, I downloaded tar ball for zope.interface from here. Then, ran this command tar xzf zope.interface-3.6.1.tar.gz That created a folder called zope.interface-3.6.1. So, cd into zope.interface-3.6.1 and run python setup.py install
Note: Depending on your user's rights, you may want to do these commands in sudo mode. Just add the keyword sudo before every command.
please rename the file twisted.py to something else. whenever you import a function from a file the interpreter will search for the file in the current location and then it searches in the library. so if you have any file in the name of "twisted.py" you should probably rename it.
after renaming it. dont fail to remove the twisted.pyc file before running it again.
It happened to me too. Finally I figure out that there is a file named twisted.py my present working directory. I removed twisted.py and twisted.pyc. Problem resolved.
Looks like Twisted may have removed the twisted.internet module from the current release. Pinning on the version required by scrapy (17.9.0) worked for me:
$ pip install twisted==17.9.0
Checking if it's installed:
$ python -c "import twisted.internet; print(twisted.internet)"
<module 'twisted.internet' from '/Users/username/dev/env/redacted-ewmlD2h2/lib/python3.7/site-packages/twisted/internet/__init__.py'>
I figured out why apt-get install python-twisted was not enough or "installing it right", as you said, user1700184.
I use Debian Wheezy and Python 2.7.
I just had to move the folder named "twisted" from /usr/lib/python2.7/dist-packages/ to /usr/lib/python2.7/
The same has to be done with the package "zope" and any other one that you do install but is not retrieved when you try run your code.
However, why this is even necessary in my case is still a mystery since my sys.path does include both /usr/lib/python2.7/ and /usr/lib/python2.7/dist-packages, so whatever was under dist-packages should have been retrieved by the interpreter.
I think it is worth noting that if you use sudo to launch python you are using your original default system python. This is NOT the python that your PATH points to. For example if you are using Anaconda and you have updated your path such that which python points to path/to/anaconda/bin/python, sudo which python will still point to usr/bin/python.
So obviously sudo python twistedTester.py will not find the twisted module. To get around this you should explicitly pass the path to the anaconda python. Like so:
sudo path/to/anaconda/bin/python twistedTester.py