Management command not found without unzipping .egg - django

I have a django app that I've packaged according to the docs here: https://docs.djangoproject.com/en/1.5/intro/reusable-apps/
I installed the app into a virtual environment using setup.py.
./setup.py install
The app's web UI runs fine from the virtual environment. But I cannot access the custom management command with this vanilla install.
(django_grm)[grm#controller django_grm]$ python ./manage.py sync_to_graphite
Unknown command: 'sync_to_graphite'
Here's what the virtual environment looks like when the command will not execute:
(django_grm)[grm#controller django_grm]$ ll /home/grm/venv/django_grm/lib/python2.7/site-packages
total 1148
...
-rw-rw-r-- 1 grm grm 243962 Jun 19 17:11 django_grm-0.0.4-py2.7.egg
...
However, once I unzip the .egg file, the management command works as expected.
(django_grm)[grm#controller django_grm]$ cd /home/grm/venv/django_grm/lib/python2.7/site-packages
(django_grm)[grm#controller site-packages]$ unzip django_grm-0.0.4-py2.7.egg
(django_grm)[grm#controller site-packages]$ ll /home/grm/venv/django_grm/lib/python2.7/site-packages
total 1152
...
-rw-rw-r-- 1 grm grm 243962 Jun 19 17:11 django_grm-0.0.4-py2.7.egg
drwxrwxr-x 6 grm grm 4096 Jun 19 17:16 dj_grm
...
(django_grm)[grm#controller site-packages]$ cd /home/grm/django_projects/django_grm/
(django_grm)[grm#controller django_grm]$ python ./manage.py sync_to_graphite
<success>
Is this normal behaviour? It feels wonky.

I strongly suggest using pip instead of setup.py. It tends to do a much better job of installing packages as well as managing them.
Once you have your virtual environment in place, it would be:
$ . env/bin/activate
$ pip install [APP_NAME]
This installs a non-zipped version of the app in the virtual environment.
If the app is a zip from somewhere, you can still use pip
$ pip install http://[URL_TO_ZIP]

Let's take a look at the part of the source that loads management commands:
def find_commands(management_dir):
"""
Given a path to a management directory, returns a list of all the command
names that are available.
Returns an empty list if no commands are defined.
"""
command_dir = os.path.join(management_dir, 'commands')
try:
return [f[:-3] for f in os.listdir(command_dir)
if not f.startswith('_') and f.endswith('.py')]
except OSError:
return []
which is called by:
# Find and load the management module for each installed app.
for app_name in apps:
try:
path = find_management_module(app_name)
_commands.update(dict([(name, app_name)
for name in find_commands(path)]))
except ImportError:
pass # No management module - ignore this app
So, yeah, Django doesn't support apps installed in a zipped file, at least here; it wants an explicit commands directory inside management_dir.
As #tghw notes, installing via pip will keep the package in a directory instead of zipping it. You can also (and probably should also) set zip_safe=False in your setup() command; this will stop setuptools/distribute/etc from trying to zip up your package no matter how you install it.

Related

How to use an already built Caffe when running py-faster-rcnn?

I'm trying to build and run py-faster-rcnn model on my Ubuntu 16.04.
However, when I run ./tools/demo.py (as stated in the installation guide), I get the following error:
➜ py-faster-rcnn git:(master) ✗ ./tools/demo.py
Traceback (most recent call last):
File "./tools/demo.py", line 18, in <module>
from fast_rcnn.test import im_detect
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 16, in <module>
import caffe
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/denis/WEB/DeepLearning/py-faster-rcnn/tools/../caffe-fast-rcnn/python/caffe/pycaffe.py", line 13, in <module>
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: No module named _caffe
Before attempting to install py-faster-rcnn, I've installed Caffe in my ~/code/caffe folder and it seems to work fine:
➜ ~ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> print caffe.__version__
1.0.0-rc3
So, if I can import caffe module in python environment, I assume I've installed it successfully.
Here're the commands I've used (from the official guide):
sudo make all
sudo make test
sudo make runtest
sudo make pycaffe
sudo make distribute
Then I've cloned the py-faster-rcnn repository in my ~/WEB/DeepLearning folder.
After that I've followed the installation instructions from the repo:
Clone the repo
cd $FRCN_ROOT/lib && make
cd $FRCN_ROOT/caffe-fast-rcnn
make -j8 && make pycaffe (I didn't run this)
cd $FRCN_ROOT && ./data/scripts/fetch_faster_rcnn_models.sh
cd $FRCN_ROOT && ./tools/demo.py
So, step 4 in the installation guide says I have to build caffe and pycaffe in $FRCN_ROOT/caffe-fast-rcnn folder. The contents of caffe-fast-rcnn folder seem to be identical with the original caffe repository from which I've built Caffe.
So, it seems that I don't need to build caffe again, right? When trying to run the demo, I've skipped the step of building caffe and got the error stated above.
After googling for a while, I've found out that my issue is connected with path environment variables, so below are my path variables in .bashrc:
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:~/code/caffe/distribute/lib:$LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=/usr/include/python2.7
export PYTHONPATH=~/code/caffe/python:$PYTHONPATH
Am I doing something wrong and I have to change my path variables somehow?
Or I really need to build caffe again, but in a caffe-fast-rcnn folder?
And what about this distribute folder I've generated in ~/code/caffe/distribute by running sudo make distribute? Is it of any use? If so, how should I use it? The official documentation is not very clear about it.
A simple, clear and detailed explanation on how to use an already built Caffe framework with other projects like Faster-RCNN would be really helpful.
I struggled with this for a while and then got it working as below.
First, check PYTHONPATH env variable. It should have python path like, for eg. based on your python version and installation
PYTHONPATH = /usr/lib/python2.7
If its empty, you can set it with python path captured in your python shell. To check python path information, open python shell and type below
>>import sys
>>for p in sys.path
... print(p)
It will list you all python path info, for eg
...
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
...
If you have installed caffe already and want it to configure to be used by python, you just need to update your PYTHONPATH env variable by adding path to your /caffe-installation-path/python folder to it, like
export PYTHONPATH = /home/mypc/caffe-master/python:$PYTHONPATH
Note:- You don't need to rebuild caffe but configure caffe and python in PYTHONPATH env variable correctly.

nltk module installation in pip through cmd

When I tried to run this command:
c:\python27\scripts\pip install nltk-3.2.1-py2.py3-none-any
I am getting the error:
no matching distribution found
Although i have installed nltk from
http://www.lfd.uci.edu/~gohlke/pythonlibs/#nltk
Kindly help.
I am working on Windows 8 64-bit Version
Installing new modules can be a nightmare if you are new to Python.First delete any old versions of NLTK if already installed. Open cmd navigate to C:\Users\user\AppData\Local\Programs\Python\Python35-32\Scripts, the default directory, using the command cd path_name_comes_here. Otherwise goto the path where you have installed python and goto the Scripts subfolder and use this path here onwards. Now the most preferred way is to use pip install module_you_want_to_install for anything in python. Pip automatically fetches everything it needs to install said module.
Simply use pip install nltk.
Another method is to use easy_install requirement_or_URL.
Some rare occasions its best to download the wheel from http://www.lfd.uci.edu/~gohlke/pythonlibs and from there you can simply use pip install downloaded_wheel_name again. But make sure to copy the name of the wheel EXACTLY.
Post installation make sure that your package is accessible from C:\Users\user\AppData\Local\Programs\Python\Python35-32\Lib\site-packages or a similar path depending on where you installed python.
Try Anaconda - Instead . Always works
C:\Users\sanan>conda install -c anaconda nltk
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment C:\Users\sanan\Miniconda3:
The following NEW packages will be INSTALLED:
nltk: 3.2.4-py36_0 anaconda
The following packages will be UPDATED:
conda: 4.3.23-py36_0 --> 4.3.25-py36_0 anaconda
The following packages will be SUPERSEDED by a higher-priority channel:
conda-env: 2.6.0-0 --> 2.6.0-0 anaconda
Proceed ([y]/n)? y
conda-env-2.6. 100% |###############################| Time: 0:00:00 22.84 kB/s
nltk-3.2.4-py3 100% |###############################| Time: 0:00:02 774.24 kB/s
conda-4.3.25-p 100% |###############################| Time: 0:00:00 578.90 kB/s
After installation is complete .
import nltk
print(nltk.__version__)
C:\Public\Code\textnorm>python attempt1.py
3.2.4

libexpat could not be found by linux system

I am developing with vddk library for ubuntu 12.10 i386. I constantly get following error:
Cannot open library: libexpat.so.0: cannot open shared object file: No such file or directory.
When I run apt-file search libexpat.so
it shows me following
lib64expat1: /usr/lib64/libexpat.so.1
lib64expat1: /usr/lib64/libexpat.so.1.6.0
lib64expat1-dev: /usr/lib64/libexpat.so
libexpat1: /lib/i386-linux-gnu/libexpat.so.1
libexpat1: /lib/i386-linux-gnu/libexpat.so.1.6.0
libexpat1-dev: /usr/lib/i386-linux-gnu/libexpat.so
I already tried to create a symlink
sudo ln -s /usr/lib/i386-linux-gnu/libexpat.so /usr/lib/i386-linux-gnu/libexpat.so.0
but it did not work.
Strange thing:
ls -l `locate libexpat.so`
ls: cannot access /lib/i386-linux-gnu/libexpat.so.1: No such file or directory
ls: cannot access /lib/i386-linux-gnu/libexpat.so.1.6.0: No such file or directory
ls: cannot access /usr/lib/vmware-vix-disklib/lib64/libexpat.so.0: No such file or directory
-rw-r--r-- 1 ubuntu ubuntu 141320 Aug 20 09:21 /home/ubuntu/vddk/lib64/libexpat.so.0
-rw-r--r-- 1 root root 141320 Feb 3 16:45 /usr/lib/vmware-vix-disk-lib/vmware-vix-disk-lib/lib64/libexpat.so.0
-rw-r--r-- 1 root root 141320 Aug 20 09:21 /usr/vmware-vix-disklib-distrib/lib64/libexpat.so.0
apt-file shows only the contents of a package, or in your case the package(s) by file name, but it works regardless of whether the package is installed or not.
You need to install libexpat1:
sudo apt-get install libexpat1
If you are about to compile and link custom C programs against libexpat1 you will also need:
sudo apt-get install libexpat1-dev
Fixed by
manually downloading and installing libexpat package form here
http://expat.sourceforge.net/ and look for downloading page. It should take you to sourceforge page and select stable package
Go to the source code download page and build with the correct flag.
Since you are building for x86, you'll need to run configure like this:
./configure CFLAGS=-m32

ImportError: No module named twisted.internet

I installed python 2.7.5 which is working fine.
I then install scrapy (which, I think, uses twisted internally). My scrapy spider is also working fine.
I installed twisted:
sudo apt-get install python-twisted
Then, I created a sample program using Echo Server code shown here
Here is the code
from twisted.internet import protocol, reactor
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
reactor.listenTCP(1234, EchoFactory())
reactor.run()
I try to run this code using this command:
$ python twistedTester.py
Traceback (most recent call last):
File "twistedTester.py", line 1, in <module>
from twisted.internet import protocol, reactor
ImportError: No module named twisted.internet
Can anyone help me with how I can debug why my twisted package is not being picked up by Python installation?
If you use pip just try:
pip install twisted
The same works with w3lib and lxml.
On some *nix systems this might give you a permission error. If that happens, try:
sudo -H pip install twisted
I figured out why this error was happening. For some reason, using apt-get to install a python package was not installing it right.
So, I had to download a tar ball and install the package from them.
I downloaded Twisted tar from here.
I did a tar xjf Twisted-13.1.0.tar.bz2 - this created a directory called Twisted-13.1.0
next, cd Twisted-13.1.0
Finally, python setup.py install
This gave me an error. Twisted required another package called zope.interface. So, I downloaded tar ball for zope.interface from here. Then, ran this command tar xzf zope.interface-3.6.1.tar.gz That created a folder called zope.interface-3.6.1. So, cd into zope.interface-3.6.1 and run python setup.py install
Note: Depending on your user's rights, you may want to do these commands in sudo mode. Just add the keyword sudo before every command.
please rename the file twisted.py to something else. whenever you import a function from a file the interpreter will search for the file in the current location and then it searches in the library. so if you have any file in the name of "twisted.py" you should probably rename it.
after renaming it. dont fail to remove the twisted.pyc file before running it again.
It happened to me too. Finally I figure out that there is a file named twisted.py my present working directory. I removed twisted.py and twisted.pyc. Problem resolved.
Looks like Twisted may have removed the twisted.internet module from the current release. Pinning on the version required by scrapy (17.9.0) worked for me:
$ pip install twisted==17.9.0
Checking if it's installed:
$ python -c "import twisted.internet; print(twisted.internet)"
<module 'twisted.internet' from '/Users/username/dev/env/redacted-ewmlD2h2/lib/python3.7/site-packages/twisted/internet/__init__.py'>
I figured out why apt-get install python-twisted was not enough or "installing it right", as you said, user1700184.
I use Debian Wheezy and Python 2.7.
I just had to move the folder named "twisted" from /usr/lib/python2.7/dist-packages/ to /usr/lib/python2.7/
The same has to be done with the package "zope" and any other one that you do install but is not retrieved when you try run your code.
However, why this is even necessary in my case is still a mystery since my sys.path does include both /usr/lib/python2.7/ and /usr/lib/python2.7/dist-packages, so whatever was under dist-packages should have been retrieved by the interpreter.
I think it is worth noting that if you use sudo to launch python you are using your original default system python. This is NOT the python that your PATH points to. For example if you are using Anaconda and you have updated your path such that which python points to path/to/anaconda/bin/python, sudo which python will still point to usr/bin/python.
So obviously sudo python twistedTester.py will not find the twisted module. To get around this you should explicitly pass the path to the anaconda python. Like so:
sudo path/to/anaconda/bin/python twistedTester.py

How to completely replace python 3 with python 2 in arch linux

I want to completely replace python 3 with python 2 in arch linux. I have already read https://wiki.archlinux.org/index.php/Python but it only provides a temporary fix. I need to ensure that when I call
#!/usr/bin/python
My program is using python 2 instead of python 3.
In Arch, /usr/bin/python is actually a symlink to python3. Assuming you've already installed python2, as root, change the symlink to point to python2:
cd /usr/bin
ls -l python
lrwxrwxrwx 1 root root 7 5 sept. 07:04 python -> python3
ln -sf python2 python
ls -l python
lrwxrwxrwx 1 root root 7 Dec 11 19:28 python -> python2
If you're using the python2-virtualenv package, then do the same for /usr/bin/virtualenv:
cd /usr/bin
ln -sf virtualenv2 virtualenv
Changing the default symlink is a bad idea, and it gets recreated on python3 updates. Instead, create a local python override:
sudoedit /usr/local/bin/python
Paste this inside and save the file:
#!/bin/bash
exec python2 "$#"
Don't forget to make it executable:
sudo chmod +x /usr/local/bin/python