I know that intall the scrapy should install the w3lib first,so I install the w3lib firstly,but when I import the scrapy in python ide,the program is crashed.
the error:
creating Twisted.egg-info
writing requirements to Twisted.egg-info\requires.txt
writing Twisted.egg-info\PKG-INFO
writing top-level names to Twisted.egg-info\top_level.txt
writing dependency_links to Twisted.egg-info\dependency_links.txt
writing manifest file 'Twisted.egg-info\SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'Twisted.egg-info\SOURCES.txt'
writing manifest file 'Twisted.egg-info\SOURCES.txt'
copying twisted\internet\_sigchld.c -> build\lib.win-amd64-2.7\twisted\internet
creating build\lib.win-amd64-2.7\twisted\internet\iocpreactor\iocpsupport
copying twisted\internet/iocpreactor/iocpsupport\iocpsupport.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreactor/i
ocpsupport
copying twisted\internet/iocpreactor/iocpsupport\winsock_pointers.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreac
tor/iocpsupport
copying twisted\python\_epoll.c -> build\lib.win-amd64-2.7\twisted\python
copying twisted\python\_initgroups.c -> build\lib.win-amd64-2.7\twisted\python
copying twisted\python\sendmsg.c -> build\lib.win-amd64-2.7\twisted\python
copying twisted\runner\portmap.c -> build\lib.win-amd64-2.7\twisted\runner
copying twisted\test\raiser.c -> build\lib.win-amd64-2.7\twisted\test
running build_ext
What's wrong?
This is how I installed scrapy on ubuntu:
sudo apt-get update
sudo apt-get install python-pip build-essential python-dev libxslt-dev libxml2-dev
sudo -H pip install Scrapy
scrapy version
The important thing that solved my issues was sudo -H pip install Scrapy specifically the -H flag.
I also exited out of the terminal and started a new terminal to ensure the all the environment variables were set correctly
Make sure you had installed the Twisted, pyOpenSSL and pycrypto.
These are my steps to install scrapy on ubuntu.
1.install gcc and lxml:
sudo apt-get install python-dev
sudo apt-get install libevent-dev
sudo apt-get install libxml2 libxml2-dev
apt-get install libxml2-dev libxslt-dev
apt-get install python-lxml
2.install twisted:
sudo apt-get install python-twisted python-libxml2 python-simplejson
sudo apt-get install build-essential libssl-dev libffi-dev python-dev
3.install pyOpenSSL:
wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz
tar -zxvf pyOpenSSL-0.13.tar.gz
cd pyOpenSSL-0.13
sudo python setup.py install
4.install pycrypto
wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz
tar -zxvf pycrypto-2.5.tar.gz
cd pycrypto-2.5
sudo python setup.py install
5.install easy_install:(if you don't have easy_install)
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py
6.install w3lib
sudo easy_install -U w3lib
7.install scrapy
sudo easy_install Scrapy
If you wanna know much,please goto my blog.
First install system dependencies
sudo apt-get install -y \
python-dev python-pip python-setuptools \
libffi-dev libxml2-dev libxslt1-dev \
libtiff5-dev libjpeg62-turbo-dev zlib1g-dev libfreetype6-dev \
liblcms2-dev libwebp-dev tcl8.5-dev tk8.5-dev python-tk
Then add followings in your requirements.txt
lxml
pyOpenSSL
Scrapy
Pillow
And finally pip install -r requirements.txt
You can look around gist.github.com as well to resolve latest dependencies issues. I'm using docker to setup scrapy deps in a separate container.
I've created one for mine needs here
Related
I've tried "apt-get install python-mysqldb" which results in:
root#ps1svr:~# apt-get install python-mysqldb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package python-mysqldb is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'python-mysqldb' has no installation candidate
Note: "apt-get install python3-mysqldb" works, however I have a lot of code written for Python 2.x which no longer runs, and this is causing enough problems that I'm probably going to have to reinstall Ubuntu 18.04
Also you can just add the Ubuntu 18.04 repositoery to install the python-mysqldb package:
sudo add-apt-repository 'deb http://archive.ubuntu.com/ubuntu bionic main'
sudo apt update
sudo apt install -y python-mysqldb
This will download, build and install it for all users, using pip
sudo apt install libmysqlclient-dev python2.7-dev
wget https://bootstrap.pypa.io/get-pip.py
sudo python2.7 get-pip.py
sudo wget https://raw.githubusercontent.com/paulfitz/mysql-connector-c/master/include/my_config.h -O /usr/include/mysql/my_config.h
sudo pip2 install MySQL-python
Answer found from MySQLdb install error - _mysql.c:44:23: error: my_config.h: No such file or directory
I'm trying to do: pip install -r requirements.txt on an AWS server. I recently pulled a git commit.
I get this error:
Command 'pip' not found, but can be installed with:
sudo apt install python-pip
So I tried entering:
sudo apt install python-pip install -r requirements.txt
and then
sudo apt install python-pip -r requirements.txt
But both attempts gave me this error:
E: Command line option 'r' [from -r] is not understood in combination with the other options.
What is the correct command to install this? Thank you.
You are mixing multiple commands.
apt ; It is Debian's package manager. It has nothing to do with python packages. You install pip through apt. There are also other ways of doing it.
pip : As understood it is python package manager. You can install dependencies for your project by listing them in requirements.txt.
The correct way would be :
sudo apt install python-pip
#install from a file requirements.txt:
sudo pip install -r requirements.txt
#install as a user :
pip install -U -r requirements.txt
I have tried a lot of online posts to install opencv but they are not working for Ubuntu 16.04. May anyone please give me the steps to install openCV 2.4.13 on it?
There's a script I had made a while back, for installing the latest version of OpenCV (4.2 as of the last update to this answer)
Here's the link to it
https://github.com/rsnk96/Ubuntu-Setup-Scripts/blob/master/Build-OpenCV.sh
NOTE: For Ubuntu 16.10+, there are some minor dependency changes you will have to make. Have a look at the comments for the same.
Since you do not want to install opencv contrib, and you specifically want opencv 2.4.13, I have modified the script below. I would suggest you go through the script and understand what is happening before you execute it
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install build-essential -y
sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev -y
sudo apt-get install python3-numpy python3-pip python3-scipy python3-matplotlib python-dev python-matplotlib python-numpy python-scipy -y
sudo apt-get install python-pip python-tk libqt4-dev libqt4-opengl-dev libeigen3-dev yasm libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev sphinx-common texlive-latex-extra libv4l-dev libdc1394-22-dev libavcodec-dev libavformat-dev libswscale-dev default-jdk ant -y
echo "GUI and openGL extensions"
sudo apt-get install qt4-default libqt4-opengl-dev libvtk5-qt4-dev libgtk2.0-dev libgtkglext1 libgtkglext1-dev -y
echo "image manipulation libraries"
sudo apt-get install libpng3 pngtools libpng12-dev libpng12-0 libpng++-dev -y
sudo apt-get install libjpeg-dev libjpeg9 libjpeg9-dbg libjpeg-progs libtiff5-dev libtiff5 libtiffxx5 libtiff-tools libjasper-dev libjasper1 libjasper-runtime zlib1g zlib1g-dbg zlib1g-dev -y
echo "video manipulation libraries"
sudo apt-get install libavformat-dev libavutil-ffmpeg54 libavutil-dev libxine2-dev libxine2 libswscale-dev libswscale-ffmpeg3 libdc1394-22 libdc1394-22-dev libdc1394-utils -y
echo "codecs"
sudo apt-get install libavcodec-dev -y
sudo apt-get install libfaac-dev libmp3lame-dev -y
sudo apt-get install libopencore-amrnb-dev libopencore-amrwb-dev -y
sudo apt-get install libtheora-dev libvorbis-dev libxvidcore-dev -y
sudo apt-get install ffmpeg x264 libx264-dev -y
sudo apt-get install libv4l-0 libv4l v4l-utils -y
echo "multiproccessing library"
sudo apt-get install libtbb-dev -y
echo "finally download and install opencv"
mkdir opencv
cd opencv
wget "https://github.com/opencv/opencv/archive/2.4.13.2.zip"
unzip opencv-2.4.13.2.zip
cd opencv-2.4.13.2
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=RELEASE \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DINSTALL_C_EXAMPLES=ON \
-DINSTALL_PYTHON_EXAMPLES=ON \
-DBUILD_EXAMPLES=ON \
-DBUILD_opencv_cvv=OFF \
-DBUILD_NEW_PYTHON_SUPPORT=ON \
-DWITH_TBB=ON \
-DWITH_V4L=ON \
-DWITH_QT=ON \
-DWITH_OPENGL=ON \
-DWITH_VTK=ON ..
echo "making and installing"
make -j8
sudo make install
echo "finishing off installation"
sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig
echo "Congratulations! You have just installed OpenCV. And that's all, folks! :P"
P.S. Create a script file out of this and then just execute the script file, rather than copy-pasting it line by line into the terminal. You can do that by copying all of it, placing it in a file with the extension .sh, and then simply running that .sh file from the terminal using $ ./filename.sh
According to this source, from Ubuntu 16.04, you can now just do this to install OpenCV for Python2.7:
pip install opencv-python
or for python3:
pip3 install opencv-python
For more information, see here.
I tested this on my machine and it works, great stuff this :).
This is much easier than all other methods I have come across thus far.
sudo apt-get install build-essential cmake git pkg-config
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev
sudo apt-get install libgtk2.0-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libatlas-base-dev gfortran
sudo apt-get install python2.7-dev
sudo pip install numpy
sudo apt-get install python-opencv
Then you can have a try:
$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv
>>> import cv2
If you have more than one opencv installation on your machine, now I tell you the most direct, effective and easy way to choose opencv version in python.
You can type the following command to check the current opencv version in python
import cv2
cv2.__version__
opencv path in python is set in system variable PYTHONPATH, you can echo this variable with echo $PYTHONPATH
If I want to use opencv3.x in python, then using vi or gedit open .bashrc in the home folder, add this content to the end
export PYTHONPATH=/home/ismart/ipa2/devel/lib/python2.7/dist-packages:/home/ismart/catkin_ws/install_isolated/lib/python2.7/dist-packages:/usr/local/opencv3.x/lib/python2.7/dist-packages
Replace the path with yours
I am trying to get the django_auth_module working but I don't think I managed to install it properly.
I downloaded the package and ran setup.py install.
Then in my settings.py file I tried to import the module ldap and it gave me the following error :
ImportError: no module named ldap
I am working on a CentOS 6 server.
Maybe it has to do with where I should install the module? The folder is in the directory just above my site folder, but maybe that's wrong...
RESOLVED :
Ok, I just needed to install the module python-ldap... problem solved!
Install missing module
Hint: To install python-ldap with pip on Ubuntu, some libraries are needed.
Use the apt-get commands to install them.
Python 2
sudo apt-get install python-dev libldap2-dev libsasl2-dev libssl-dev
sudo pip install python-ldap
Python 3
sudo apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev
sudo pip3 install pyldap
Try the below command to install for Python3.
sudo apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev
sudo pip3 install pyldap
I need to:
sudo apt-get install rubygems
sudo gem install <lots of gems>
does the bootstrap action have sudo access?
The answer is yes. You can test your bootstrap script like this:
elastic_mapreduce --create --alive --ssh
This will create a node and give you a ssh connection to it, from which you can test your bootstrap script.
UPDATE: For reference here is what I'm running:
#!/bin/bash
sudo apt-get -y -V install irb1.8 libreadline-ruby1.8 libruby libruby1.8 rdoc1.8 ruby ruby1.8 ruby1.8-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install bson bson_ext json tzinfo i18n activesupport --no-rdoc --no-ri
UPDATE2: to install aws-sdk
#!/bin/bash
# ruby developer packages
sudo apt-get -y -V install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8
sudo apt-get -y -V install libreadline-ruby1.8 libruby1.8 libopenssl-ruby
# nokogiri requirements
sudo apt-get -y -V install libxslt-dev libxml2-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install aws-sdk --no-rdoc --no-ri
-y on apt-get makes it not prompt you
I wget rubygems because the version you get with apt-get is way out of date, and some gems won't build using an old version.
Yup, and here is a list of commands that I ran to setup my instance (for all the people who google for this question later):
sudo apt-get update
sudo apt-get -y install emacs
sudo apt-get -y install rubygems
sudo gem install fastercsv --source http://rubygems.org
sudo gem install crack --source http://rubygems.org
sudo gem install json_pure --source http://rubygems.org
exit