Django on Elastic Beanstalk "requirements.txt" is invalid - django

Question

I'm trying to deploy a django project with Elastic Beanstalk cli. I followed the Official Amazon Tutorial as well as this Tutorial on realpython.com.
The Deploymentprocess works fine as long as I stick to a newly created Django Project with just a view dependencies for pip.
When try it with my Django Project The Deployment fails because the requirements.txt is invalid. I created it inside my virutalenv like this: pip freeze > requirements.txt
braintree==3.32.0
cffi==1.9.1
cryptography==1.7
Django==1.10.4
django-allauth==0.29.0
django-betterforms==1.1.4
django-contrib-comments==1.7.3
django-formtools==1.0
django-payments==0.9.6
django-tinymce==2.4.0
enum34==1.1.6
idna==2.1
ipaddress==1.0.17
oauthlib==2.0.1
Pillow==3.4.2
pyasn1==0.1.9
pycparser==2.17
PyJWT==1.4.2
python-openid==2.2.5
requests==2.12.3
requests-oauthlib==0.7.0
six==1.10.0
stripe==1.43.0
suds-jurko==0.6
xmltodict==0.10.2
In my despair I tried to activate the dependencies one by one. The Deployment process is so slow that I gave up after a while. But now I know now that Django==1.10.4, django-allauth==0.29.0, django-betterforms==1.1.4, django-contrib-comments==1.7.3, django-formtools==1.0 and django-tinymce==2.4.0 are not the problem. With django-payments==0.9.6 it fails but its not the only dependency that causes the problem.
This is in my activity.log:
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/opt/python/run/venv/bin/python2.7 -c "import setuptools, tokenize;
__file__='/tmp/pip-build-BF9Oen/cffi/setup.py';
exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-BsXWzo-record/install-record.txt --single-version-externally-managed --compile --install-headers
/opt/python/run/venv/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-build-BF9Oen/cffi
You are using pip version 7.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2016-12-13 14:49:05,155 ERROR
Error installing dependencies: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1
Traceback (most recent call last):
File "/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py", line 22, in main
install_dependencies()
File "/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py", line 18, in install_dependencies
check_call('%s install -r %s' % (os.path.join(APP_VIRTUAL_ENV, 'bin', 'pip'), requirements_file), shell=True)
File "/usr/lib64/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command
'/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt'
returned non-zero exit status 1 (Executor::NonZeroExitStatus)
I't could have something to do with missing Software. I know how to use the .ebextensions .config files but I don't know whats missing. Is there a descent way to debug this locally?

Answer 1

You seem to have problems with both FFI and Python. You have Cryptography in use but don't probably have all the required libraries Install Python and FFI requirements and OpenSSL dependencies for Cryptography.
$ eb ssh # SSH into your EB environment
$ sudo su # gain root privileges
$ yum install python-devel # or python27-devel
$ yum install libffi-devel # CFFI / FFI requirements
$ yum install openssl-devel # Cryptography requirements
If Pip complains, make sure you have the latest pip installed:
# repeat SSH and root privilege steps
$ source /opt/python/run/venv/bin/activate # activate Python environment
$ pip install --upgrade pip # upgrade to latest pip
To make the changes permanent, append them to your EB configuration. This also works if you do not have direct SSH access.
# .ebextensions/10_packages.config excerpt
packages:
yum:
gcc: []
libffi-devel: []
openssl-devel: []
python-devel: [] # or python27-devel
.ebextensions folder files with the .config suffix are state declarations for the Elastic Beanstalk environment. That is, they are instructions that are run on each Elastic Beanstalk deployment when you run eb deploy or your CI environment deploys code to EB.
Amazon Linux is an RPM based Linux that uses YUM for package management. You will find RHEL / CentOS / OpenSUSE instructions for Python package installation helpful for the Amazon Linux distribution.

Answer 2

There typically isn’t a way to debug these issues locally, because people don’t normally run Amazon linux on their local machines, and EB has a very specific environment setup that is not easy to reproduce (at least it wasn’t for me).
If you want to do it fast, just ssh to one of your EB/EC2 instances and try to install the pip requirements one-by-one by hand. If you want to do it right, you do have to go through this list of requirements one-by-one. (i.e., start with one requirement in the requirements file, deploy, then two requirements, then three requirements, etc.). You may want to start with upgrading pip by adding
pip==9.0.1
At the top of your requirements file and also making sure the following Amazon Linux (if that is what you are using) are installed:
python-devel
libffi
libffi-devel