Docker build timeouts on AWS Beanstalk for Python app - python-2.7

I have a Python2/Django application running on AWS Beanstalk. I use Dockerfile for deployment. Everything was working fine until I added firebase-admin==3.2.1 dependency to my requirements.txt. What happens is that the Docker image build time took longer than usual and Beanstalk build process started to fail. With more analysis, I noticed that a sub-dependency is what takes long time to install, grpcio==1.31.0. It is used by Firebase lib to communicate with Google APIs.
I tried to increase the Command Timeout setting to 20 min and more but nothing changed. EB deployment process consumes the whole time interval and eventually fails. I tried to get into the EC2 instance itself to find anything useful that I can tweak but found nothing.
This is a sample of the eb-engine.log that, I believe, explains the error part.
Stored in directory: /root/.cache/pip/wheels/6f/8e/a7/c5ce8f1742b7d2b9e07cd1064cf201293d157c3c4e1021a74a
Running setup.py bdist_wheel for psutil: started
Running setup.py bdist_wheel for psutil: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/39/a0/f5/c4fa280463e29aea07797acb5312358fefb067c1f4f98e11b1
Running setup.py bdist_wheel for Pyphen: started
Running setup.py bdist_wheel for Pyphen: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/78/e3/76/42853bf2f7573a72fb5fe377a8406c979a5479ad6930506cde
Running setup.py bdist_wheel for cairocffi: started
Running setup.py bdist_wheel for cairocffi: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/3a/2e/92/89e9dcce600a1a78bebe571fdd096669cba354b29c5e1140b9
Running setup.py bdist_wheel for CairoSVG: started
Running setup.py bdist_wheel for CairoSVG: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/b0/f0/12/0c41649d33dbc95ce0188188feefc8f90aa877363322fd3ace
Running setup.py bdist_wheel for gevent: started
Running setup.py bdist_wheel for gevent: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/66/89/6a/7438b05445edc27023a9d8935801d4e70ed962e02541e57fbd
Running setup.py bdist_wheel for httplib2: started
Running setup.py bdist_wheel for httplib2: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/9a/a8/c3/72684de9397a5633e21ba89911174fbc9eb0b1fb9005c6eaf0
Running setup.py bdist_wheel for grpcio: started
Running setup.py bdist_wheel for grpcio: still running...
Running setup.py bdist_wheel for grpcio: still running...
Running setup.py bdist_wheel for grpcio: still running...
2020/11/05 16:12:14.123278 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to build docker image: Command /bin/sh -c docker build -t aws_beanstalk/staging-app /var/app/staging/ failed with error Command timed out after 300 seconds
2020/11/05 16:12:14.123302 [INFO] Executing cleanup logic
2020/11/05 16:12:14.131856 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to build the Docker image. The deployment failed.","timestamp":1604592734,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1604592734,"severity":"ERROR"}]}]}
2020/11/05 16:12:14.133315 [INFO] Platform Engine finished execution on command: app-deploy
As you will notice in the previous sample that the Docker build process fails after exactly 300 seconds regardless of the Command Timeout setting value.
Stop running the command. Error: failed to build docker image: Command /bin/sh -c docker build -t aws_beanstalk/staging-app /var/app/staging/ failed with error Command timed out after 300 seconds

Related

How to debug Cloud Run (Visual Studio Code) Local running problems?

i'm working with Google Cloud Run and Visual Studio Code. I'm trying to build a simple Flask service and trying to test it locally in the "emulator" on my Win10 laptop (by choosing the 'Cloud Run: Run/Debug Locally' task) and when I do that it seems to start trying but then always fails with the error below.. How do I mitigate such a message, as it's not very informative.. ! Thanks.
Deploy started
Deploy completed
Status check started
Resource pod/rbfcloudrun-7875f79d7-kwmsd status updated to In Progress
Resource deployment/rbfcloudrun status updated to In Progress
Resource pod/rbfcloudrun-7875f79d7-kwmsd status updated to In Progress
Resource pod/rbfcloudrun-7875f79d7-kwmsd status updated to In Progress
Resource deployment/rbfcloudrun status failed with waiting for rollout to finish: 0 of 1 updated replicas are available...
Status check failed
Update failed with error code STATUSCHECK_CONTAINER_RESTARTING
1/1 deployment(s) failed
Skaffold exited with code 1.
Cleaning up...
Finished clean up.
For reference, my dockerfile is something like:
FROM python:3
# Allow statements and log messages to immediately appear in the Knative logs
ENV PYTHONUNBUFFERED True
ENV PORT 8080
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
# Install production dependencies.
RUN pip install -r requirements.txt
RUN pip install flask
RUN pip install gunicorn
# Run the web service on container startup. Here we use the gunicorn
# webserver, with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
# Timeout is set to 0 to disable the timeouts of the workers to allow Cloud Run to handle instance scaling.
# "main:app" - because main.py is the entry file, and app is the name of the webapp instance in that file
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
..my requirements.txt file looks like:
Flask==2.0.2
requests==2.26.0
ptvsd==4.3.2 # Required for debugging.

AWS beanstalk fails installing pip requisites when downloading tensorflow==2.4.0

I'm trying to deploy a flask app that predicts whether a lung x-ray picture is COVID-19 positive using a Tensorflow CNN. It is running on python 3.7
This is my requirements.txt
tensorflow==2.4.0
Flask==1.1.2
Keras==2.4.3
numpy==1.19.5
Pillow==8.3.1
After deployment it, the environment's health status is labelled severe with a '502 Bad gateway' error.
These are the errors in the eb-engine.log
2021/08/23 10:47:40.676374 [INFO] Installing dependencies with requirements.txt by using Pip
2021/08/23 10:47:40.676387 [INFO] Running command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt
2021/08/23 10:47:48.296114 [INFO] Collecting tensorflow==2.4.0
Downloading tensorflow-2.4.0-cp38-cp38-manylinux2010_x86_64.whl (394.8 MB)
2021/08/23 10:47:48.300932 [ERROR] An error occurred during execution of command [app-deploy] - [InstallDependency]. Stop running the command. Error: fail to install dependencies with requirements.txt file with error Command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt failed with error exit status 2. Stderr:ERROR: Exception:
Traceback (most recent call last):
File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
status = self.run(options, args)
File "... repeats itself until here:
File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 896, in _pack
return self._buffer.write(obj)
MemoryError
2021/08/23 10:47:48.300957 [INFO] Executing cleanup logic
2021/08/23 10:47:48.309000 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to install application dependencies. The deployment failed.","timestamp":1629715668,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1629715668,"severity":"ERROR"}]}]}
2021/08/23 10:47:48.312983 [INFO] Platform Engine finished execution on command: app-deploy
2021/08/23 12:34:34.689865 [INFO] Starting...
2021/08/23 12:34:34.689926 [INFO] Starting EBPlatform-PlatformEngine
2021/08/23 12:34:34.689965 [INFO] reading event message file
2021/08/23 12:34:34.690627 [INFO] no eb envtier info file found, skip loading env tier info.
2021/08/23 12:34:34.690716 [INFO] Engine received EB command cfn-hup-exec
why is AWS-Beanstalk failing to install Tensorflow?
I changed the environment EC2 capacity from t2.small to c5.xlarge and the problem was solved.
Your application needs more memory space.

Your requirements.txt is invalid. AWS Elastic Beanstalk, Django

i am new to aws and django. I was tryinh to upload my code to aws elastic beanstalk using code commit while i am getting the following error
2020/03/12 13:42:30.200017 [ERROR] Command timed out after 300 seconds
2020/03/12 13:42:30.202535 [ERROR] Command /bin/sh [-c python3 -m pipenv install -r requirements.txt --skip-lock] failed with error signal: killed
2020/03/12 13:42:30.204812 [ERROR] An error occurred during execution of command [app-deploy] - [SetUpPythonEnvironment]. Stop running the command.
Error: fail to install dependencies with requirements.txt file with error Command /bin/sh [-c python3 -m pipenv install -r requirements.txt --skip-lock] failed with error signal: killed
2020/03/12 13:42:30.204825 [INFO] Executing cleanup logic
2020/03/12 13:42:30.210383 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[]}]}
2020/03/12 13:42:30.211588 [INFO] Platform Engine finished execution on command: app-deploy

MarkupSafe, SQLAlchemy and pycrypto fail because wheel build fail

Running windows with python 2.7, setuptools, pip and wheel at latest release. Trying to install dependencies using
pip install -r requirements.txt
failing because "one or more wheels failed to build".
Log:
Building wheels for collected packages: MarkupSafe, SQLAlchemy, pycrypto
Running setup.py bdist_wheel for MarkupSafe: started
Running setup.py bdist_wheel for MarkupSafe: finished with status 'error'
.
.
.
Failed to build wheel for MarkupSafe
Running setup.py bdist_wheel for SQLAlchemy: started
Running setup.py bdist_wheel for SQLAlchemy: finished with status 'error'
.
.
.
Failed to build wheel for SQLAlchemy
Running setup.py bdist_wheel for pycrypto: started
Running setup.py bdist_wheel for pycrypto: finished with status 'error'
.
.
.
Failed to build wheel for pycrypto
Failed to build MarkupSafe SQLAlchemy pycrypto
(Removed complete output because over character limit but all reach a certain point then say some file is not found)
However according to http://pythonwheels.com/ there are no wheels for those packages so why is pip trying to build wheels for them?
Try using python -m easy_install SQLAlchemy instead.

Git repos hungs up unexpectedly

There is a strange thing.. I can clone small files repos, but when it starts to take some time, it hangs up. Like:
$ pip install -e git+https://github.com/django/django.git#egg=Django
Obtaining Django from git+https://github.com/django/django.git#egg=Django
Cloning https://github.com/django/django.git to ./virtualenvs/prp/src/django
error: RPC failed; result=56, HTTP code = 0
fatal: The remote end hung up unexpectedly
Complete output from command /usr/bin/git clone -q https://github.com/django/django.git /home/mamber/virtualenvs/prp/src/django:
----------------------------------------
Command /usr/bin/git clone -q https://github.com/django/django.git /home/mamber/virtualenvs/prp/src/django failed with error code 128 in None
Storing complete log in /home/mamber/.pip/pip.log
I've tried with some other public repos and getting the same result.
I also tried just cloning the repo, without virtualenv and pip, same result:
$ git clone https://github.com/django/django.git
Cloning into 'django'...
error: RPC failed; result=56, HTTP code = 0
fatal: The remote end hung up unexpectedly
Maybe there is some timeout config I need to change. I see no traffic in my system monitor while waiting.
I am doing that in a fresh installed Linux Mint, with fresh installed apt-get install git (with no configurations).
Any ideas?