Buildozer - requirement pysftp - python-2.7

I'm able to use pysftp on kivy, but I can't package my app using this requirement. Using buildozer, in the message says I'm using pip 8.1.1 and there is another version, 8.1.2. But I checked my version and my one already is 8.1.2. And, of course, I can install pysftp using pip install pysftp. My log file is this:
# Check configuration tokens
# Ensure build layout
# Check configuration tokens
# Preparing build
# Check requirements for android
# Run 'dpkg --version'
# Cwd None
Programa de gerenciamento de pacotes Debian 'dpkg' versão 1.17.5 (amd64).
Isto é software livre; veja a GNU General Public License versão 2 ou
posterior para condições de cópia. Não existe NENHUMA garantia.
# Search for Git (git)
# -> found at /usr/bin/git
# Search for Cython (cython)
# -> found at /usr/local/bin/cython
# Search for Java compiler (javac)
# -> found at /usr/lib/jvm/java-7-openjdk-amd64/bin/javac
# Search for Java keytool (keytool)
# -> found at /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/keytool
# Install platform
# Apache ANT found at /home/awulll/.buildozer/android/platform/apache-ant-1.9.4
# Android SDK found at /home/awulll/.buildozer/android/platform/android-sdk-20
# Android NDK found at /home/awulll/.buildozer/android/platform/android-ndk-r9c
# Check application requirements
# Run './distribute.sh -l'
# Cwd /home/awulll/dapapp/.buildozer/android/platform/python-for-android
Available modules: android apsw audiostream bidi boost cherrypy c_igraph click cprotobuf cymunk django docutils ecdsa enum34 evdev ffmpeg ffmpeg2 ffpyplayer ffpyplayer_tito flask freetype gevent greenlet harfbuzz hostpython igraph itsdangerous jinja2 jpeg kivent_core kivent_cymunk kivy leveldb libevent libpq libsodium libswift libtorrent libxml2 libxslt libyaml lxml m2crypto markupsafe midistream msgpack mysql_connector netifaces numpy opencv openssl paramiko pil plyer plyvel png polygon protobuf psutil psycopg2 pyasn1 pycrypto pygame pyjnius pylibpd pyopenssl pyparsing pyqrcode python pyyaml sdl setuptools six sqlalchemy sqlite3 storm swift thrift twisted txws werkzeug wokkel zeroconf zope
# Remove directory and subdirectory /home/awulll/dapapp/.buildozer/applibs
# Create directory /home/awulll/dapapp/.buildozer/applibs
# Run 'bash -c "source venv/bin/activate && env"'
# Cwd /home/awulll/dapapp/.buildozer
XDG_VTNR=8
XDG_SESSION_ID=c1
SSH_AGENT_PID=2191
GPG_AGENT_INFO=/run/user/1000/keyring-JXgOZD/gpg:0:1
VTE_VERSION=3409
SHELL=/bin/bash
TERM=xterm
XDG_SESSION_COOKIE=9cf5cdba17a429f4e3205d8a54ac9c66-1463226321.499664-1491359051
GJS_DEBUG_OUTPUT=stderr
WINDOWID=39845895
GNOME_KEYRING_CONTROL=/run/user/1000/keyring-JXgOZD
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
USER=awulll
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
SSH_AUTH_SOCK=/run/user/1000/keyring-JXgOZD/ssh
SESSION_MANAGER=local/awulll-mint:#/tmp/.ICE-unix/2054,unix/awulll-mint:/tmp/.ICE-unix/2054
USERNAME=awulll
DEFAULTS_PATH=/usr/share/gconf/default.default.path
VIRTUAL_ENV=/home/awulll/dapapp/.buildozer/venv
XDG_CONFIG_DIRS=/etc/xdg/xdg-default:/etc/xdg
PATH=/home/awulll/dapapp/.buildozer/venv/bin:/home/awulll/.buildozer/android/platform/apache-ant-1.9.4/bin:/home/awulll/bin:/usr/lib/jvm/java-1.7.0-openjdk-amd64/bin://usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
DESKTOP_SESSION=default
ANDROIDAPI=19
GDM_XSERVER_LOCATION=local
PWD=/home/awulll/dapapp/.buildozer
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
LANG=pt_BR.UTF-8
GNOME_KEYRING_PID=1978
MANDATORY_PATH=/usr/share/gconf/default.mandatory.path
ANDROIDNDK=/home/awulll/.buildozer/android/platform/android-ndk-r9c
PS1=(venv)
MDM_XSERVER_LOCATION=local
GDMSESSION=default
PACKAGES_PATH=/home/awulll/.buildozer/android/packages
CINNAMON_VERSION=2.4.8
HOME=/home/awulll
SHLVL=2
XDG_SEAT=seat0
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
XDG_SESSION_DESKTOP=default
ANDROIDSDK=/home/awulll/.buildozer/android/platform/android-sdk-20
LOGNAME=awulll
XDG_DATA_DIRS=/usr/share/default:/usr/share/gnome:/usr/local/share/:/usr/share/:/usr/share/mdm/
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-fdxlKZ9y2o,guid=226dcf0498f8da9740aa1f9157370fd2
MDMSESSION=default
ANDROIDNDKVER=r9c
WINDOWPATH=8
TEXTDOMAIN=im-config
DISPLAY=:0
XDG_RUNTIME_DIR=/run/user/1000
MDM_LANG=pt_BR.UTF-8
XDG_CURRENT_DESKTOP=X-Cinnamon
TEXTDOMAINDIR=/usr/share/locale/
XAUTHORITY=/home/awulll/.Xauthority
COLORTERM=gnome-terminal
_=/usr/bin/env
# Install distribute
# Run 'curl http://python-distribute.org/distribute_setup.py | venv/bin/python'
# Cwd /home/awulll/dapapp/.buildozer
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
# Install requirement datetime in virtualenv
# Run 'pip install --target=/home/awulll/dapapp/.buildozer/applibs datetime'
# Cwd /home/awulll/dapapp/.buildozer
Collecting datetime
/home/awulll/dapapp/.buildozer/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/home/awulll/dapapp/.buildozer/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting zope.interface (from datetime)
Collecting pytz (from datetime)
Using cached pytz-2016.4-py2.py3-none-any.whl
Collecting setuptools (from zope.interface->datetime)
Using cached setuptools-21.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools, zope.interface, pytz, datetime
Successfully installed datetime pytz setuptools-20.9.0 zope.interface
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# Install distribute
# Run 'curl http://python-distribute.org/distribute_setup.py | venv/bin/python'
# Cwd /home/awulll/dapapp/.buildozer
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
# Install requirement pysftp in virtualenv
# Run 'pip install --target=/home/awulll/dapapp/.buildozer/applibs pysftp'
# Cwd /home/awulll/dapapp/.buildozer
Collecting pysftp
/home/awulll/dapapp/.buildozer/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/home/awulll/dapapp/.buildozer/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Using cached pysftp-0.2.8.tar.gz
Collecting paramiko>=1.7.7 (from pysftp)
Using cached paramiko-2.0.0-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from paramiko>=1.7.7->pysftp)
Using cached pyasn1-0.1.9-py2.py3-none-any.whl
Collecting cryptography>=1.1 (from paramiko>=1.7.7->pysftp)
Using cached cryptography-1.3.2.tar.gz
Collecting idna>=2.0 (from cryptography>=1.1->paramiko>=1.7.7->pysftp)
Using cached idna-2.1-py2.py3-none-any.whl
Collecting six>=1.4.1 (from cryptography>=1.1->paramiko>=1.7.7->pysftp)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting setuptools>=11.3 (from cryptography>=1.1->paramiko>=1.7.7->pysftp)
Using cached setuptools-21.0.0-py2.py3-none-any.whl
Collecting enum34 (from cryptography>=1.1->paramiko>=1.7.7->pysftp)
Using cached enum34-1.1.5-py2.py3-none-any.whl
Collecting ipaddress (from cryptography>=1.1->paramiko>=1.7.7->pysftp)
Using cached ipaddress-1.0.16-py27-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.1->paramiko>=1.7.7->pysftp)
Using cached cffi-1.6.0.tar.gz
Complete output from command python setup.py egg_info:
No working compiler found, or bogus compiler options
passed to the compiler from Python's distutils module.
See the error messages above.
(If they are about -mno-fused-madd and you are on OS/X 10.8,
see http://stackoverflow.com/questions/22313407/ .)
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-eC7Zv8/cffi/
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# Command failed: pip install --target=/home/awulll/dapapp/.buildozer/applibs pysftp
#
# Buildozer failed to execute the last command
# The error might be hidden in the log above this error
# Please read the full log, and search for it before
# raising an issue with buildozer itself.
# In case of a bug report, please add a full log with log_level = 2
real 0m5.286s
user 0m3.188s
sys 0m0.357s
I already tried check the link suggested in th log, but it doesn't help me too.

It looks like one of the pysftp dependencies is pulling in cffi as a further dependency, which needs compilation and therefore can't be built by python-for-android without a recipe. p4a already has a cffi recipe so it will work - maybe the easiest option is to add a pysftp recipe.
You could also try getting the newest buildozer release and running buildozer android_new debug instead. Changes to dependency resolution might mean it work, but it also might not help - especially if buildozer itself is managing this part.

Related

GCP Composer Airflow - unable to install packages using PyPi

I have created a Composer environment with image version -> composer-2.0.13-airflow-2.2.5
when i try to install software using PyPi, it fails.
details below :
Command :
gcloud composer environments update $AIRFLOW --location us-east1 --update-pypi-packages-from-file requirements.txt
requirement.txt
---------------
google-api-core
google-auth
google-auth-oauthlib
google-cloud-bigquery
google-cloud-core
google-cloud-storage
google-crc32c
google-resumable-media
googleapis-common-protos
google-endpoints
joblib
json5
jsonschema
pandas
requests
requests-oauthlib
Error :
Karans-MacBook-Pro:composer_dags karanalang$ gcloud composer environments update $AIRFLOW --location us-east1 --update-pypi-packages-from-file requirements.txt
Waiting for [projects/versa-sml-googl/locations/us-east1/environments/versa-composer3] to be updated with [projects/versa-sml-googl/locations/us-east1/operations/c23b77a9-f46b-4222-bafd-62527bf27239]..
.failed.
ERROR: (gcloud.composer.environments.update) Error updating [projects/versa-sml-googl/locations/us-east1/environments/versa-composer3]: Operation [projects/versa-sml-googl/locations/us-east1/operations/c23b77a9-f46b-4222-bafd-62527bf27239] failed: Failed to install PyPI packages. looker-sdk 22.4.0 has requirement attrs>=20.1.0; python_version >= "3.7", but you have attrs 17.4.0.
Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/60ac972a-8f5e-4b4f-a4a7-d81049fb19a3?project=939354532596 for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installation
Pls note:
I have an older Composer cluster (Composer version - 1.16.8, Airflow version - 1.10.15), where the above command works fine.
However, it is not working with the new cluster
What needs to be done to debug/fix this ?
tia!
I was able to get this working using the following code :
path = "gs://dataproc-spark-configs/pip_install.sh"
CLUSTER_GENERATOR_CONFIG = ClusterGenerator(
project_id=PROJECT_ID,
zone="us-east1-b",
master_machine_type="n1-standard-4",
worker_machine_type="n1-standard-4",
num_workers=4,
storage_bucket="dataproc-spark-logs",
init_actions_uris=[path],
metadata={'PIP_PACKAGES': 'pyyaml requests pandas openpyxl kafka-python'},
).make()
with models.DAG(
'Versa-Alarm-Insights-UsingComposer2',
# Continue to run DAG twice per day
default_args=default_dag_args,
schedule_interval='0 0/12 * * *',
catchup=False,
) as dag:
create_dataproc_cluster = DataprocCreateClusterOperator(
task_id="create_dataproc_cluster",
cluster_name="versa-composer2",
region=REGION,
cluster_config=CLUSTER_GENERATOR_CONFIG
)
The earlier command which involved installing packages by reading from file was working in Composer1 (Airflow 1.x), however failing with Composer 2.x (Airflow 2.x)
From the error, it is clear that you are running old version of attrs package.
run the below command and try
pip install attrs==20.3.0
or
pip install attrs==20.1.0

Deployment Failed with Error: Package deployment using ZIP Deploy failed

I am trying to deploy a django project in azure from GitHub repository but I am getting error during the deployment. I am getting this error:
Failed to deploy web package to App Service.
2022-01-02T15:32:52.2053869Z ##[error]Deployment Failed with Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
How can I resolve it?
2022-01-02T15:07:42.4101966Z with:
2022-01-02T15:07:42.4102399Z app-name: stockprojection
2022-01-02T15:07:42.4102889Z slot-name: Production
2022-01-02T15:07:42.4115035Z publish-profile: ***
2022-01-02T15:07:42.4115476Z package: .
2022-01-02T15:07:42.4116058Z ##[endgroup]
2022-01-02T15:07:43.9633760Z Package deployment using ZIP Deploy initiated.
2022-01-02T15:32:51.1581899Z Updating submodules.
2022-01-02T15:32:51.1583435Z Preparing deployment for commit id 'f47d703f2a'.
2022-01-02T15:32:51.1584587Z Repository path is /tmp/zipdeploy/extracted
2022-01-02T15:32:51.1585520Z Running oryx build...
2022-01-02T15:32:52.1954843Z Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.9 -i /tmp/8d9ce01a48e2911 --compress-destination-dir -p virtualenv_name=antenv --log-file /tmp/build-debug.log
2022-01-02T15:32:52.1957351Z Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
2022-01-02T15:32:52.1958337Z You can report issues at https://github.com/Microsoft/Oryx/issues
2022-01-02T15:32:52.1992563Z
2022-01-02T15:32:52.1993572Z Oryx Version: 0.2.20210826.1, Commit: f8651349d0c78259bb199593b526450568c2f94a, ReleaseTagName: 20210826.1
2022-01-02T15:32:52.1994721Z
2022-01-02T15:32:52.1995241Z Build Operation ID: |da22zDrMO/Q=.cac2beba_
2022-01-02T15:32:52.1996043Z Repository Commit : f47d703f2a594452a83ae4301b5d6ce8
2022-01-02T15:32:52.1996497Z
2022-01-02T15:32:52.1997051Z Detecting platforms...
2022-01-02T15:32:52.1997609Z Detected following platforms:
2022-01-02T15:32:52.1998100Z python: 3.9.7
2022-01-02T15:32:52.1999279Z Version '3.9.7' of platform 'python' is not installed. Generating script to install it...
2022-01-02T15:32:52.1999807Z
2022-01-02T15:32:52.2000549Z Using intermediate directory '/tmp/8d9ce01a48e2911'.
2022-01-02T15:32:52.2001012Z
2022-01-02T15:32:52.2001533Z Copying files to the intermediate directory...
2022-01-02T15:32:52.2002087Z Done in 0 sec(s).
2022-01-02T15:32:52.2002345Z
2022-01-02T15:32:52.2002782Z Source directory : /tmp/8d9ce01a48e2911
2022-01-02T15:32:52.2003367Z Destination directory: /home/site/wwwroot
2022-01-02T15:32:52.2003726Z
2022-01-02T15:32:52.2003938Z
2022-01-02T15:32:52.2004699Z Downloading and extracting 'python' version '3.9.7' to '/tmp/oryx/platforms/python/3.9.7'...
2022-01-02T15:32:52.2005361Z Downloaded in 1 sec(s).
2022-01-02T15:32:52.2005817Z Verifying checksum...
2022-01-02T15:32:52.2006290Z Extracting contents...
2022-01-02T15:32:52.2006715Z Done in 24 sec(s).
2022-01-02T15:32:52.2006959Z
2022-01-02T15:32:52.2007462Z Python Version: /tmp/oryx/platforms/python/3.9.7/bin/python3.9
2022-01-02T15:32:52.2008171Z Creating directory for command manifest file if it doesnot exist
2022-01-02T15:32:52.2008806Z Removing existing manifest file
2022-01-02T15:32:52.2009356Z Python Virtual Environment: antenv
2022-01-02T15:32:52.2009917Z Creating virtual environment...
2022-01-02T15:32:52.2010491Z Activating virtual environment...
2022-01-02T15:32:52.2011006Z Running pip install...
2022-01-02T15:32:52.2011541Z [15:08:42+0000] Collecting beautifulsoup4==4.10.0
2022-01-02T15:32:52.2012658Z [15:08:42+0000] Downloading beautifulsoup4-4.10.0-py3-none-any.whl (97 kB)
2022-01-02T15:32:52.2013461Z [15:08:43+0000] Collecting Django==3.2.7
2022-01-02T15:32:52.2014278Z [15:08:43+0000] Downloading Django-3.2.7-py3-none-any.whl (7.9 MB)
2022-01-02T15:32:52.2015027Z [15:08:45+0000] Collecting djangorestframework==3.13.0
2022-01-02T15:32:52.2016199Z [15:08:45+0000] Downloading djangorestframework-3.13.0-py3-none-any.whl (958 kB)
2022-01-02T15:32:52.2018261Z [15:08:46+0000] Collecting matplotlib==3.4.2
2022-01-02T15:32:52.2019288Z [15:08:48+0000] Downloading matplotlib-3.4.2-cp39-cp39-manylinux1_x86_64.whl (10.3 MB)
2022-01-02T15:32:52.2020051Z [15:08:49+0000] Collecting nltk==3.6.2
2022-01-02T15:32:52.2020817Z [15:08:49+0000] Downloading nltk-3.6.2-py3-none-any.whl (1.5 MB)
2022-01-02T15:32:52.2021432Z [15:08:50+0000] Collecting nsepy==0.8
2022-01-02T15:32:52.2022116Z [15:08:50+0000] Downloading nsepy-0.8.tar.gz (33 kB)
2022-01-02T15:32:52.2022924Z [15:08:53+0000] Collecting numpy==1.19.5
2022-01-02T15:32:52.2023827Z [15:08:54+0000] Downloading numpy-1.19.5-cp39-cp39-manylinux2010_x86_64.whl (14.9 MB)
2022-01-02T15:32:52.2024538Z [15:08:57+0000] Collecting pandas==1.3.1
2022-01-02T15:32:52.2025512Z [15:08:57+0000] Downloading pandas-1.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
2022-01-02T15:32:52.2026293Z [15:08:59+0000] Collecting requests==2.26.0
2022-01-02T15:32:52.2027195Z [15:08:59+0000] Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-01-02T15:32:52.2027944Z [15:09:01+0000] Collecting scikit_learn==1.0.1
2022-01-02T15:32:52.2028928Z [15:09:01+0000] Downloading scikit_learn-1.0.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (24.7 MB)
2022-01-02T15:32:52.2029713Z [15:09:07+0000] Collecting tensorflow==2.5.0
2022-01-02T15:32:52.2032336Z WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/dc/0e/72070fe6d5774bfc6b73e857f1747ba99b95cb808c46b709ab7c9ec757eb/tensorflow-2.5.0-cp39-cp39-manylinux2010_x86_64.whl
2022-01-02T15:32:52.2034884Z [15:09:23+0000] Downloading tensorflow-2.5.0-cp39-cp39-manylinux2010_x86_64.whl (454.4 MB)
2022-01-02T15:32:52.2035989Z An unknown error has occurred. Check the diagnostic log for details.
2022-01-02T15:32:52.2040696Z ##[error]Failed to deploy web package to App Service.
2022-01-02T15:32:52.2053869Z ##[error]Deployment Failed with Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2022-01-02T15:32:52.2056058Z App Service Application URL: http://stockprojection.azurewebsites.net```

Errors when trying to call pycurl in a lambda on AWS

I want to use pycurl in order to have TTFB and TTLB, but am unable to call pycurl in an AWS lambda.
To focus on the issue, let say I call this simple lambda function:
import json
import pycurl
import certifi
def lambda_handler(event, context):
client_curl = pycurl.Curl()
client_curl.setopt(pycurl.CAINFO, certifi.where())
client_curl.setopt(pycurl.URL, "https://www.arolla.fr/blog/author/edouard-gomez-vaez/") #set url
client_curl.setopt(pycurl.FOLLOWLOCATION, 1)
client_curl.setopt(pycurl.WRITEFUNCTION, lambda x: None)
content = client_curl.perform()
dns_time = client_curl.getinfo(pycurl.NAMELOOKUP_TIME) #DNS time
conn_time = client_curl.getinfo(pycurl.CONNECT_TIME) #TCP/IP 3-way handshaking time
starttransfer_time = client_curl.getinfo(pycurl.STARTTRANSFER_TIME) #time-to-first-byte time
total_time = client_curl.getinfo(pycurl.TOTAL_TIME) #last requst time
client_curl.close()
data = json.dumps({'dns_time':dns_time,
'conn_time':conn_time,
'starttransfer_time':starttransfer_time,
'total_time':total_time,
})
return {
'statusCode': 200,
'body': data
}
I have the following error, which is understandable:
Unable to import module 'lambda_function': No module named 'pycurl'
I followed the tuto https://aws.amazon.com/fr/premiumsupport/knowledge-center/lambda-layer-simulated-docker/ in order to create a layer, but then have the following error while generated the layer with docker (I extracted the interesting part):
Could not run curl-config: [Errno 2] No such file or directory: 'curl-config': 'curl-config'
I even tried to generate the layer just launching on my own machine:
pip install -r requirements.txt -t python/lib/python3.6/site-packages/
zip -r mypythonlibs.zip python > /dev/null
And then uploading the zip as a layer in aws, but I then have another error when lanching the lambda:
Unable to import module 'lambda_function': libssl.so.1.0.0: cannot open shared object file: No such file or directory
It seems that the layer has to be built on a somehow extended target environment.
After a couple of hours scratching my head, I managed to resolve this issue.
TL;DR: build the layer by using a docker image inherited from the aws one, but with the needed libraries, for instance libcurl-devel, openssl-devel, python36-devel. Have a look at the trick Note 3 :).
The detailed way:
Prerequisite: having Docker installed
In a empty directory, copy your requirements.txt containing pycurl (in my case: pycurl~=7.43.0.5)
In this same directory, create the following Dockerfile (cf Note 3):
FROM public.ecr.aws/sam/build-python3.6
RUN yum install libcurl-devel python36-devel -y
RUN yum install openssl-devel -y
ENV PYCURL_SSL_LIBRARY=openssl
RUN ln -s /usr/include /var/lang/include
Build the docker image:
docker build -t build-python3.6-pycurl .
build the layer using this image (cf Note 2), by running:
docker run -v "$PWD":/var/task "build-python3.6-pycurl" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"
Zip the layer by running:
zip mylayer.zip python > /dev/null
Send the file mylayer.zip to aws as a layer and make your lambda points to it (using the console, or following the tuto https://aws.amazon.com/fr/premiumsupport/knowledge-center/lambda-layer-simulated-docker/).
Test your lambda and celebrate!
Note 1. If you want to use python 3.8, just change 3.6 or 36 by 3.8 and 38.
Note 2. Do not forget to remove the python folder when regenerating the layer, using admin rights if necessary.
Note 3. Mind the symlink in the last line of the DockerFile. Without it, gcc won't be able to find some header files, such as Python.h.
Note 4. Compile pycurl with openssl backend, for it is the ssl backend used in the lambda executing environment. Or else you'll get a libcurl link-time ssl backend (openssl) is different from compile-time ssl backend error when executing the lambda.

pip install breaks due to SSL certs

When I try to do a pip install Flask I get some ssl errors. If I try to add exceptions there is no difference. I googled around and see some discussion on this from a year ago but nothing else (see https://github.com/pypa/pip/issues/5063)
(venv) pip install -U flask --trusted-host=pypi.python.org --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
Collecting flask
Could not fetch URL https://pypi.python.org/simple/flask/: There was a problem confirming
the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement flask (from versions: )
No matching distribution found for flask
NOTE : THIS MIGHT ALREADY SOMETHING THAT YOU HAVE TRIED
This is probably due to the fact that you are using --trusted-host=pypi.python.org .
This has happened since sometime during April 2018, the Python Package Index was migrated from pypi.python.org to pypi.org. This means "trusted-host" commands using the old domain no longer work.
So the command you are looking for would be pip install -U flask --trusted-host pypi.org --trusted-host=files.pythonhosted.org.
For furthur details , have a look at this this.

File not found error while testing python impyla

I am trying to set up a connection between python and impala. Based on the instructions here I am trying to set up impyla.
I am on a vagrant ubuntu/xenial64 box with python 2.7.12. After reading about some issues with the latest thrift I downgraded to the specified version. After pip installing and setting up the environment variables, I am trying to run the test but it is failing with the file not found error, as follows:
ubuntu#ubuntu-xenial:~/.local/lib/python2.7/site-packages/impala$ py.test --connect impyla
======================================== test session starts ========================================
platform linux2 -- Python 2.7.12, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: /home/ubuntu/.local/lib/python2.7/site-packages/impala, inifile:
=================================== no tests ran in 0.00 seconds ====================================
ERROR: file not found: impyla
ubuntu#ubuntu-xenial:~/.local/lib/python2.7/site-packages/impala$
I think I am missing out on something very fundamental here. I have no previous experience with python tests. I would like to stick with python 2.7 because of some other dependencies.
PS:
Please note the latest version provided in the repo readme is 0.13.1 but with my pip install impyla it came out to be 0.14.0.
I am running the test from my site-package/impala directory as the readme says to be in the directory where impyla is.
There is no mention of any inifile in the readme (as far as I could see.)
EDIT 1:
When I run it from site-package directory without --connect I get the same error. But with the argument, it says that the argument is unrecognizable. The output is as follows:
ubuntu#ubuntu-xenial:~/.local/lib/python2.7/site-packages$ py.test --connect impyla
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --connect
inifile: None
rootdir: /home/ubuntu/.local/lib/python2.7/site-packages
ubuntu#ubuntu-xenial:~/.local/lib/python2.7/site-packages$
Any help on how to troubleshoot this would be helpful.