Psycopg2 Error When Running Django Via Serverless - django

Question

File "/var/task/django/db/backends/postgresql/base.py", line 29, in
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: libpq.so.5: cannot open shared object file: No such file or directory
I'm receiving the following error after deploying a Django application via Serverless. If I deploy the application via our Bitbucket Pipeline.
Here's the Pipeline:
- step: &Deploy-serverless
caches:
- node
image: node:11.13.0-alpine
name: Deploy Serverless
script:
# Initial Setup
- apk add curl postgresql-dev gcc python3-dev musl-dev linux-headers libc-dev
# Load our environment.
...
- apk add python3
- npm install -g serverless
# Set Pipeline Variables
...
# Configure Serverless
- cp requirements.txt src/requirements.txt
- printenv > src/.env
- serverless config credentials --provider aws --key ${AWS_KEY} --secret ${AWS_SECRET}
- cd src
- sls plugin install -n serverless-python-requirements
- sls plugin install -n serverless-wsgi
- sls plugin install -n serverless-dotenv-plugin
Here's the Serverless File:
service: serverless-django
plugins:
- serverless-python-requirements
- serverless-wsgi
- serverless-dotenv-plugin
custom:
wsgi:
app: arc.wsgi.application
packRequirements: false
pythonRequirements:
dockerFile: ./serverless-dockerfile
dockerizePip: non-linux
pythonBin: python3
useDownloadCache: false
useStaticCache: false
provider:
name: aws
runtime: python3.6
stage: dev
region: us-east-1
iamRoleStatements:
- Effect: "Allow"
Action:
- s3:GetObject
- s3:PutObject
Resource: "arn:aws:s3:::*"
functions:
app:
handler: wsgi.handler
events:
- http: ANY /
- http: "ANY {proxy+}"
timeout: 60
Here's the Dockerfile:
FROM lambci/lambda:build-python3.7
RUN yum install -y postgresql-devel python-psycopg2 postgresql-libs
And here's the requirements:
amqp==2.6.1
asgiref==3.3.1
attrs==20.3.0
beautifulsoup4==4.9.3
billiard==3.6.3.0
boto3==1.17.29
botocore==1.20.29
celery==4.4.7
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
coverage==5.5
Django==3.1.7
django-cachalot==2.3.3
django-celery-beat==2.2.0
django-celery-results==2.0.1
django-filter==2.4.0
django-google-analytics-app==5.0.2
django-redis==4.12.1
django-timezone-field==4.1.1
djangorestframework==3.12.2
Djaq==0.2.0
drf-spectacular==0.14.0
future==0.18.2
idna==2.10
inflection==0.5.1
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
kombu==4.6.11
livereload==2.6.3
lunr==0.5.8
Markdown==3.3.4
MarkupSafe==1.1.1
mkdocs==1.1.2
nltk==3.5
psycopg2-binary==2.8.6
pyrsistent==0.17.3
python-crontab==2.5.1
python-dateutil==2.8.1
python-dotenv==0.15.0
pytz==2021.1
PyYAML==5.4.1
redis==3.5.3
regex==2020.11.13
requests==2.25.1
sentry-sdk==1.0.0
six==1.15.0
soupsieve==2.2
sqlparse==0.4.1
structlog==21.1.0
tornado==6.1
tqdm==4.59.0
uritemplate==3.0.1
urllib3==1.26.3
uWSGI==2.0.19.1
vine==1.3.0
Werkzeug==1.0.1
And here's the database settings:
# Database Defintions
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql_psycopg2",
"HOST": load_env("PSQL_HOST", "127.0.0.1"),
"NAME": load_env("PSQL_DATABASE", ""),
"PASSWORD": load_env("PSQL_PASSWORD", ""),
"USER": load_env("PSQL_USERNAME", ""),
"PORT": load_env("PSQL_PORT", "5432"),
"TEST": {
"NAME": "arc_unittest",
},
},
}
Am at a loss for what exactly the issue is. Thoughts?
File "/var/task/django/db/backends/postgresql/base.py", line 29, in
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2._psycopg'
I receive this similar error when deploying locally.

Answer 1

In my case, I needed to replace the psycopg2-binary with aws-psycopg2 to be Lambda friendly.

Answer 2

Apparently it is a dependency error, in debian base distributions you solve it by installing the libpq-dev package

Answer 3

You can also use a lambda layer and use one already created in this repo
At the bottom of you Lambda console, you have a "Layers" section where you can click "Add Layer" and specify the corresponding ARN based on your Python version and AWS region.
It will also have the benefit of reducing your package size.