Deploy Pytidylib module in aws lambda by using Lamda Layers - amazon-web-services

Question

I am trying to deploy pytidylib python module into AWS lambda function by using layers .
I have created the path as described in aws docs and created new layer.
Now the code of pytidylib needs some libraries from /usr/lib but i have installed libraries in /python/lib/python3.7/site-packages/ , so to resolve this i added the path in environ PATH of aws linux server platform , but still the issue is not resolved.
Below is my code :-
def lambda_handler(event, context):
"""Read file from s3 on trigger."""
s3 = boto3.client("s3")
print(sys.platform)
ld_library_path = os.environ["LD_LIBRARY_PATH"]
print("old ld_library_path is ",ld_library_path)
ld_library_path = ld_library_path +
":/opt/python/lib/python3.7/site-packages/"
os.environ["LD_LIBRARY_PATH"] = ld_library_path
print("ld_library_path after set is
",os.environ["LD_LIBRARY_PATH"])
ld_library_path after set is /var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib:/opt/python/lib/python3.7/site-packages/
I want to understand is there any way i can make this work through some changes in code and make the pytidylib module run through layers .
Below is the error:-
[ERROR] OSError: Could not load libtidy using any of these names:
libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 68, in lambda_handler
document, errors = tidy_document(doc)
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 222, in tidy_document
return get_module_tidy().tidy_document(text, options)
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 234, in get_module_tidy
_tidy = Tidy()
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 99, in __init__
+ ",".join(lib_names))

Answer 1

I tried to replicate your issue and for me Pytidylib layer works as expected.
This is the way I used to construct the layer, in case you want to give it a try. It involves a docker tool described in the recent AWS blog:
How do I create a Lambda layer using a simulated Lambda environment with Docker?
I created Pytidylib layer as follows:
Create empty folder, e.g. mylayer.
Go to the folder and create requirements.txt file with the content of
Pytidylib
Run the following docker command (may adjust python version to your needs):
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
Create layer as zip:
zip -r pytidylayer.zip python > /dev/null
Create lambda layer based on pytidylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.
Add the layer to the lambda and test using the following lambda function:
import json
import tidylib
def lambda_handler(event, context):
print(dir(tidylib))
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
The function executed correctly:
['PersistentTidy', 'Tidy', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'release_tidy_doc', 'sink', 'tidy', 'tidy_document', 'tidy_fragment']

Answer 2

I solved this by adding path of the tidy library (libtidy.so.5.2.0) into the environment variable of LD_LIBRARY_PATH of linux server
For me library was pre installed in ubuntu 18.04 server in /usr/lib .Copy the library from this path ,place it inside the tidylib folder create a zip, and follow the steps of lambda-layers creation .