How do I install fontconfig on AWS lambda? - amazon-web-services

This is a continuation of How do you install phantomjs on AWS lambda? I've figured out how to get phantomjs running on an aws lambda, but when I use it to generate pdfs (using the html-pdf nodejs library), the content is missing text. If I create a docker container that's using FROM node:10.16.0-jessie on it, the pdfs render fine. If I create a docker container using FROM amazonlinux:2.0.20190508 (which I think is similar to the AWS lambda container), the text is missing on my PDFs.
I've fixed this problem in amazonlinux:2.0.20190508 by running yum install fontconfig. But, I don't know how to do the equivalent of a yum install fontconfig inside a real lambda. If you look at the link above, you'll see that an answer there attempts to provide that information, but for whatever reason, it still doesn't work correctly. I believe the reason is there's still a missing step on how to get the fontconfig install properly extracted from the amazonlinux:2.0.20190508 container.
In summary, here is my question: After I run yum install fontconfig in amazonlinux:2.0.20190508, how do I extract it from the container and package it up so that an AWS Lambda can use it?
By the way, I'm sure there are other answers that seem to be answering this question, but the AWS lambda built-in dependencies change so frequently, none of those answers work anymore.

In my case, I did this:
Create a Docker file with content:
FROM amazonlinux:2.0.20190508
RUN yum -y install fontconfig freetype
and build it as example:latest
Run docker and mount folder:
docker run -v D:\dockerFiles:/mnt --rm -it example:latest
Found libraries inside docker by path /lib64
libbz2.so.1 libexpat.so.1 libfontconfig.so.1 libfreetype.so.6 libpng15.so.15
Copy it to /mnt/lib. Take lib from D:\dockerFiles and zip folder lib.
Create aws lambda layer and add to lambda.

Related

Node.JS native addons on LINUX [duplicate]

I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.
This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.
How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?
While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.
For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.
Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
For extra comfort, the script can also create the zip file for deployment
Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
Deploy!
It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.
While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.
You can read the original answer here: https://stackoverflow.com/a/34019739/303184
Steps to make it work:
Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.
Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.
You will probably need to install git & g++ compiler on the EC2. You can do this running
sudo yum install git gcc-c++
Finally, clone your app to your new EC2 and install your app's dependecies:
nvm use 0.10.36
npm install --production
You can then easily download the node_modules using scp or such.
Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.
AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/

How to have Aspose.Cells 19.10 correctly render fonts in Linux .Net Service?

I am building a .net service in docker that will take .xlsx files and render them as .tiff. When I run my service locally on a windows environment the images correctly render with Times New Roman text. When I spin up a docker container and submit the same file it returns a blank .tiff image.
I have tried numerous approaches to have Aspose.Cells work correctly in the docker container based off the Aspose documentation relating to setting the default font source, default font folder, and font substitution but none of the suggested methods work. Below I listed the approach that I believe should work with Aspose.Cells
Installed fonts-liberation fontconfig and ttf-mscorefonts-installer on the docker container to get access to fonts located in /usr/share/fonts/truetype/
In the service I have set my source to the following:
FontConfigs.SetFontFolder("/usr/share/fonts/truetype/", true);
The true in this statement will scan subfolders to find the .ttf font files. I expect Aspose to use this file path in the docker container to get needed fonts for rendering.
I verified that I have fonts available in the "/usr/share/fonts/truetype/" folder under sub-folders of liberation and msttcorefonts.
Can anyone offer advice of another approach or potential issue I am running into? I can provide more information if needed as well.
Well, there are some guidelines using Aspose.Cells (especially for rendering features) to work on docker container or similar environments which you should refer to. We also recommend you to kindly try using latest version/fix (e.g Aspose.Cells v20.3)
Here are our steps:
The docker we used was: microsoft/dotnet, install it first:
sudo docker pull microsoft/dotnet
When running it, we got an issue in this docker when saving to PDF or image.
We did the following things to sort it out:
We installed libgdiplus:
apt-get update
apt-get install -y libgdiplus
cd /usr/lib && ln -s libgdiplus.so gdiplus.dll
And install libc6-dev:
apt-get install -y --no-install-recommends libc6-dev
Hope, this helps a bit.
PS. I am working as Support developer/ Evangelist at Aspose.

Script works on AWS EC2, but not on AWS Lambda after zipping

I am creating a simple AWS Lambda function using M2Crypto library. I followed the steps for creating deployment package from here. The lambda function works perfectly on an EC2 Linux instance (AMI).
This is my Function definition:
CloudOAuth.py
from M2Crypto import BIO, RSA, EVP
def verify(event, context):
pem = "-----BEGIN PUBLIC KEY-----\n{0}\n-----END PUBLIC KEY-----".format("hello")
bio = BIO.MemoryBuffer(str.encode(pem))
print(bio)
return
Deployment Package structure:
When I run the Lambda, I get the following issue and I also tried including libcrypto.so.10 from /lib64 directory, but didn't help.
Issue when running Lambda
/var/task/M2Crypto/_m2crypto.so: symbol sk_deep_copy, version libcrypto.so.10 not defined in file libcrypto.so.10 with link time reference`
Python: 2.7
M2Crypto: 0.27.0
I would guess that the M2Crypto was built with different version of OpenSSL than what's on Lambda. See the relevant code. If not (the upstream maintainer speaking here), please, file a bug at https://gitlab.com/m2crypto/m2crypto/issues
I just want to add some more details on to #mcepl's answer. The most important is that OpenSSL version on AWS Lambda and the environment (in my case ec2) where you build your M2Crypto library should match.
To check openssl version on Lambda, use print in your handler:
print(ssl.OPENSSL_VERSION)
To check openssl version on your build environment, use:
$ openssl version
Once they match, it works.
Don't hesitate to downgrade or upgrade OpenSSL on your build environment to match the Lambda environment. I had to downgrade my openssl on ec2 to match lambda runtime environment.
sudo yum -y downgrade openssl-devel-1.0.1k openssl-1.0.1k
Hope it will help anyone trying to use M2Crypto :)
copying my answer for a similar question here:
AWS lambda runs code on an old version of amazon linux (amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2) as mentioned in the official documentation
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
So to run a code that depends on shared libraries, it needs to be compiled in the same environment so it can link correctly.
What I usually do in such cases is that I create virtualenv using docker container. The virtualenv can than be packaged with lambda code.
Please note that if you need install anything using yum (in the docker container), you must use same release server as the amazon linux version:
yum --releasever=2017.03 install ...
virtualenv can be built using an EC2 instance as well instead of docker container (though, I find docker method easier). Just make sure that the AMI used for EC2 is same as the one used by lambda.

AWS Lambda : How to use Pillow library?

I'm trying to create an AWS lambda function in order to create thumbnail of my uploaded images.
My script is running well locally, I followed this tutorial to deploy my function but I have a problem with the Pillow library, indeed when I'm testing my function I can see this following log :
I found this post with the same issue but in my case I can't execute command line on the machine.
You must include the libjpeg.so in your lambda package, but it will also require some tweaking with the patchelf utility. Assuming that you prepare the lambda package via "pip install module-name -t" (rather than via virtualenv), do the following:
cd into/your/local/lambda/package/dir
cp -L $(ldd PIL/_imaging.so|grep libjpeg|awk '{print $3}') PIL/
patchelf --set-rpath PIL PIL/_imaging.so
# zip, deploy and test the package
This script works for Pillow version 3.2.0.
Regarding patchelf: under Ubuntu it can be 'apt install'ed, but under other Linuxes it may need to be built from source.
The problem here is that Pillow uses native libraries that must be built for the exact correct environment.
I solved this by installing my requirements in a Docker container that replicates very closely the AWS Lambda environment, lambci/lambda. I used the build-python3.8 version.
I installed my requirements there and zipped up the whole contents of /var/lang/lib/python3.8/site-packages/ directory along with my lambda function file.
I tried this with a standard Amazon Linux Docker image and it didn't work. Only the lambci/lambda image worked for me.

How to install audiowaveform program on AWS Elastic Beanstalk

Just FYI ... context here is AWS Elastic Beanstalk. I'm trying to the install audiowaveform program on 64bit Amazon Linux 2015.03 v1.4.3 (the customer AMI ID is ami-6b50291c). Running this ... 👇
$ sudo yum install git cmake libmad-devel libsndfile-devel gd-devel boost-devel
... successfully installs all packages except libmad-devel and libsndfile-devel. Below is the relevant output ...
Failed to set locale, defaulting to C
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/2015.03 | 2.1 kB 00:00
amzn-updates/2015.03 | 2.3 kB 00:00
Package git-2.1.0-1.38.amzn1.x86_64 already installed and latest version
Package cmake-2.8.12-2.20.amzn1.x86_64 already installed and latest version
No package libmad-devel available.
No package libsndfile-devel available.
Package gd-devel-2.0.35-11.10.amzn1.x86_64 already installed and latest version
Package boost-devel-1.53.0-14.21.amzn1.x86_64 already installed and latest version
Nothing to do
That said, this is not a problem with audiowaveform ... all this means is that the repositories enabled for Amazon Linux AMIs do not have libmad-devel and libsndfile-devel by default. I probably have to simply add my own sources I guess.
Also to note is that no yum packages exist for audio waveform so I have to build this manually.
Obtain the source ... 👇
$ git clone https://github.com/bbcrd/audiowaveform.git
$ cd audio waveform
Then build and install ... 👇
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
Question 1
On AWS EB ... the EC2 instances are configured to use Amazon sources which don't have the above packages i.e. libmad-devel and libsndfile-devel. What would be the recommended approach to adding these packages so that they are available to yum?
I stress recommended because I feel that changing the sources from Amazon's could not be the best approach. Nor is adding another source that could conflict with Amazon's packages ... etc etc etc ...
Question 2
Assuming I'm able to install libmad-devel and libsndfile-devel. I still have to build this manually since there are no packages of audiowaveform. On AWS EB I could write a script to do this as each instance is being instantiated ... but I feel this isn't ideal, slow and kinda error-prone. Anyone have advice on how I can do this better?
Probably prepare an AMI with this already built that's based off ami-6b50291c. Thoughts?
Core Objective
I don't have to use audiowaveform ... my objective really is to extract the peak points of some audio (MP3). I will set this up as a separate question.
Amazon Elastic Beanstalk tends to be very restricted in terms of what software you can install on it. I solved it by dockerizing my application environment. This is possible now even on Elastic Beanstalk.
Learn more about Elastic Beanstalk's support for Docker ...
AWS Elastic Beanstalk makes it easy for you to deploy and manage
applications in the AWS cloud. After you upload your application,
Elastic Beanstalk will provision, monitor, and scale capacity (Amazon
EC2 instances), while also load balancing incoming requests across all
of the healthy instances.
Docker automates the deployment of applications in the form of
lightweight, portable, self-sufficient containers that can run in a
variety of environments. Containers can be populated from pre-built
Docker images or from a simple recipe known as a Dockerfile.
Docker’s container-based model is very flexible. You can, for example,
build and test a container locally and then upload it to the AWS Cloud
for deployment and scalability. Docker’s automated deployment model
ensures that the runtime environment for your application is always
properly installed and configured, regardless of where you decide to
host the application.
This way ... you can do whatever you want in the container and that container will run on the kernel provided by the Amazon Linux AMI instance (obviously completely isolated).
I'm also somehow having hard time getting yum to find libsndfile on Amazon Linux AMI (RedHat 7.4). Repositories I've added to yum never seem to contain it. (How to add new repos is described here )
Finally I just downloaded and installed the rpms directly:
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/x86_64/RPMS.classic//libsndfile-1.0.28-alt1.x86_64.rpm
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/x86_64/RPMS.classic//libsndfile-devel-1.0.28-alt1.x86_64.rpm
sudo yum localinstall libsndfile-devel-1.0.28-alt1.x86_64.rpm
This way I got PySoundfile working finally.