I'm trying to use the Sharp library in AWS Lambda but it requires the module be compiled for the lambda environment. The instructions say to create an ec2 instance and compile it on there - but I noticed that there are a few tools to help with that but they are all at least a year old with no maintenance. Is there a package that comes with Serverless, or something that's considered the standard way now?
I've found these but they are all at least a year old since a commit
https://github.com/node-hocus-pocus/thaumaturgy
https://github.com/Max-Kolodezniy/aws-lambda-build
https://github.com/tomdale/lambda-packager
Maybe there is a directory somewhere where I can just download a precompiled Sharp library for AWS lambda?
I made it work using sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz tarball, that was created on AWS EC2 instance running Nodejs 6.10.1. The tarball contains node_modules/ directory with sharp system binaries (libvips library) specific to the Lambda execution environment.
Project structure
To avoid conflicts between my local node_modules/ (Nodejs 7.5 on Mac) and node_modules/ inside the tarball (Nodejs 6.10 on Linux), I'm creating my Lambda service under a subdirectory.
Project structure looks as follows:
node_modules/
service/
node_modules/ <= sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz
utils/
handler.js
package.json <= engines: node 6.10.1
serverless.yml
src/
jasmine.json
package.json
Most of the dependencies I need are for development and testing purpose. These are maintained inside root package.json file (also includes sharp, but compiled for my Nodejs 7.5 environment, offering to test image manipulations locally).
My service/handler.js and service/utils/ contains ES6 compatible source code with Lambda function handler – it is transpiled from src/ directory.
If I need other dependencies for production (besides sharp), I install them to services/package.json using --prefix option. But not aws-lambda, neither aws-sdk – they are globally installed within Lambda, meaning no need to include them in deployable .zip file.
npm i -S lodash --prefix services/
It ensures installation of lodash version compatible with Lambda environment, because service/package.json defines Nodejs version to rely on:
{
"private": true,
"engines": { "node" : "6.10.1" },
"dependencies": {
...
}
}
However, there's a nuance — other production dependencies doesn't have to be environment dependent. If so, they won't work, because you install them from your local machine, which isn't equal to Lambda's one.
Lambda function deployment
Since Lambda requires .zip archive, I compress contents of my service/ directory. And my Lambda functions works. Everything is ES6 compatible, sharp has Lambda environment binaries and my other production dependency versions correlates with Nodejs 6.10.1.
Additionally, I'd suggest to use Serverless ⚡️ (I use it too). It dramatically simplifies Lambda functions development and deployment.
Nik's answer definitely helped me to get to a working solution! One thing that I would add is that the people behind serverless-sharp-image updated their package so the tarball works with node v6.10 now so I don't see a reason to have two different node environments referenced. I do everything in v6.10.
https://github.com/adieuadieu/serverless-sharp-image/tree/master/lib
Had similar problem and managed to install binaries for Linux x64 platform by
npm install --arch=x64 --platform=linux --target=8.10.0 sharp
Then just upload Lambda as usual and it works just fine.
Above works on Mac as well as windows and details are in documentation at http://sharp.pixelplumbing.com/en/stable/install/#aws-lambda
For anyone stumbling upon this post now. I've accomplished this by copying my package.json file into an AWS Cloud9 IDE and simply running npm install. From there, just download the node_modules/ folder.
Related
I am working on an AWS SAM serverless project, on a Lambda function written in Node.js.
The Lambda execution environment already provides the AWS SDK, this it is not necessary to push this dependency into the deployment.
The problem arises when aws-sdk comes up as a nested dependency of another package.
For instance, I need aws-appsync, which depends in turn on aws-sdk.
Because of that, the deployment size is too big. The entire aws-appsync package with its dependency weights about 140mb, a notable portion of it being the AWS SDK. In this situation, the maximum deployment size is exceeded and the deployment procedure fails.
Can I make npm install a package with all its dependencies except a specific one? I would exclude aws-sdk from the dependencies in this case.
An easy solution for this is to add the aws-sdk as a devDependencies instead of a normal dependency. You can the run npm i --production in your deployment pipeline before bundling the code and uploading it. This will ensure that the devDependencies are not available in the node_modules. (I don't think it will remove them if they are there already, so if you're doing it locally, you might have to delete the node_modules folder before running the npm command)
I am using webpack and serverless to deploy to aws lambda. So far I have been able to configure it to bundle all dependencies into one ts file, but aws complains there is no package.json. So, I found a way to upload the node modules folder as well, which also brought in the package.json but since I am on windows the aws instances don't like the libraries.
How do I include package.json when I run the serverless package or deploy commands so that aws lambda can run the install?
include:
- package.json
Doesn't work.
If you're using the Serverless Webpack plugin, you should be able to get whatever native modules you need installed by using the packagerOptions config for the plugin and specifying the linux platform for x64 architecture along with the list of npm modules to package.
See the Custom scripts section of the plugin's documentation for more info.
For example, if your Lambda function depends on the sharp npm package, you'd add something like the following to your serverless.yml file:
custom:
webpack:
includeModules: true
packagerOptions:
scripts:
- npm_config_platform=linux npm_config_arch=x64 yarn add sharp
I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.
This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.
How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?
While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.
For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.
Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
For extra comfort, the script can also create the zip file for deployment
Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
Deploy!
It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.
While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.
You can read the original answer here: https://stackoverflow.com/a/34019739/303184
Steps to make it work:
Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.
Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.
You will probably need to install git & g++ compiler on the EC2. You can do this running
sudo yum install git gcc-c++
Finally, clone your app to your new EC2 and install your app's dependecies:
nvm use 0.10.36
npm install --production
You can then easily download the node_modules using scp or such.
Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.
AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/
I wish to implement the answer that is outlined here: https://stackoverflow.com/a/50397276/1980516
However, I find that I keep running into Unable to import module 'index' at exactly this line:
const _archiver = require('archiver');
So, I'm guessing that I cannot do this via the online console. Instead, I probably have to create a deployment package.
How do I go about this? I apparently need AWS CLI, Node.js, npm and I'm new to all of it. In the Amazon docs I can't find a practical list of how to set up my local development environment.
What tools do I install, which versions and in what order exactly?
Edit: Windows :)
My guess is that you need to npm install archiver and package the node_modules dependencies along with your index.js (handler file for your lambda entry point). You can zip up and deploy/upload it to your lambda.
Also have a look at https://github.com/serverless/serverless framework, that will do these type of things easier.
Have a look at AWS SAM, the Serverless Application Model. It provides a local development setup for things like Lambda functions and API Gateway endpoints, and a way to easily package and deploy things. The exact steps you need are:
Create an AWS account and an IAM user with admin privileges
Install node.js
Install the AWS CLI (and configure it with aws configure)
Install SAM CLI and Docker (the local instances run in docker containers)
Initialize a new SAM project with sam init --runtime nodejs (or other runtime version if need)
Run through the quickstart to get an idea of how to define a SAM template, build a SAM app, and deploy.
If you don't want to use the framework or local development environment and just want to create the source bundle, there are docs. The gist is:
Install nodejs (e.g. using homebrew or an installer)
npm install the modules you need
Zip up your code including the node_modules folder
Upload the zip via the AWS Console
I understand that AWS Lambda runs on the application layer of an isolated environment.
In many situations, functions need to use third-party tools that must be installed first on the linux machine. For example, a media processing function uses exiftool to extract metadata from image, so I install exiftool first.
Now I want to migrate the media processing code into AWS Lambda. My question is, how can I use those tools that I originally must install on linux? My code is written in Java, and exiftool is necessary.
To expand on Daniel's answer, if you wanted to bundle exiftool, you would follow steps 1 and 2 for Unix/Linux platforms from the official install instructions. You would then include exiftool and lib in your function's zip file. To run exiftool you would do something like:
const exec = require('child_process').exec;
exports.handler = (event, context, callback) => {
// './exiftool' gave me permission denied errors
exec('perl exiftool -ver', (error, stdout, stderr) => {
if (error) {
callback(`error: ${error}`);
return;
}
callback(null, `stderr: ${stderr} \n stdout: ${stdout}`);
});
}
Everything your Lambda function executes must be included in the deployment package you upload.
That means if you want to run Java code, you can reference other Java libraries. (Likewise, if you want to run Node.js code, you can reference other Node libraries.)
Regardless of the tools you use, the resulting .zip file must have the following structure:
All compiled class files and resource files at the root level.
All required jars to run the code in the /lib directory.
(source)
Or you can upload a .jar file.
exiftool, on the other hand, is a Perl command-line program. I suspect that on your local machine, you shell out from your Java code and run it.
You cannot do that in AWS Lambda. You need to find a Java package that extracts EXIF information (I am sure there are plenty to choose from) and include that in your deployment package. You cannot install software packages on Lambda.
https://aws.amazon.com/lambda/faqs/
Q: What languages does AWS Lambda support?
AWS Lambda supports code written in Node.js (JavaScript), Python, and Java (Java 8 compatible). Your code can include existing libraries, even native ones. Please read our documentation on using Node.js, Python and Java.
So basically you can call out to native processes if they are pre-installed but only from JavaScript and Java as the parent process.
To get a rough idea of what is installed have a look at what packages are installed:
https://gist.github.com/royingantaginting/4499668
This list won't be a 100% accurate, to do that you would need to look directly at the AMI image (ami-e7527ed7)
exiftool doesn't appear to be installed by default. I doubt the account running the lambda function would have enough rights to install anything globally but you could always bundle exiftool with your Node or Java function.
You may also want to have a look at lambdash (https://github.com/alestic/lambdash) which allows you to run command from your local command line on a remote lamdba instance
This can now be done using AWS Lambda Layers.
An example of how to prepare a layer for exiftool specifically can be found here:
https://gist.github.com/hughevans/6b8c57839b8194ba910428de4375794a