I am using webpack and serverless to deploy to aws lambda. So far I have been able to configure it to bundle all dependencies into one ts file, but aws complains there is no package.json. So, I found a way to upload the node modules folder as well, which also brought in the package.json but since I am on windows the aws instances don't like the libraries.
How do I include package.json when I run the serverless package or deploy commands so that aws lambda can run the install?
include:
- package.json
Doesn't work.
If you're using the Serverless Webpack plugin, you should be able to get whatever native modules you need installed by using the packagerOptions config for the plugin and specifying the linux platform for x64 architecture along with the list of npm modules to package.
See the Custom scripts section of the plugin's documentation for more info.
For example, if your Lambda function depends on the sharp npm package, you'd add something like the following to your serverless.yml file:
custom:
webpack:
includeModules: true
packagerOptions:
scripts:
- npm_config_platform=linux npm_config_arch=x64 yarn add sharp
Related
I am trying to deploy a service to aws using serverless. I am deploying it using gitlab cicd instead of doing it locally. Initially my serverless version was latest(had not mentioned any specific version) but then when I pushed my code to gitlab and i got few errors in the pipeline as the latest version is not stable. So had to change the version to a stable version. Now when i pushed my code changes to gitlab, my deployment failed and i got
Serverless Error ----------------------------------------
Cannot run local installation of the Serverless Framework by the outdated global version. Please upgrade via:
npm install -g serverless Note: Latest release can run any version of the locally installed Serverless Framework.
I dont want to upgrade my serverless version.
in my gitlab-ci.yml i have changed
- npm install -g serverless
to this
- npm install -g serverless#2.69.1
Is there any way I can fix this ?
Any help would be appreciated, thank you.
in your case, the most likely reason for that is the fact that you have a local installation of Serverless or some of the plugins/your other dependencies have Serverless v3 in peer dependencies and install it by default in npm#7 and higher.
To resolve it, either remove local installation or pin the version of the locally installed Serverless (in devDependencies of package.json of your project).
./node_modules/.bin/sls deploy does the trick.
However, the proper answer is in the docs:
There are 2 scenarios:
Using v3 globally, and v2 in specific projects.
This is the simplest. Upgrade the global version to v3, and install v2 in specific projects (via NPM). The serverless command will automatically run the correct version (v3 can run v2).
Using v2 globally, and v3 in specific projects.
To achieve that, install v3 in specific projects (via NPM). Then, use serverless for v2 projects, and npx serverless for v3 projects.
https://www.serverless.com/framework/docs/guides/upgrading-v3#using-v2-and-v3-in-different-projects
I am working on an AWS SAM serverless project, on a Lambda function written in Node.js.
The Lambda execution environment already provides the AWS SDK, this it is not necessary to push this dependency into the deployment.
The problem arises when aws-sdk comes up as a nested dependency of another package.
For instance, I need aws-appsync, which depends in turn on aws-sdk.
Because of that, the deployment size is too big. The entire aws-appsync package with its dependency weights about 140mb, a notable portion of it being the AWS SDK. In this situation, the maximum deployment size is exceeded and the deployment procedure fails.
Can I make npm install a package with all its dependencies except a specific one? I would exclude aws-sdk from the dependencies in this case.
An easy solution for this is to add the aws-sdk as a devDependencies instead of a normal dependency. You can the run npm i --production in your deployment pipeline before bundling the code and uploading it. This will ensure that the devDependencies are not available in the node_modules. (I don't think it will remove them if they are there already, so if you're doing it locally, you might have to delete the node_modules folder before running the npm command)
I wish to implement the answer that is outlined here: https://stackoverflow.com/a/50397276/1980516
However, I find that I keep running into Unable to import module 'index' at exactly this line:
const _archiver = require('archiver');
So, I'm guessing that I cannot do this via the online console. Instead, I probably have to create a deployment package.
How do I go about this? I apparently need AWS CLI, Node.js, npm and I'm new to all of it. In the Amazon docs I can't find a practical list of how to set up my local development environment.
What tools do I install, which versions and in what order exactly?
Edit: Windows :)
My guess is that you need to npm install archiver and package the node_modules dependencies along with your index.js (handler file for your lambda entry point). You can zip up and deploy/upload it to your lambda.
Also have a look at https://github.com/serverless/serverless framework, that will do these type of things easier.
Have a look at AWS SAM, the Serverless Application Model. It provides a local development setup for things like Lambda functions and API Gateway endpoints, and a way to easily package and deploy things. The exact steps you need are:
Create an AWS account and an IAM user with admin privileges
Install node.js
Install the AWS CLI (and configure it with aws configure)
Install SAM CLI and Docker (the local instances run in docker containers)
Initialize a new SAM project with sam init --runtime nodejs (or other runtime version if need)
Run through the quickstart to get an idea of how to define a SAM template, build a SAM app, and deploy.
If you don't want to use the framework or local development environment and just want to create the source bundle, there are docs. The gist is:
Install nodejs (e.g. using homebrew or an installer)
npm install the modules you need
Zip up your code including the node_modules folder
Upload the zip via the AWS Console
I'm trying to use the Sharp library in AWS Lambda but it requires the module be compiled for the lambda environment. The instructions say to create an ec2 instance and compile it on there - but I noticed that there are a few tools to help with that but they are all at least a year old with no maintenance. Is there a package that comes with Serverless, or something that's considered the standard way now?
I've found these but they are all at least a year old since a commit
https://github.com/node-hocus-pocus/thaumaturgy
https://github.com/Max-Kolodezniy/aws-lambda-build
https://github.com/tomdale/lambda-packager
Maybe there is a directory somewhere where I can just download a precompiled Sharp library for AWS lambda?
I made it work using sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz tarball, that was created on AWS EC2 instance running Nodejs 6.10.1. The tarball contains node_modules/ directory with sharp system binaries (libvips library) specific to the Lambda execution environment.
Project structure
To avoid conflicts between my local node_modules/ (Nodejs 7.5 on Mac) and node_modules/ inside the tarball (Nodejs 6.10 on Linux), I'm creating my Lambda service under a subdirectory.
Project structure looks as follows:
node_modules/
service/
node_modules/ <= sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz
utils/
handler.js
package.json <= engines: node 6.10.1
serverless.yml
src/
jasmine.json
package.json
Most of the dependencies I need are for development and testing purpose. These are maintained inside root package.json file (also includes sharp, but compiled for my Nodejs 7.5 environment, offering to test image manipulations locally).
My service/handler.js and service/utils/ contains ES6 compatible source code with Lambda function handler – it is transpiled from src/ directory.
If I need other dependencies for production (besides sharp), I install them to services/package.json using --prefix option. But not aws-lambda, neither aws-sdk – they are globally installed within Lambda, meaning no need to include them in deployable .zip file.
npm i -S lodash --prefix services/
It ensures installation of lodash version compatible with Lambda environment, because service/package.json defines Nodejs version to rely on:
{
"private": true,
"engines": { "node" : "6.10.1" },
"dependencies": {
...
}
}
However, there's a nuance — other production dependencies doesn't have to be environment dependent. If so, they won't work, because you install them from your local machine, which isn't equal to Lambda's one.
Lambda function deployment
Since Lambda requires .zip archive, I compress contents of my service/ directory. And my Lambda functions works. Everything is ES6 compatible, sharp has Lambda environment binaries and my other production dependency versions correlates with Nodejs 6.10.1.
Additionally, I'd suggest to use Serverless ⚡️ (I use it too). It dramatically simplifies Lambda functions development and deployment.
Nik's answer definitely helped me to get to a working solution! One thing that I would add is that the people behind serverless-sharp-image updated their package so the tarball works with node v6.10 now so I don't see a reason to have two different node environments referenced. I do everything in v6.10.
https://github.com/adieuadieu/serverless-sharp-image/tree/master/lib
Had similar problem and managed to install binaries for Linux x64 platform by
npm install --arch=x64 --platform=linux --target=8.10.0 sharp
Then just upload Lambda as usual and it works just fine.
Above works on Mac as well as windows and details are in documentation at http://sharp.pixelplumbing.com/en/stable/install/#aws-lambda
For anyone stumbling upon this post now. I've accomplished this by copying my package.json file into an AWS Cloud9 IDE and simply running npm install. From there, just download the node_modules/ folder.
Currently I have a AWS EB stack setup using the default 64bit Amazon Linux 2016.03 v2.1.3 running Node.js AMI.
Our codebase is written in ES6 and we use Babel to transpile the code into ES5. Our current deployment process is running babel locally to build the /dist directory and commiting the dist into our git repository and using eb deploy to deploy the application to EB.
I would like to be able to remove the step of building the distribution locally and perform this operation on the EB server.
I have tried using .ebextensions with a command to execute npm run build which yields an error Return code: 127 Output: /bin/sh: npm: command not found.. I have also tried using files to insert the file into the appdeploy/pre which yields the same error. And I have tried using container_commands which also yields an error stating that npm is not available.
Which of the available deployment hooks that AWS EB provides would be the correct place to use npm run build?
You can use npm hooks.
For example:
Here I have a npm start command that fires my server but before it I have a babel . -d ./dist to compile the files on the dist directory. So I have a task called prestart ( using the npm hook pre naming convention ) that does that.