AWS lambda layers error when call API "cannot find module" - amazon-web-services

I try to use layers of AWS Lambda, watched a tutorial about it, but I get an error "cannot find module ..."
service: aws-nodejs
package:
exclude:
- .gitignore
- package.json
- .git/**
provider:
name: aws
profile: sandbox
runtime: nodejs12.x
layers:
testLayer:
path: testLayer
compatibleRuntimes:
- nodejs12.x
allowedAccounts:
- '*'
functions:
hello:
handler: handler.hello
layers:
- arn:aws:lambda:us-east-1:*:layer:testLayer:15
events:
- http:
path: test
method: get
cors: true
When i deploy it, i don't have any errors in my terminal, and on AWS, i see my layer and when i download it, i have my package.json with moment dependency, and the node_modules folder with moment
my handler.js looks like this :
'use strict';
module.exports.hello = async (event, context) => {
const moment = require('moment')
const a = moment('2016-01-01')
return {
statusCode: 200,
body: JSON.stringify({
message: 'Hey' + a
}),
};
};
My files structure :
testLayer/
node_modules/
moment/
package.json
serverless.yml
handler.js
package.json
Do you have any idea of what i doing wrong ?

Yes, it's possible that your node modules are there in your lambda layer and yet lambda is throwing error of "cannot find module..".
This may be happening because the zip that you have created is not as per directory structure mentioned in AWS documentation.
As per official documentation :
Including Library Dependencies in a Layer You can move runtime
dependencies out of your function code by placing them in a layer.
Lambda runtimes include paths in the /opt directory to ensure that
your function code has access to libraries that are included in
layers.
To include libraries in a layer, place them in one of the folders
supported by your runtime.
Node.js – nodejs/node_modules, nodejs/node8/node_modules (NODE_PATH)
Example AWS X-Ray SDK for Node.js
nodejs/node_modules/aws-xray-sdk
Make sure your zip contains correct directory structure else try importing your modules from /opt/your_node_module_directory

You can check node paths that provided by lambda with:
console.log(process.env.NODE_PATH);
After seeing the absolute path, you should set your import according to it. In my case it was:
import axios from '/opt/nodejs/node_modules/axios/index.js';

Related

Unzipped size must be smaller than 262144000 bytes - AWS Lambda Error

I have tried to upload my application using servless/lambda function AWS, but i got this issue:
An error occurred: AppLambdaFunction - Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 8ea0d887-5743-4db1-96cd-6c5efa57b081).
What is the best way to resolve it?
Look my dependencies:
"dependencies": {
"ethereumjs-tx": "^1.3.7",
"aws-sdk": "^2.4.52",
"body-parser": "^1.18.3",
"compression": "^1.7.4",
"consign": "^0.1.6",
"cors": "^2.8.5",
"express": "^4.16.4",
"helmet": "^3.16.0",
"moment": "^2.24.0",
"openzeppelin-solidity": "^2.3.0",
"serverless": "^1.48.2",
"serverless-http": "^1.9.1",
"serverless-offline": "^4.9.4",
"truffle": "^5.1.9",
"truffle-hdwallet-provider": "^1.0.17",
"web3": "^1.2.5-rc.0"
},
Serverless.yml:
provider:
name: aws
runtime: nodejs8.10
stage: v1
region: us-east-1
timeout: 30
memorySize: 512
package:
excludeDevDependencies: true
exclude:
- .git/**
- .vscode/**
- venv/**
functions:
app:
handler: handler.run
events:
- http:
path: /
method: ANY
cors: true
- http:
path: /{proxy+}
method: ANY
cors: true
plugins:
- serverless-offline
Use the directive exclude at your serverless.yml file. In case of Python, I've been used it as follows:
package:
exclude:
- node_modules/**
- venv/**
The build process will exclude them from the build before sending to AWS.
Tip I got in this issue at Github. The documentation for this directive is detailed here.
You can use module bundlers to package the code.
Using module bundlers such as webpack
You can consider using plugins like serverless-webpack. The serverless-webpack plugin is using webpack to build the project and it will only include the bare minimum files required to run your application. It will not include the entire node_modules directory. so that your deployment package will be smaller.
a note about using of Lambda layers
Like others mentioned, you can use the layers and move some of the libraries and code to the layer. Layers are mainly used to share code between functions. The unzipped deployed package including layers cannot exceed 250MB.
hope this helps.
References:
https://github.com/serverless-heaven/serverless-webpack
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
I've had success resolving this error message using the serverless-esbuild plugin, and configuring it as follows in serverless.yml:
service: service_name
frameworkVersion: '3'
provider:
name: aws
runtime: nodejs12.x
plugins:
- serverless-esbuild
custom:
esbuild:
bundle: true
minify: false
sourcemap: true
exclude: 'aws-sdk'
target: node14
define:
'require.resolve': undefined
platform: node
concurrency: 10
You can load larger packages into AWS Lambda indirectly using s3:
Load your package into a bucket/key on S3
In the Lambda console choose Function Code -> Code Entry Type -> Upload a file from S3
See my answer here
You can deploy a Lambda function using a Docker image and it bypasses this problem, allowing a function with its dependencies to be as large as 10 gb.
Adding exclude under package is deprecated. We can use pattern to remove node_modules.
Example to remove files in the serverless.yml
...remaining props
package:
patterns:
- '!.git/**'
- '!test/**'
- '!e2e/**'
- '!src/**'
- '!node_modules/**'
Deprecation for exclude and
Pattern
Recently i faced the same issue, my total package size was more than 40mbs and it was also including the venv (python virtual environment) folder that resides in the project directory. I excluded it and build size got reduced to 16 mbs. and the project was deployed successfully. I added the following in serverless.yaml
package:
patterns:
- '!node_modules/**'
- '!venv/**'
- '!apienv/**'
- '!__pycache__/**'

AWS serverless deployment with webpack built modules failing

I have created a serverless aws-nodejs template project and in it I have organized my js files in the following way -
project:root -
| .env
| src -
| controllers -
<js_files_here>
| helpers -
<js_files_here>
| models -
<js_files_here>
| routes -
<yml_files_here>
And this is my serverless.yml -
service: rest-api
provider:
name: aws
runtime: nodejs8.10
stage: dev
region: ap-south-1
plugins:
- serverless-bundle
- serverless-offline
functions:
${file(./src/routes/index.yml)}
and in one of my js files I am trying to use -
require('dotenv').config({ path: './.env' });
So I am trying to load some of the environment variables from this .env file. Now this is working as expected when I test these files locally with - sls offline start
but when I deploy them to a aws account, the apis stop working as expected and also when I see the package (rest-api.zip) file in the .serverless directory I do not see all the files from src directory packaged in there.
So, how do I fix this issue and deploy my project correctly with serverless on aws ?
Your problem is that WebPack failed to include the file in its transitive closure when trying to find out all the files you need, due to dotenv importing it dynamically.
You could use a WebPack plugin to explicitly include your env file, such as this one.

AWS Layer: adding "/opt/" path when using a Nodejs layer

So I uploaded this layer to AWS using the Serverless framework:
service: webstorm-layer
provider:
name: aws
runtime: nodejs8.10
region: us-east-1
layers:
nodejs:
path: nodejs # path to contents on disk
name: node-webstormlibs # optional, Deployed Lambda layer name
description: JS shared libs for node
compatibleRuntimes:
- nodejs8.10
allowedAccounts:
- '*'
The libraries I need are inside the "nodejs" directory, there lays my packages.json file and all the "node_modules" directories. So far all looks fine, but when I try to run a lambda that uses the "node-webstormlibs" layer, I'm getting the message:
"errorMessage": "Cannot find module 'pg'",
The pg module actually exists in the zip file that creates the layer. Then, I have doubts about how to import a module that is inside the layer. In some tutorials I see:
import pg from "pg";
like always, but in other I see:
import pg from "/opt/pg";
or even:
import pg from "/opt/nodejs/node_modules/pg";
I don't know if the "path:" option in my serverless.yml is correct though.
In the server the path is:
NODE_PATH=/opt/nodejs/node8/node_modules/:/opt/nodejs/node_modules:$LAMBDA_RUNTIME_DIR/node_modules
UPDATE
Putting all in the dir /nodejs/node8, made the trick.

Serverless Offline undefined module when loaded from lambda layer

I have the following project tree
Where nodejs folder is a lambda layer defined in the following serverless.yaml
service: aws-nodejs # NOTE: update this with your service name
provider:
name: aws
runtime: nodejs8.10
stage: dev
plugins:
- serverless-offline
layers:
layer1:
path: nodejs # required, path to layer contents on disk
name: ${self:provider.stage}-layerName # optional, Deployed Lambda layer name
functions:
hello:
handler: handler.hello
layers:
- {Ref: Layer1LambdaLayer}
events:
- http:
path: /dev
method: get
The layer1 only contains UUID package.
So when I try to run the lambda locally using serverless offline plugin, it says can't find module UUID.
But when I deploy the code to AWS, it run like a charm.
Any way we can get lambda layers running locally for testing purpose? and for speeding up the development?
Or is there any way where I can dynamically set the node_module path to point to the layer folder during the development and once I need to push to production, it change the path to the proper one
Ok after many trials, I figure out a working solution
I added a npm run command which export a temporary node_module path to the list of paths
"scripts": {
"offline": "export NODE_PATH=\"${PWD}/nodejs/node_modules\" && serverless offline"
},
So, node can lookup for the node modules inside the sub folders
I got around this by running serverless-offline in a container and copying my layers into the /opt/ directory with gulp. I set a gulp watch to monitor any layer changes and to copy them to the /opt/ directory.
I use layers in serverless offline via installing a layer from local file system as a dev dependency.
npm i <local_path_to_my_layer_package> --save-dev
BTW this issue was fixed in sls 1.49.0.
Just run:
sudo npm i serverless
Then you should specify package include in serverless.yml's layer section
service: aws-nodejs # NOTE: update this with your service name
provider:
name: aws
runtime: nodejs8.10
stage: dev
plugins:
- serverless-offline
layers:
layer1:
path: nodejs # required, path to layer contents on disk
package:
include:
- node_modules/**
name: ${self:provider.stage}-layerName # optional, Deployed Lambda layer name
functions:
hello:
handler: handler.hello
layers:
- {Ref: Layer1LambdaLayer}
events:
- http:
path: /dev
method: get
Tested on nodejs10.x runtime

How upload node_modules with serverless to aws?

I have project on serverless framework. I need to resize image. I wrote lambda function and install module sharp. Also I use serverless-webpack. In webpack I add externals: ['sharp'] and add in serverless.yml next:
custom:
webpack:
includeModules:
packagePath: './src/package.json'
I deployed it successfully but when I run lambda I get
error: Cannot find module 'sharp'
maybe I doing something wrong. If need more information I can write it.
You can use forceInclude
# serverless.yml
custom:
webpack:
includeModules:
forceInclude:
- sharp
Reference document