Installing Sharp on Elastic Beanstalk - amazon-web-services

I'm rather new to AWS. I'm running a Node.JS app on Elastic Beanstalk, and everything was working fine. However, because I need to do some image processing I decided to add Puppeteer and Sharp.
However, the installation on AWS fails and the application crashes. The log says the following:
sharp#0.21.3 install /tmp/deployment/application/node_modules/sharp
(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
ERR! sharp EACCES: permission denied, mkdir '/tmp/.npm/_libvips'
info sharp Attempting to build from source via node-gyp but this may fail due to the above error
info sharp Please see https://sharp.pixelplumbing.com/page/install for required dependencies
gyp ERR! configure error
gyp ERR! stack Error: EACCES: permission denied, mkdir '/tmp/deployment/application/node_modules/sharp/build'
gyp ERR! System Linux 4.14.97-74.72.amzn1.x86_64
gyp ERR! command "/opt/elasticbeanstalk/node-install/node-v10.15.1-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v10.15.1-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /tmp/deployment/application/node_modules/sharp
gyp ERR! node -v v10.15.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sharp#0.21.3 install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the sharp#0.21.3 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
I'm really not familiar with Elastic Beanstalk, but that seems like an access restriction error. After searching for a bit, I found a possible solution: Adding the following code as an .extension file.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/00_set_tmp_permissions.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
chown -R nodejs:nodejs /tmp/.npm
However, that didn't seem to do anything. Does anybody know where the error could be? Or where I should look for a solution? I'm not usually at home with this kind of stuff, but I'm forced to work with it for a while. Any help and advice would be greatly appreciated!

This is permission issues with AWS.
To solve this you have to follow the next steps:
Create a new file .npmrc
Add the following unsafe-perm=true
This will let to write files.

unsafe-perm=true is not working anymore with Node 16 and Amazon Linux 2, because it can't install the Sharp package. I was trying to install Strapi v4 on Elastic Beanstalk.
You need to perform permission updates in platform hooks according to this answer: https://github.com/lovell/sharp/issues/3221#issuecomment-1126528844
You can find additonal information to platform hooks here:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html
You don't need to do anything about the execute permissions when your Amazon Linux 2 was released after April 29, 2022.
Solution:
Step 1: Create the following Platform Hooks paths in the root directory of your application bundle.
- .platform/hooks/prebuild
- .platform/confighooks/prebuild
The Platform Hook (.platform/hooks/prebuild) will execute the commands found in the bash script after the application is extracted but before the application configuration and proxy server process initiates.
The Platform Hook (.platform/confighooks/prebuild) will execute the commands found in the bash script during a configuration deployment/environment update.
Step 2: Once the above Platform Hooks paths have been created in your application bundle, the following bash script (00_npm_install.sh) will need to be created and placed under the "Prebuild" directory:
The bash script (00_npm_install.sh) will include the following:
#!/bin/bash
cd /var/app/staging
sudo -u webapp npm install sharp
Step 3: Validate the Application Bundle Structure
Sample project structure:
~/my-app/
├── app.js
├── index.html
├── .npmrc_bkp
├── package.json
├── package-lock.json
├── .platform
│ ├── confighooks
│ │ └── prebuild
│ │ └── 00_npm_install.sh
│ └── hooks
│ └── prebuild
│ └── 00_npm_install.sh
└── Procfile
Step 4: Deploy the Application!
Summary:
During the application deployment process, the application bundle is initially extracted and deployed within the /var/app/staging folder. In this same first step, all Node.js dependencies found in your (package.json) file will also be installed here but exist under /var/app/staging/node_modules/....
As mentioned in your comment, the /var/app/staging folder is owned by the webapp user but the npm install command is executed by root user. Therefore, the bash script workaround solution goes to the /var/app/staging/ directory and install the Sharp dependency by executing the sudo -u webapp npm install sharp command as the webapp user. All of this is done before the application and proxy server configuration process initiates as shown in "Instance deployment workflow".
As a result, the workaround solution will make the Elastic Beanstalk deployment process skip installing the Sharp dependency all together as it already exists! Afterwards, the application deployment process continues to the final provisioning step which moves all files/content from the /var/app/staging/ folder to the final /var/app/current/ directory location.
Lastly, the second Platform Hook (.platform/confighooks/prebuild) was implemented to avoid running into the same permissions error when performing a configuration deployment. A configuration deployment occurs when you make configuration changes that only update environment instances without recreating them!
Hope the following solution helps!

Related

getting error whenever try to deploy backend with cdk via gitlab , in local it will works as expected

I am trying to deploy backend serverless resources with CDK in AWS cloud but getting errors while deploying things via GitLab build.
trying with a local visual studio it works fine.
error is below :
$ npm run deploy-backend
> ui-dashboard-infra#0.0.0 deploy-backend /builds/project/control-panel/infra
> cdk deploy --app 'ts-node stacks/backend' --all --require-approval never
Bundling asset dashboard-A/restFunctionFallbackus-east-1/Code/Stage...
/builds/project/control-panel/infra/node_modules/#aws-cdk/core/lib/asset-staging.ts:398
throw new Error(`Failed to bundle asset ${this.node.path}, bundle output is located at ${bundleErrorDir}: ${err}`);
^
Error: Failed to bundle asset dashboard-A/restFunctionFallbackus-east-1/Code/Stage, bundle output is located at /builds/project/control-panel/infra/cdk.out/bundling-temp-4607286efdd890681baca6f8a5d9b9b32929fa84e9947500fbe203e5b5d06a0f-error: Error: spawnSync bash ENOENT
at AssetStaging.bundle (/builds/project/control-panel/infra/node_modules/#aws-cdk/core/lib/asset-staging.ts:398:13)
at AssetStaging.stageByBundling (/builds/project/control-panel/infra/node_modules/#aws-cdk/core/lib/asset-staging.ts:246:10)
at stageThisAsset (/builds/project/control-panel/infra/node_modules/#aws-cdk/core/lib/asset-staging.ts:137:35)
at Cache.obtain (/builds/project/control-panel/infra/node_modules/#aws-cdk/core/lib/private/cache.ts:24:13)
at new AssetStaging (/builds/project/control-panel/infra/node_modules/#aws-cdk/core/lib/asset-staging.ts:162:44)
at new Asset (/builds/project/control-panel/infra/node_modules/#aws-cdk/aws-s3-assets/lib/asset.ts:68:21)
at AssetCode.bind (/builds/project/control-panel/infra/node_modules/#aws-cdk/aws-lambda/lib/code.ts:180:20)
Subprocess exited with error 1
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! ui-dashboard-infra#0.0.0 deploy-backend: `cdk deploy --app 'ts-node stacks/backend' --all --require-approval never`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the ui-dashboard-infra#0.0.0 deploy-backend script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-08-10T08_29_17_449Z-debug.log
I have faced a similar kind of issue, and the problem is that there is a bug in cdk versions prior to the once released before 25th Feb 2021.
For that, please make sure either you are using the latest version of cdk, or you DO NOT HAVE ANY WHITE SPACES IN YOUR FILE PATHNAME (any directory name having space or your username that has a space might cause this problem). Once I removed the white spaces from the pathname, everything started to work as expected.
I got the same error as you. In my case, the issue was that I had esbuild installed as a dependency in the CDK folder, but not in the folder of my Lambda function.
I was able to fix the issue by installing esbuild as a dev dependency in my Lambda function folder. If you don't want to do that, you can also install it globally in your pipeline by adding the command npm i -g esbuild to your pipeline (consider pinning it to a particular version ie. npm i -g esbuild#x.y.z).
No idea why it was working locally without esbuild installed in the function folder, but then broke when doing the same thing in a Gitlab pipeline… I didn't have esbuild installed globally on my local machine or anything.

AWS CodeBuild: deploy the code throws npm error in pipeline

I am deploying the code build first time. I am getting the below error when deploying the code in the server. The pipeline shows succeeded when code merged but when running npm ci in codebuild it throw an error.
[Container] 2021/06/14 12:44:52 Running command npm ci npm ERR! cipm can only install packages with an existing package-lock.json or npm-shrinkwrap.json with lockfileVersion >= 1. Run an install with npm#5 or later to generate it, then try again.
Looks like you'll need to generate the package-lock.json so that the command can succeed. See this answer for instructions on how to generate that file.Generate that file and check it into source code and then try re-running your build. As the error suggests, you'll need npm 5 or later although the answer I linked to is for npm 6 or 7.

Deploying NestJS application on Elastic Beanstalk

I am trying to deploy my NestJS application to AWS elastic beanstalk, but did not had any success, can someone please write step by step how can I achieve that?
Full explanation:
I have a nestjs app with typeorm but didn’t configured it to work with RDS, so we will leave it for now (maybe there is a connection, idk).
First of all I made a CodePipline that when I am pushing new version to my github repo it automatically deploying the whole repo to my eb instance that works on node 12.x.
Now, I want that on every git push, the instance will install the dependencies, build the nest app, and start the server from the /dist/main.js.
I have added a Procfile with:
web: npm install && npm run build && npm run start:prod
I have also added PORT environment variable on EB that configured on main.ts and when not found to use 8080.
And my package.json scripts are like a newly created nest app:
"scripts": {
"prebuild": "rimraf dist",
"build": "nest build",
"format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
"start": "nest start",
"start:dev": "nest start --watch",
"start:debug": "nest start --debug --watch",
"start:prod": "node dist/main",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"test": "jest",
"test:watch": "jest --watch",
"test:cov": "jest --coverage",
"test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
"test:e2e": "jest --config ./test/jest-e2e.json"
}
When deploying I see in the logs that it did something like this:
Jan 23 20:01:14 web: > app#0.0.1 start /var/app/current
Jan 23 20:01:14 ip-172-31-17-171 web: > sudo npm i -g #nestjs/cli && node dist/main
Jan 23 20:01:14 web: We trust you have received the usual lecture from the local System
Jan 23 20:01:14 web: Administrator. It usually boils down to these three things:
Jan 23 20:01:14 web: #1) Respect the privacy of others.
Jan 23 20:01:14 web: #2) Think before you type.
Jan 23 20:01:14 web: #3) With great power comes great responsibility.
Jan 23 20:01:14 web: sudo: no tty present and no askpass program specified
Jan 23 20:01:14 web: npm ERR! code ELIFECYCLE
Jan 23 20:01:14 web: npm ERR! errno 1
Jan 23 20:01:14 web: npm ERR! app#0.0.1 start: `sudo npm i -g #nestjs/cli && node dist/main`
Jan 23 20:01:14 web: npm ERR! Exit status 1
Jan 23 20:01:14 web: npm ERR!
Jan 23 20:01:14 web: npm ERR! Failed at the app#0.0.1 start script.
So I am getting 502 when entering the env url.
So maybe it is a permission issue with npm? Or I need to deploy on some way only the /dist folder?
What do you think the problem is?
It is my first time trying to deploy a backend server to eb :)
I got stuck with the deploying the nestjs app to AWS elastic beanstalk for a while so that I decide make the guide here. I make the deploy guide step by step here for deploying the simple app.
The reason WHY the deploying is usually FAILED because it does not know NEST dependencies (it does not install dev dependencies) and Typescript code which need to compile before uploading to server.
In my case, Use the compression for making the zip file (you can use the command line eb but try to use this first for simple).
Firstly, Preparing the zip file, this is really IMPORTANT for avoiding the FAILED while it is deploying.
You need to make dist folder. For doing this, we remove the dist folder first and run the command nest build (* this one do the tsc for compile Typescript for us).
Now, we have the dist folder, the next thing we need to do is copying the package.json ( * this one installing the DEPENDENCIES for us ) into dist folder to let server know installing the dependencies that we do not bring the node_module here.
In package.json, yarn start to run nest start but the AWS does not know Nest command so that we need to point it out by using node main.js (this file is in dist file, you need to make the path to main.js correctly). For doing this, we create the Procfile (reference this) and add the conntent is web: node src/main.js. Remember copying it into the dist file.
Compressing file correctly as this
Secondly, from AWS console creating the application or environment then uploading the zip file and get successful status. If not, check the log file to know why it failed. Maybe it does not the nest command, etc...
Hoping this guide will help you.
Cheer!
I was stuck with the following EBS eb-engine.log error:
"Instance deployment: You didn't include a 'package.json' file in your
source bundle. The deployment didn't install a specific Node.js
version."
"Instance deployment failed to generate a 'Procfile' for Node.js.
Provide one of these files: 'package.json', 'server.js', or 'app.js'.
The deployment failed."
Using all previous answers, the following works for me with CodePipeline and EBS:
buildspec.yml
version: 0.2
phases:
install:
commands:
- npm install
build:
commands:
- npm run build
post_build:
commands:
- cp -R node_modules/ dist/node_modules # nodejs needs this
- cp Procfile dist/Procfile # EBS needs this
artifacts:
files:
- "**/*"
discard-paths: no
base-directory: dist
Procfile
web: node main.js
The way it works is that CodeBuild handles Nest building, and CodeDeploy only deploys what's in dist/
Node will need node_modules so post build I copy it to dist/
But EBS complains there's no package.json or any file it recognizes in dist/, and since Procfile is usually at root, copy it so EBS finds it and starts by using node main.js
"build": "nest build && cp package.json dist/",
"postbuild": "cd dist && zip ../dist.zip -r * .[^.]*",
"start": "node main.js"
Modifying just these three scripts, it works for me. Procfile is not required.
Finally in the root folder of the project, the dist.zip file that was generated I upload it to Amazon
Change your artifacts to copy everything, so that Codebuild can copy your node_modules, package.json, dist folder automatically.
artifacts:
files:
- "**/*"
Then in your package.json, change the start command to production by default -
"start": "node dist/main"
Here is my buildspec.yml and package.json which works fine on Nestjs deployment on Elastic beanstalk
The mysterious part is this: where is sudo npm i -g #nestjs/cli coming from?
I'm also deploying to Elastic Beanstalk but I only have web: npm run start:prod in my Procfile. The build happens in CI, for that I'm using GitHub Actions. The general outline of the workflow file is as follows:
Checkout
Setup Node
Install dependencies and build app
Generate deployment ZIP archive
Upload deployment archive to Elastic Beanstalk
I suggest you make your Procfile be like mine, run the build script locally, generate the deployment archive using something like zip -r deployment.zip dist package* Procfile and upload it to Elastic Beanstalk and see what the outcome is.

AWS Elastic Beanstalk invalid binary packages

I am working on migrating a project from Heroku to AWS. I keep getting an error after deployment saying: 'invalid ELF header'. I have found posts with similar issues when using AWS Lambda, but I do not understand why I would have an issue with binary packages in Elastic Beanstalk.
Doesn't Elastic Beanstalk provide a configured environment to run my code similar to Docker? I feel like this problem must be more complex as I cannot find anyone else with this issue in Elastic Beanstalk.
Here is the exact error I am getting:
Error: /var/app/current/node_modules/bcrypt/lib/binding/bcrypt_lib.node: invalid ELF header
at Object.Module._extensions..node (internal/modules/cjs/loader.js:730:18)
at Module.load (internal/modules/cjs/loader.js:600:32)
at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
at Function.Module._load (internal/modules/cjs/loader.js:531:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object.<anonymous> (/var/app/current/node_modules/bcrypt/bcrypt.js:6:16)
at Module._compile (internal/modules/cjs/loader.js:701:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
at Module.load (internal/modules/cjs/loader.js:600:32)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! beer#0.1.0 start: `node server/server.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the beer#0.1.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /tmp/.npm/_logs/2019-04-15T00_54_06_983Z-debug.log
I am working on a Mac, my code contains both client and server code written in React, node.js, and SQL. I am uploading my code via a ZIP file containing multiple folders, including node_modules/.
Some libraries are sensitive to the operating system and / or CPU architecture on which they are built. This is especially true of modules implemented in low-level languages, or that link to system libraries. In this case, you appear to be using bcrypt which is largely written in C++.
This is one reason that the zip file you upload to Elastic Beanstalk shouldn't include your node_modules/ folder (or anything else that Git is ignoring). The easiest way to create a zip for uploading to Elastic Beanstalk is probably to use git archive:
git archive -v -o myapp.zip --format=zip HEAD
This will respect your ignores, whereas manually zipping will include them.
Your archive should include package.json and package-lock.json in its root. Elastic Beanstalk will install its own node_modules/ from these files if they are present. This should ensure that all libraries are compatible with its operating system.

ENOENT error when doing ionic start

I upgraded to beta.23 and started to get an error when created a new project using 'ionic start foo --v2 --verbose'.
The error in red says
"Unable to spawn commandError: spawn npm ENOENT (CLI v2.0.0-beta.23)."
I have npm installed and can run from Windows 8 cmd line.
I tried upgrade to node v5.9.1.
I confirmed npm is in the PATH environment variable and can run npm from cmd shell.
Any ideas how to fix or debug further would be appreciated.
Here is the complete output:
C:\ionic2>ionic start foo --v2 --verbose
Task setting: title=start, name=start, summary=Starts a new Ionic project in the
specified PATH, [options]=any flags for the command, <PATH>=directory for the n
ew project, [template]=Starter templates can either come from a named template,
(ex: tabs, sidemenu, blank),
a Github repo, a Codepen url, or a local directory.
Codepen url,
Defaults to Ionic "tabs" starter template, --appname|-a=Human readable name for
the app (Use quotes around the name), --id|-i=Package name for <widget id> confi
g, ex: com.mycompany.myapp, title=Skip npm package installation, boolean=true, t
itle=Create a basic structure without Cordova requirements, boolean=true, title=
Setup the project to use Sass CSS precompiling, boolean=true, title=List starter
templates available, boolean=true, --io-app-id=The Ionic.io app ID to use, --te
mplate|-t=Project starter template, boolean=true, title=Start a Ionic v2 project
, boolean=true, title=(with --v2 only) Use TypeScript in starter, --zip-file|-z=
URL to download zipfile for starter template, module=./ionic/start, disableChang
ePwd=true
Utils.preprocessCliOptions _=[start, foo], skip-npm=false, no-cordova=false, w=f
alse, sass=false, s=false, list=false, l=false, v2=true, v=false, typescript=fal
se, -ts=false, verbose=true, $0=C:\Program Files\nodejs\node.exe C:\Users\bradtk
e\AppData\Roaming\npm\node_modules\ionic\bin\ionic
One awesome Ionic app coming right up...
Creating Ionic app in folder C:\ionic2\foo based on tabs project
Downloading: https://github.com/driftyco/ionic2-app-base/archive/master.zip
[=============================] 100% 0.0s
Downloading: https://github.com/driftyco/ionic2-starter-
tabs/archive/master.zip
[=============================] 100% 0.0s
Installing npm packages...
Running exec command: npminstall
Utils.fail Unable to run spawn commandError: spawn npm ENOENT undefined
Utils.fail stack undefined
Utils.errorHandler is set, calling that now
Cli.Utils.errorHandler msg Unable to run spawn commandError: spawn npm ENOENT st
ring
Unable to run spawn commandError: spawn npm ENOENT (CLI v2.0.0-beta.23)
Your system information:
Cordova CLI: 6.1.0 (cordova-lib#undefined)
Ionic CLI Version: 2.0.0-beta.23
Ionic App Lib Version: 2.0.0-beta.13
OS: Windows 8.1
Node Version: v5.9.1
Regards,
bruce
I have the exact same issue. Worked in beta 22.
Reverting to the earlier beta resolved the issue:
npm install -g ionic#2.0.0-beta.22
Move into folder of your application and run npm install. I don't know what is happening but sound like start script trying to execute npminstall rather than npm install:
Installing npm packages...
Running exec command: npminstall
This is a known issue with the CLI on windows. Downgrading to cli beta22 can work as workaround for now, but this will be fixed in the next release
Upgrading npm worked for me.
npm install -g npm