AWS Elastic Beanstalk node app degraded, no errors - amazon-web-services

I'm suffering for a while to make elastic beanstalk work with my node app. No steps are going easily. After solving the running out of memory problem, now it's returning 502 bad gateway, and no errors are shown in the logs, or in the events.
Here are the few last lines of the log file:
2023/01/26 18:35:22.614796 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2023/01/26 18:35:22.625486 [INFO] web.service
2023/01/26 18:35:22.625641 [INFO] Executing instruction: RunConfigDeployPostDeployHooks
2023/01/26 18:35:22.625655 [INFO] Executing platform hooks in .platform/confighooks/postdeploy/
2023/01/26 18:35:22.625669 [INFO] The dir .platform/confighooks/postdeploy/ does not exist
2023/01/26 18:35:22.625673 [INFO] Finished running scripts in /var/app/current/.platform/confighooks/postdeploy
2023/01/26 18:35:22.625685 [INFO] Executing cleanup logic
2023/01/26 18:35:22.625789 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment completed successfully.","timestamp":1674758122625,"severity":"INFO"}]}]}
2023/01/26 18:35:22.625945 [INFO] Platform Engine finished execution on command: config-deploy
2023/01/26 18:37:09.894419 [INFO] Starting...
2023/01/26 18:37:09.894459 [INFO] Starting EBPlatform-PlatformEngine
2023/01/26 18:37:09.894483 [INFO] reading event message file
2023/01/26 18:37:09.894643 [INFO] Engine received EB command cfn-hup-exec
2023/01/26 18:37:09.989819 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-west-1:538453521205:stack/awseb-e-sp3bw3fa7a-stack/768078f0-9d7c-11ed-8201-0aa34aaab773 -r AWSEBAutoScalingGroup --region eu-west-1
2023/01/26 18:37:10.305828 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-west-1:538453521205:stack/awseb-e-sp3bw3fa7a-stack/768078f0-9d7c-11ed-8201-0aa34aaab773 -r AWSEBBeanstalkMetadata --region eu-west-1
2023/01/26 18:37:10.620926 [INFO] checking whether command bundle-log is applicable to this instance...
2023/01/26 18:37:10.620942 [INFO] this command is applicable to the instance, thus instance should execute command
2023/01/26 18:37:10.620946 [INFO] Engine command: (bundle-log)
2023/01/26 18:37:10.621008 [INFO] Executing instruction: GetBundleLogs
2023/01/26 18:37:10.621014 [INFO] Bundle Logs...
As you can see there are no fails, the last block is getting the logs that you're reading. I cannot see it trying to run npm start.
Here's my package.json:
{
"name": "test",
"version": "0.0.1",
"author": "me",
"license": "private",
"description": "to be written",
"main": "index.js",
"private": true,
"scripts": {
"build": "webpack",
"dev": "nodemon NODE_ENV=development",
"start": "node index.js"
},
// dependencies, devDependencies
"engines": {
"node": ">=14 <=17.2.0"
}
}
I also gave the PORT environment variable under Configuration > Software > Environment properties with value 5000.
In theory it should automatically look for start script and execute it, but I cannot see any tries for that in the above logs.
When I connect to the ec2 instance of ebs, and route to my app, and run npm start I get the following:
> test#0.0.1 start
> node index.js
Executing command aws secretsmanager get-secret-value --secret-id mysecret --output json --region eu-west-1
node:internal/modules/cjs/loader:1249
return process.dlopen(module, path.toNamespacedPath(filename));
^
Error: /var/app/current/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: invalid ELF header
When I run aws secretsmanager get-secret-value --secret-id mysecret --output json --region eu-west-1 manually, it gives the expected output, no errors.
Because of the running out of memory while installing node_modules issue stated above, I solved it by installing the dependencies locally, and deploying the whole thing to ebs. It was on a mac.
Anyone any clue?

Related

Elastic Beanstalk Docker deploy fails with "no space left on device"

I am following a tutorial to deploy a Flask application with Docker to AWS Elastic Beanstalk (EB). I created an AWS Elastic Container Registry (ECR) and ran some commands which successfully pushed the Docker image to the ECR:
docker build -t app-backend
docker tag app-backend:latest [URL_ID].dkr.ecr.us-east-1.amazonaws.com/app-backend:latest
docker push [URL_ID].dkr.ecr.us-east-1.amazonaws.com/app-backend:latest
Then I tried to deploy to EB:
eb init (selecting a Docker EB application I created on the AWS GUI)
eb deploy
On "eb init" I get the error "Cannot setup CodeCommit because there is no Source Control setup, continuing with initialization", but I assume this can be ignored as it otherwise looked fine. On "eb deploy" though, the deployment fails. In "eb-engine.log" (found in the AWS GUI), I see error messages like:
[ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to pull docker image: Command /bin/sh -c docker pull [URL_ID].dkr.ecr.us-east-1.amazonaws.com/app-backend:latest failed with error exit status 1. Stderr:failed to register layer: Error processing tar file(exit status 1): write /root/.cache/pip/http/5/e/7/3/b/[long number]: no space left on device
When I manually run the pull command the error references (locally, not from the EB instance), the command seems to respond as expected:
docker pull [URL_ID].dkr.ecr.us-east-1.amazonaws.com/app-backend:latest
What could be causing this deployment failure?
My Dockerrun.aws.json file looks like this:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "[URL_ID].dkr.ecr.us-east-1.amazonaws.com/app-backend",
"Update": "true"
},
"Ports": [
{
"ContainerPort": 5000,
"HostPort": 5000
}
]
}
I solved this by following how to prevent error "no space left on device" when deploying multi container docker application on AWS beanstalk?.
Basically you find your Elastic Beanstalk instance in the EC2 AWS GUI, you modify the volumes to add space to the EB instance. Then you follow the link in that Stack Overflow post to repartition your EB instance by SSHing into it with eb ssh and then using commands like df -H and lsblk to see how much space in in each partition. And use commands like:
sudo growpart /dev/xvda 1
sudo xfs_growfs -d /
to repartition the hard drive as to use all the new space you added in the AWS EC2 GUI. You can check with df -H and lsblk to see if the repartitioning gave you more space.
Then the eb deploy command should work. If SSH isn't setup yet, you may have to do eb ssh --setup first.

How to authenticate AWS EBS (Beanstalk) on ECS (Container Storage)? AccessDeniedException

I'm new to using AWS EBS and ECS, so please forgive me if I ask a question which might be obvious for others. To the issue:
I've configured EBS and ECS in the same AWS cloud region. I've pushed my container and can see the image in ECS. The EBS environment has been built using a Dockerrun.aws.json pointing to the ECS. During the build of the EBS I get the following error:
2020/11/02 20:50:25.858567 [INFO] authenticate with ECR if the image is in an ECR repo
2020/11/02 20:50:25.858582 [INFO] Running command /bin/sh -c aws ecr get-login --no-include-email --registry-ids 731178912345 --region eu-west-1
2020/11/02 20:50:28.355911 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to authenticate with ECR for registry 731178912XXX in eu-west-1: Command /bin/sh -c aws ecr get-login --no-include-email --registry-ids 731178912XXX --region eu-west-1 failed with error exit status 255. Stderr:
An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts::731178912XXX:assumed-role/aws-elasticbeanstalk-ec2-role/i-0bff9b5324348ea71 is not authorized to perform: ecr:GetAuthorizationToken on resource: *
I read somewhere that the identify is automatically passed in to process the authentication. I couldn't find any indication on what to do to resolve the issue. Maybe someone could help me out?
After adding the permission as discussed I get these:
2020/11/03 09:17:46.490399 [INFO] pull docker image if update is not false in Dockerrun.aws.json
2020/11/03 09:17:46.490431 [INFO] Running command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest
2020/11/03 09:17:46.576648 [WARN] failed to execute command: docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest, retrying...
2020/11/03 09:17:46.576673 [INFO] Running command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest
2020/11/03 09:17:46.659379 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to pull docker image: Command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest failed with error exit status 1. Stderr:Error response from daemon: pull access denied for 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project, repository does not exist or may require 'docker login': denied: User: arn:aws:sts::731178912XXX:assumed-role/aws-elasticbeanstalk-ec2-role/i-0aa453e0d44cdfa90 is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:eu-west-1:731178912XXX:repository/user/project
Your aws-elasticbeanstalk-ec2-role instance role does not have permissions to execute ecr:GetAuthorizationToken.
You can address this, by adding an inline policy to the aws-elasticbeanstalk-ec2-role role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
}
]
}
Please note that more permissions may be needed after this one. You can add them in the same way.

AWS ElasticBeanstalk deployment is throwing error, "An error occurred during execution of command [app-deploy] - [PostBuildEbExtension]"

I am trying to deploy my Laravel application into ElasticBeanstalk environment. I am using "eb deploy" command for the deployment. The command was working fine. I had been deploying my application successfully. At some point, I updated my CloudFormation template to change the environment name of the ElasticBeanstalk.
Then I also updated the .elasticbeanstalk/config.yml file as follow.
branch-defaults:
master:
environment: PatheinDirectoryTesting
group_suffix: null
environment-defaults:
MyanEat-test-env:
branch: null
repository: null
global:
application_name: PatheinDirectoryApplication
default_ec2_keyname: null
default_platform: arn:aws:elasticbeanstalk:eu-west-1::platform/64bit Amazon Linux
2 v3.1.0 running PHP 7.3
default_region: eu-west-1
include_git_submodules: true
instance_profile: null
platform_name: null
platform_version: null
profile: null
sc: git
workspace_type: Application
Then in the terminal, I updated the "eb" command to use the right environment running the following command.
eb use PatheinDirectoryTesting
Then I run "eb deploy" to deploy my application. It deployed the zip file successfully. Then it threw the error after uploading the zip.
Then I run the "eb logs" to get the error in the logs. This is the error I found in the logs.
2020/08/29 21:42:29.413325 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:eu-west-1:733553390213:stack/awseb-e-gntnptfj8v-stack/90004470-ea3e-11ea-8e57-02d39f83e350 -r AWSEBAutoScalingGroup --regio
n eu-west-1 --configsets Infra-EmbeddedPostBuild
2020/08/29 21:42:31.691170 [ERROR] An error occurred during execution of command [app-deploy] - [PostBuildEbExtension]. Stop running the command. Error: Container commands build failed. Please refer to /var/log/cfn-init.log for more
details.
I tried deleting the CloudFormation template and deploying again. Then I deploy the application. The error persists. How can I fix it?

Amazon ECS agent on ubuntu not starting

I am currently trying to build a custom ubuntu ami for AWS batch and following the document mentioned here
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html
However when I try to start the docker agent on that machine it always keeps giving me this error
2018-07-04T23:34:01Z [INFO] Amazon ECS agent Version: 1.18.0, Commit: c0defea9
2018-07-04T23:34:01Z [INFO] Loading state! module="statemanager"
2018-07-04T23:34:01Z [INFO] Event stream ContainerChange start listening...
2018-07-04T23:34:01Z [INFO] Creating root ecs cgroup: /ecs
2018-07-04T23:34:01Z [INFO] Creating cgroup /ecs
2018-07-04T23:34:01Z [WARN] Disabling TaskCPUMemLimit because agent is unabled to setup '/ecs' cgroup: cgroup create: unable to create controller: mkdir /sys/fs/cgroup/systemd/ecs: read-only file system
2018-07-04T23:34:01Z [WARN] Error getting valid credentials (AKID ): NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-07-04T23:34:01Z [INFO] Registering Instance with ECS
2018-07-04T23:34:01Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-07-04T23:34:01Z [ERROR] Error registering: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I made sure the instance has the ecsInstanceRole associated with that.
Can you guys let me know what I am missing?
Not certain how you are starting the ecs-agent. Ran into the error of
Disabling TaskCPUMemLimit because agent is unabled to setup '/ecs cgroup: cgroup create: unable to create controller: /sys/fs/cgroup/systemd/ecs: read-only file system
We resolved this by adding the volume --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro to the systemd unit file that we having launching ecs.
Outside of that, I assume the issue resides with the ecsInstanceRole. Can you verify it has the following permissions? AmazonEC2ContainerRegistryFullAccess, AmazonEC2ContainerServiceFullAccess, AmazonEC2ContainerServiceforEC2Role
Below is the full systemd file for ecs-agent.
[Unit]
Description=Docker Container %I
Requires=docker.service
After=docker.service
[Service]
Restart=always
ExecStartPre=-/usr/bin/docker rm -f %i
ExecStart=/usr/bin/docker run --name %i \
--restart=on-failure:10 \
--volume=/var/run:/var/run \
--volume=/var/log/ecs/:/log:Z \
--volume=/var/lib/ecs/data:/data:Z \
--volume=/etc/ecs:/etc/ecs \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
--net=host \
--env-file=/etc/ecs/ecs.config \
--env LOGSPOUT=ignore \
amazon/amazon-ecs-agent:latest
ExecStop=/usr/bin/docker stop %i
[Install]
WantedBy=default.target
I ran into the same messages. You need to create the IAM role and launch the instance with that role, per this documentation: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html

Amazon ECS "the referenced cluster was inactive"

I followed the steps to install the ECS client on Ubuntu 16, but when I try to run the ECS container agent, it keeps restarting and when I have a look at the logs
2016-12-07T06:01:39Z [INFO] Starting Agent: Amazon ECS Agent - v1.13.1 (efe53c6)
2016-12-07T06:01:39Z [INFO] Loading configuration
2016-12-07T06:01:39Z [INFO] Checkpointing is enabled. Attempting to load state
2016-12-07T06:01:39Z [INFO] Loading state! module="statemanager"
2016-12-07T06:01:39Z [INFO] Event stream ContainerChange start listening...
2016-12-07T06:01:39Z [INFO] Detected Docker versions [1.17 1.18 1.19 1.20 1.21 1.22 1.23]
2016-12-07T06:01:39Z [INFO] Registering Instance with ECS
2016-12-07T06:01:39Z [ERROR] Could not register module="api client" err="ClientException: The referenced cluster was inactive.
status code: 400, request id: 9eaa4124-bc42-11e6-9cf1-7559dea2bdf8"
2016-12-07T06:01:39Z [ERROR] Error registering: ClientException: The referenced cluster was inactive.
status code: 400, request id: 9eaa4124-bc42-11e6-9cf1-7559dea2bdf8
I didn't find a reference for this error on google and I'm wondering what's wrong...
Do I need to create the cluster name on the ECS dashboard ?
I have attacher the container role to my EC2 instance, which allows for cluster creation so I don't think the problem comes from here...
My docker run config
sudo docker run --name ecs-agent \
--detach=true \
--restart=on-failure:10 \
--volume=/var/run/docker.sock:/var/run/docker.sock \
--volume=/var/log/ecs/:/log \
--volume=/var/lib/ecs/data:/data \
--net=host \
--env=ECS_LOGFILE=/var/log/ecs-agent.log \
--env=ECS_LOGLEVEL=info \
--env=ECS_DATADIR=/data \
--env=ECS_CLUSTER=my-cluster \
--env=ECS_ENABLE_TASK_IAM_ROLE=true \
--env=ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true \
amazon/amazon-ecs-agent:latest
You need to call aws ecs create-cluster --region $REGION --cluster my-cluster, call the CreateCluster API through the SDK, or create it in the console. The ECS agent will only automatically create a cluster named default, and only when ECS_CLUSTER is unspecified.