I have an DotNet Core AWS Elastic Beanstalk environment which has started failing to deploy. The environment waits up to 10 minutes for the healthcheck to pass, but consistent gets "403 - Forbidden: Access is denied.".
I've RDP'd to the environment and the folder C:\inetpub\AspNetCoreWebApps is empty. In working environments, this contains the code.
I've tried redeploying the entire environment and deploying a package from a week ago which was previously fine. Additionally, I've tried deploying using the AWS Toolkit for Visual Studio and by uploading a package rather than using CodePipeline. All of these actions result in the same behaviour.
I'm struggling to find any logs which indicate why the code isn't being deployed to the environment. Requesting the last 100 lines doesn't contain anything useful and I can't find any deployment logs on the filesystem. In the pulled logs there is no AWS.DeploymentCommands log file.
The environment is configured to deploy as rolling with batch +1. As such, it is a new EC2 which is being written to.
What is the next step in debugging the cause of the failure?
I was able to diagnose the problem - a public file referenced in the ebextension folder had been deleted. The log file I was looking for was in C:\cfn\log\cfn-init.txt.
Related
I am experiencing an issue in deploying a multicontainer EBS app which appears to be undocumented by AWS. The error reads: client: TaskStoppedBeforePullBeginError: Task stopped before image pull could begin for task: where client is my container name. I am using the deprecated multicontainer Docker environment as the latest Docker environment on Amazon Linux 2 is undocumented and has a lot of issues when I tried it.
Things I have tried:
verified that the image is pullable and runnable from my local machine (i.e. the image runs without immediately exiting)
used eb-cli to verify that my Dockerrun.aws.json configuration is working by running it all locally using eb local run
I honestly have no idea how I could go about further troubleshooting the issue. Any here would be much appreciated. Here are the last 100 lines of logs produced by EBS.
I have gotten a little bit stuck with this one. I am trying to use Code Deploy on a Windows Server EC2 instance with no luck, it keeps getting stuck before Application Stop and all phases are Pending until it fails then they are all Skipped.
What I've checked so far:
I have installed the Code-Deploy Agent on the server and made sure it was running
I have checked and double checked the in-bound and out-bound permissions on the EC2 instance (allowed all HTTP/HTTPS requests)
I have checked the IAM role on the Code Deploy application itself (I have given all the permissions i can think of)
I checked the appspec.yml (it only needs to transfer build files from the build phase to a folder on the EC2 itself
version: 0.0
os: windows
files:
- source: \path
destination: \path
hooks:
BeforeInstall:
AfterInstall:
ApplicationStart:
I have no idea why this would happen (I've deployed on Linux instances without this problem - the agent always started reading the appspec.yml)
Any help would be appreciated. Thanks!
By design, ApplicationStop is always executed from your last successful deployment's archive since that's when you started your application. This way CodeDeploy makes sure the scripts used for starting and stopping an application belong to the same revision [1]. We don't have complete data, but it could be that the ApplicationStop script from last deployment is causing the issue.
As per [1]:
If the cause of the failure is a script from the last successful
deployment that never runs successfully, create a deployment and
specify that the ApplicationStop, BeforeBlockTraffic, and
AfterBlockTraffic failures should be ignored. There are two ways to do
this:
Use the CodeDeploy console to create a deployment. On the Create
deployment page, under ApplicationStop lifecycle event failure, choose
Don't fail the deployment to an instance if this lifecycle event on
the instance fails.
Use the AWS CLI to call the create-deployment command and include the
--ignore-application-stop-failures option.
[1] https://docs.aws.amazon.com/codedeploy/latest/userguide/troubleshooting-deployments.html#troubleshooting-deployments-lifecycle-event-failures
If any future readers com across this thread then may I refer them to the following article which helped me with a "Net::OpenTimeout" error in CodeDeploy that manifested itself in a similar way.
https://aws.amazon.com/premiumsupport/knowledge-center/codedeploy-net-opentimeout-errors/
I have a powershell script in the .ebextensions folder but its not executing. It used to work but suddenly has stopped working and there have been no changes to elastic beanstalk or the scripts that i know off. I have two instance running under elastic beanstalk and if I log into one and then deploy my package it doesnt take the instance down. I can literally watch the service (IIS Rest service) files being updated.
Elastic Beanstalk is configured for autoscaling min 2 instances max 4. I've tried doing a deploy though the AWS Elastic Beanstalk console but no joy.
Does it need to actually take the instances down in order to run the scripts? I find it odd that it can update the service without stopping the instance.
Also do .ebextension scripts not execute against running instances? is there a way to force it to execute?
What else can I check to see why the ebextensions are not being picked up?
Thanks
It's working now. My ebextensions script was not formatted properly, i.e. didn't have the necessary indentations that YAML requires so the commands where being ignored. However no error was raised by EB.
Another problem is that eb cli uses git to bundle your app, which means that you need to perform a commit so that all changes are properly bundled.
Until I get my changes right, I keep doing git add & git commit --amend all the time...
I am running an elasticbeanstalk application, with multiple environments. This particular application is hosting docker containers which host a webservice.
To upload and deploy a new version of the application to one of the environments, I can go through the web client and click on "Upload and Deploy" and from the file option I select my latest Dockerrun.aws.json file, which references the latest version of the container that is privately hosted. The upload and deploy works fine and without issue.
To make it simpler for myself and others to deploy I'd like to be able to use the CLI to upload and deploy the Dockerrun.aws.json file. If I use the cli eb deploy command without any special configuration the normal process of zipping up the whole application and sending it to the host occurs and fails (it cannot reason out that it only needs to read the Dockerrun.aws.json file).
I found a documentation tidbit about controlling what is uploaded using the .elasticbeanstalk/config.yml file.
Using this syntax:
deploy:
artifact: Dockerrun.aws.json
The file is uploaded and actually deploys successfully to the first batch of instances, and then always fails to deploy to the second set of instances.
The failure error is of the flavor: 'container exited unexpectedly...'
Can anyone explain, or provide link to the canonical approach for using the CLI to deploy single docker container applications?
So it turns out that the method I listed about with the config.yml was correct. The reason I was seeing a partially successful deployment was because the previously running docker container on the hosts was not being stopped by EB.
I think that what was happening was that EB is sending something like
sudo docker kill --signal=SIGTERM $CONTAINER_ID instead of the more common sudo docker stop $CONTAINER_ID
The specific container I was running didn't respond to SIGTERM and so it would just sit there. When I tested it locally with SIGKILL it would (obviously) stop properly, but SIGTERM alone wouldn't stop it.
The issue wasn't the deployment methodology but rather confusion in the output that EB generated and my misinterpretation.
Since you have asked for a link, I am providing a link which I initially used to successfully test and deploy docker using elasticbeanstalk cli.
Kindly see if this helps you as well: https://fangpenlin.com/posts/2014/11/25/running-docker-with-aws-elastic-beanstalk/
I have an AWS Elastic Beanstalk with an environment setup (Windows Server 2012, IIS 8, Load Balanced). When I first create the environment with a .NET application, everything appears to work just fine. However, when I redeploy the application - using the AWS tools for Visual Studio 2012 - the new version does not seem to be deployed. I see the new deployment bundle up inthe proper S3 location, and the event viewer in the console indicates that everything is going fine:
Environment update is starting.
Deploying new version to instance(s).
Command execution completed successfully.
New application version was deployed to running EC2 instances.
Environment update completed successfully.
However, no new files appear on the server. Just for a check, I deleted all of the files in the c:\inetpub\wwwroot directory (the application deploys as the root app) and when the redeploy completes, I still do not see any files in this directory. I've tried to snapshot the logs, but there don't appear to be any (the list comes back empty). I've checked the deployments log files on the server itself (via RDP) and they are also empty. I've checked the server's event viewer as well - also void of any messages. It is almost as if the server is not actually running the deployment.
I am not sure what I could be doing wrong, but any guidance or suggestions are appreciated.
Have you looked under your 'Application Versions'?
It is possible that the bundle has been uploaded but not running on the instances.
The problem was because I was using a custom AMI for the beanstalks. I found out that the AMI I was using was not beanstalk-friendly, even though I created it from a beanstalk EC2 instance that I had customized. There was something in the configuration that made the new machines not deploy properly. In any case, for now I decided that I should just update my deployment package to include the stuff I needed (e.g., C++ redistributable) as opposed to trying to customize the machine images (i.e., Command for Elastic Beanstalk configuration to install Visual C++ Redistributable).