I am trying to update source code on an EC2 instance using cfn-hup service in cloud formation (AWS).
When I update the stack with new source code using build number, the source code does changes at EC2.
cfn-hup service running fine and all configurations are OK.
Below are the logs of cfn-hup.
2016-03-05 08:48:19,912 [INFO] Data has changed from previous state; action for cfn-auto-reloader-hook will be run
2016-03-05 08:48:19,912 [INFO] Running action for cfn-auto-reloader-hook
2016-03-05 08:48:20,191 [WARNING] Action for cfn-auto-reloader-hook exited with 1; will retry on next iteration
Can anyone plz help me on this.
The error states Action for cfn-auto-reloader-hook exited with 1. This means that the action specified in your cfn-auto-reloader-hook has been executed, but returned an error code of 1 indicating a failure state. The good news is that everything else is set up correctly (the cfn-hup script is installed and running, it correctly detected a metadata change, and it found the cfn-auto-reloader hook).
Look at the action= line in your cfn-hup entry for this hook. A typical hook will look something like this:
[cfn-auto-reloader-hook]
triggers=post.update
path=Resources.WebServerInstance.Metadata.AWS::CloudFormation::Init
action=some_shell_command_here
runas=root
To find the hook, run cat /etc/cfn/hooks.d/cfn-auto-reloader.conf on the instance, or trace back where these file contents are defined in your CloudFormation template (e.g., in the example LAMP stack, this hook is created by the files section of an AWS::CloudFormation::Init Metadata Resource, used by the cfn-init helper script). Try manually executing the line in a local shell. If it fails, use the relevant output or error logs to continue debugging. Change the command and cfn-hup should succeed the next time it runs.
This means one of your cfn-init items is failing. If it works the first time, it's likely that you have a commands section item that needs a "test" clause to determine if it needs to run or not.
Related
I am learning deploying a pubsub service to run under Cloud Run, by following the guidelines given here
Steps I followed are:
Created a new project folder "myProject" in my local machine
Added below files:
app.jsindex.jsDockerfile
Executed below command to ship the code
gcloud builds submit --tag gcr.io/Project-ID/pubsub
It's mentioned in the tutorial document that
Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Container Registry and can be re-used if desired.
But in my case it's returning with error: (Ref: screenshot)
I have verified the build logs, "It's success"
So I thought to ignore this error and proceed with the next step to deploy the app by running the command:
gcloud run deploy sks-pubsub-cloudrun --image gcr.io/Project-ID/pubsub --no-allow-unauthenticated
When I run this command it immediately asking to specify the region (26 is my choice) from the list.
Next it fails with error:
Deploying container to Cloud Run service [sks-pubsub-cloudrun] in project [Project-ID] region [us-central1]
Deploying new service... Cloud Run error: The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable.
Logs for this revision might contain more information.
As I am new to this GCP & Dockerizing services, not understanding this issue and unable to fix it. I researched many blogs and articles yet no proper solution for this error.
Any help will be appreciated.
Tried to run the container locally and it's failing with error.
I'm using VS Code IDE, and "Cloud Code: Debug on Cloud Run Emulator" to debug the code.
Starting to debug the app using configuration `Cloud Run: Run/Debug Locally` from .vscode/launch.json
To view more detailed logs, go to Output channel : "Cloud Run: Run/Debug Locally - Detailed"
Dependency check started
Dependency check succeeded
Unpausing minikube
The minikube profile 'cloud-run-dev-internal' has been scheduled to stop automatically after exiting Cloud Code. To disable this on future deployments, set autoStop to false in your launch configuration d:\POC\promo_run_pubsub\.vscode\launch.json
Configuring minikube gcp-auth addon
Using GCP project 'Project-Id' with minikube gcp-auth
Failed to configure minikube gcp-auth addon. Your app might not be able to authenticate Google or GCP APIs it calls. The addon has been disabled. More details can be found in the detailed logs.
Update initiated
Deploy started
Deploy completed
Status check started
Resource pod/promo-run-pubsub-5d4cd64bf9-8pf4q status updated to In Progress
Resource deployment/promo-run-pubsub status updated to In Progress
Resource pod/promo-run-pubsub-5d4cd64bf9-8pf4q status updated to In Progress
Resource deployment/promo-run-pubsub status failed with waiting for rollout to finish: 0 of 1 updated replicas are available...
Status check failed
Update failed with error code STATUSCHECK_CONTAINER_TERMINATED
1/1 deployment(s) failed
Skaffold exited with code 1.
Cleaning up...
Finished clean up.
I have a Django app that is deployed on kubernetes. The container also has a mount to a persistent volume containing some files that are needed for operation. I want to have a check that will check that the files are there and accessible during runtime everytime a pod starts. The Django documentation recommends against running checks in production (the app runs in uwsgi), and because the files are only available in the production environment, the check will fail when unit tested.
What would be an acceptable process for executing the checks in production?
This is a community wiki answer posted for better visibility. Feel free to expand it.
Your use case can be addressed from Kubernetes perspective. All you have to do is to use the Startup probes:
The kubelet uses startup probes to know when a container application
has started. If such a probe is configured, it disables liveness and
readiness checks until it succeeds, making sure those probes don't
interfere with the application startup. This can be used to adopt
liveness checks on slow starting containers, avoiding them getting
killed by the kubelet before they are up and running.
With it you can use the ExecAction that would execute a specified command inside the container. The diagnostic would be considered successful if the command exits with a status code of 0. An example of a simple command check could be one that checks if a particular file exists:
exec:
command:
- stat
- /file_directory/file_name.txt
You could also use a shell script but remember that:
Command is the command line to execute inside the container, the
working directory for the command is root ('/') in the container's
filesystem. The command is simply exec'd, it is not run inside a
shell, so traditional shell instructions ('|', etc) won't work. To use
a shell, you need to explicitly call out to that shell.
Hi I am trying to deploy a node application from cloud 9 to ELB but I keep getting the below error.
Starting environment deployment via CodeCommit
--- Waiting for Application Versions to be pre-processed --- ERROR: Pre-processing of application version app-491a-200623_151654 has
failed. ERROR: Some application versions failed to process. Unable to
continue deployment.
I have attached an image of the IAM roles that I have. Any solutions?
Go to your console and open up your elastic beanstalk console. Go to both applications and environments and delete them. Then in your terminal hit
eb init #Follow instructions
eb create --single ##Follow instructions.
It would fix the error, which is due to some application states which are failed. If you want to check those do
aws elasticbeanstalk describe-application-versions
I was searching for this answer as a result of watching a YouTube tutorial for how to pass the AWS Certified Developer Associate exam. If anyone else gets this error as a result of that tutorial, delete the 002_node_command.config file created in the tutorial and commit that change, as that is causing the error to occur.
A failure within the pre-processing phase, may be caused by an invalid manifest, configuration or .ebextensions file.
If you deploy an (invalid) application version using eb deploy and you enable the preprocess option, The details of the error will not be revealed.
You can remove the --process flag and enable the verbose option to improve error output.
in my case I deploy using this command:
eb deploy -l "XXX" -p
And can return a failure when I mess around with .ebextensions:
ERROR: Pre-processing of application version xxx has failed.
ERROR: Some application versions failed to process. Unable to continue deployment.
With that result I can't figure up what is wrong,
but deploying without -p (or --process)and adding -v (verbose) flag:
eb deploy -l "$deployname" -v
It returns something more useful:
Uploading: [##################################################] 100% Done...
INFO: Creating AppVersion xxx
ERROR: InvalidParameterValueError - The configuration file .ebextensions/16-my_custom_config_file.config in application version xxx contains invalid YAML or JSON.
YAML exception: Invalid Yaml: while scanning a simple key
in 'reader', line 6, column 1:
(... details of the error ...)
, JSON exception: Invalid JSON: Unexpected character (#) at position 0.. Update the configuration file.
Now I can fix the problem.
I have many EC2 instances that retain Celery jobs for processing. To efficiently start the overall task of completing the queue, I have tested AWS-RunBashScript in AWS' SSM with a BASH script that calls a Python script. For example, for a single instance this begins with sh start_celery.sh.
When I run the command in SSM, this is the following output (compare to other output below, after reading on):
/home/ec2-user/dh2o-py/venv/local/lib/python2.7/dist-packages/celery/utils/imports.py:167:
UserWarning: Cannot load celery.commands extension u'flower.command:FlowerCommand':
ImportError('No module named compat',)
namespace, class_name, exc))
/home/ec2-user/dh2o-py/tasks/task_harness.py:49: YAMLLoadWarning: calling yaml.load() without
Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
task_configs = yaml.load(conf)
Running a worker with superuser privileges when the worker accepts messages serialized with pickle is a very bad idea!
If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).
User information: uid=0 euid=0 gid=0 egid=0
failed to run commands: exit status 1
Note that only warnings are thrown. When I SSH to the same instance and run the same command (i.e. sh start_celery.sh), the following (same) output results BUT the process runs:
I have verified that the process does NOT run when doing this via SSM, and I have no idea why. As a work-around, I tried running the sh start_celery.sh command with bootstrapping in user data for each EC2, but that failed too.
So, why does SSM fail to actually run the process that I succeed in doing by actually via SSH to each instance running identical commands? The details below relate to machine and Python configuration:
Am trying to create CodeDeploy Deployment Group using the Cloud Formation Stack. Every time I run the stack, am getting script errors like Bad Interpreter, rm/ll command not found, /r /n errors. I tried to change the shell script files using dos2unix and zip those files and upload to CodeDeploy but no success.
Following is the error statement I get in logs:
2018-09-01 10:41:45 INFO [codedeploy-agent(2681)]: [Aws::CodeDeployCommand::Client 200 0.037239 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":4,\"script_name\":\"BeforeInstall.sh\",\"message\":\"Script at specified location: BeforeInstall.sh run as user root failed with exit code 127\",\"log\":\"LifecycleEvent - BeforeInstall\\nScript - BeforeInstall.sh\\n[stderr]/usr/bin/env: bash\\r: No such file or directory\\n\"}"},host_command_identifier:"WyJjb20uYW1hem9uLmFwb2xsby5kZXBsb3ljb250cm9sLmRvbWFpbi5Ib3N0Q29tbWFuZElkZW50aWZpZXIiLHsiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTEvUHJvZC9hcm46YXdzOnNkczp1cy1lYXN0LTE6OTkzNzM1NTM2Nzc4OmRlcGxveW1lbnQvZC05V0kzWk5DNlYiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTE6OTkzNzM1NTM2Nzc4Omluc3RhbmNlL2ktMDk1NGJlNjk4OTMzMzY5MjgiLCJjb21tYW5kTmFtZSI6IkJlZm9yZUluc3RhbGwiLCJjb21tYW5kUG9zaXRpb24iOjMsImNvbW1hbmRBdHRlbXB0IjoxfV0=")
2018-09-01 10:41:45 ERROR [codedeploy-agent(2681)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: InstanceAgent::Plugins::CodeDeployPlugin::ScriptError - Script at specified location: BeforeInstall.sh run as user root failed with exit code 127 - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:173:in `execute_script'
......
......
2018-09-01 10:41:45 INFO [codedeploy-agent(2681)]: [Aws::CodeDeployCommand::Client 200 0.018288 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"Script at specified location: BeforeInstall.sh run as user root failed with exit code 127\",\"log\":\"\"}"},host_command_identifier:"WyJjb20uYW1hem9uLmFwb2xsby5kZXBsb3ljb250cm9sLmRvbWFpbi5Ib3N0Q29tbWFuZElkZW50aWZpZXIiLHsiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTEvUHJvZC9hcm46YXdzOnNkczp1cy1lYXN0LTE6OTkzNzM1NTM2Nzc4OmRlcGxveW1lbnQvZC05V0kzWk5DNlYiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTE6OTkzNzM1NTM2Nzc4Omluc3RhbmNlL2ktMDk1NGJlNjk4OTMzMzY5MjgiLCJjb21tYW5kTmFtZSI6IkJlZm9yZUluc3RhbGwiLCJjb21tYW5kUG9zaXRpb24iOjMsImNvbW1hbmRBdHRlbXB0IjoxfV0=")
What can be the possible reason for failing?
The logs indicate that there is some problem with your scripts, specifically BeforeInstall.sh. Something in that script is failing with an exit code of 127. I would recommend adding logs to that script to see where it's actually failing. Once you identify the command that's failing, you can see what exit code 127 means for that particular command.
If you want help debugging that particular script, you should open up another question and provide the script, including the logs when it's gets run.
A note of CodeDeploy lifecycle hooks
In your case, your BeforeInstall script is failing, which will be the script that gets deployed with your application. However, if had been your ApplicationStop script that was failing, it's important to understand that ApplicationStop uses scripts from the last successful deployment, so if the last successful deployment had a fault script, it can cause future deployments to fail until these steps are followed.