So I'm trying to deploy a dockerfile on Elastic Beanstalk, but I can't get past this error - "jq: error: Cannot iterate over null".
Successfully built [myContainerId]
Successfully built aws_beanstalk/staging-app
[2015-01-29T10:35:59.494Z] INFO [16343] - [CMD-AppDeploy/AppDeployStage0/AppDeployPreHook/04run.sh] : Starting activity...
[2015-01-29T10:36:05.507Z] INFO [16343] - [CMD-AppDeploy/AppDeployStage0/AppDeployPreHook/04run.sh] : Activity execution failed, because: command failed with error code 1: /opt/elasticbeanstalk/hooks/appdeploy/pre/04run.sh
jq: error: Cannot iterate over null
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Thu Jan 29 10:36:05 UTC 2015:. Check snapshot logs for details. (Executor::NonZeroExitStatus)
at /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.1.0/gems/executor-1.0/lib/executor/exec.rb:81:in `sh'
from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.1.0/gems/executor-1.0/lib/executor.rb:15:in `sh'
from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.1.0/gems/beanstalk-core-1.1/lib/elasticbeanstalk/executable.rb:63:in `execute!'
from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.1.0/gems/beanstalk-core-1.1/lib/elasticbeanstalk/hook-directory-executor.rb:29:in `block (2 levels) in run!'
from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.1.0/gems/beanstalk-core-1.1/lib/elasticbeanstalk/activity.rb:169:in `call'
from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.1.0/gems/beanstalk-core-1.1/lib/elasticbeanstalk/activity.rb:169:in `exec'
There aren't any other errors in the logs. My Docker container is successfully built, so it seems unlikely the error is coming from there.
My Dockerrun.aws.json looks like :
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "blah",
"Update": "false"
},
"Ports": [
{
"ContainerPort": "8080"
}
]
}
I'm banging my head against a wall with this one, nothing I change seems to affect it and googling hasn't been of any help.
Any ideas?
If others are looking for how to avoid the Cannot iterate over null error in their own jq commands, add a question mark after []. For example
echo '{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "blah",
"Update": "false"
},
"Ports": [
{
"ContainerPort": "8080"
}
]
}'|jq -c '.Volumes[]?|[.HostDirectory,.ContainerDirectory]'
where [] was replaced with []? does not display the error.
From the manual:
.[]?
Like .[], but no errors will be output if . is not an array or object.
The problem is that your Dockerrun.aws.json file is missing the Volumes property.
The 04pre.sh script uses the jq tool in order to query the JSON file.
Specifically, it runs the following command on your file, which yields to an error:
$ jq -c '.Volumes[] | [.HostDirectory, .ContainerDirectory]' < Dockerrun.aws.json
jq: error: Cannot iterate over null
Specifying an empty "Volumes" array should resolve the error.
Note that an Elastic Beanstalk deployment to the Docker platform can fail with the "jq: error ... Cannot iterate over null..." error message in various phases of an Elastic Beanstalk deployment, for various different reasons. This can include your application (Docker container) quitting or erroring on startup.
Although this particular reported occurrence of the problem may have been specific to a missing "Volumes" property in Dockerrun.aws.json, your problem may not be. If you get this error, then the best way to go about diagnosing the problem is to download your full EB logs and check the diagnostics in the following log file /var/log/eb-activity.log.
If the problem was caused by your application failing to startup then you'll find the error in /var/log/eb-docker/containers/eb-current-app/unexpected-quit.log.
In my case issue was caused by ADD in Dockerfile instead of COPY instruction. Changing it fixed the issue.
Related
When I upload Dockerrun.aws.json to Elastic Beanstalk, I get the following errors in EB.
On events:
- During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
- Failed to deploy application.
- Unsuccessful command execution on instance id(s) 'i-XXXXXXXXXXXXXXX'. Aborting the operation.
On Health:
Overall:
- Command failed on all instances.
- Incorrect application version found on all instances. Expected version "Sample Application" (deployment 1).
i-XXXXXXXXXXXX:
- Application deployment failed at 2020-06-13T11:21:07Z with exit status 1 and error: Engine execution has encountered an error.
- Incorrect application version "43" (deployment 5). Expected version "Sample Application" (deployment 1).
On Logs:
Error response from daemon: You cannot remove a running container XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Stop the container before attempting removal or force remove
Error response from daemon: conflict: unable to delete XXXXXXXXXXXX (cannot be forced) - image is being used by running container XXXXXXXXXXXX
My Dockerrun.aws.json file:
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "try-new-new",
"Key": ".docker/config.json"
},
"Image": {
"Name": "registry.gitlab.com/XXXXXXXXX/XXXXXXXXXX",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "80"
}
]
}
My .docker/config.json file:
{
"auths" :
{
"https://registry.gitlab.com/" :
{
"auth" : "XXXXXXXXXXXXXXXXX",
"email" : "abc#abc.com"
}
}
}
I noticed that the problem is an Authentication problem. Unfortunately, I was unable to authorize AWS to pull the container from Registry.gitlab.com.
For now (though not a complete solution) I have created a solution by moving my container to the AWS Elastic Container Repository.
I have been playing with the Dockerrun.aws.json file and I can't figure out how it should work. I am new to AWS Docker config stuff and just need the communities help with this.
Everytime I deploy to the docker image to Elastic Beanstalk I get this error.
[2017-12-19T19:38:40.877Z] ERROR [31251] : Command CMD-AppDeploy failed!
[2017-12-19T19:38:40.879Z] INFO [31251] : Command processor returning results:
{"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"(TRUNCATED)...spec \":\": invalid volume specification: ':'.\nSee 'docker run --help'.\nFailed to run Docker container: docker: Error response from daemon: invalid volume spec \":\": invalid volume specification: ':'.\nSee 'docker run --help'.. Check snapshot logs for details. \nHook /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI","returncode":125,"events":[{"msg":"Successfully pulled image:latest","severity":"TRACE","timestamp":1513712291},{"msg":"Successfully built aws_beanstalk/staging-app","severity":"TRACE","timestamp":1513712317},{"msg":"Failed to run Docker container: docker: Error response from daemon: invalid volume spec \":\": invalid volume specification: ':'.\nSee 'docker run --help'.. Check snapshot logs for details.","severity":"ERROR","timestamp":1513712320}]}],"truncated":"true"}
....
[2017-12-19T19:48:57.688Z] ERROR [31884] : Command execution failed: Activity failed. (ElasticBeanstalk::ActivityFatalError)
caused by: 5503805a9751c924331669930c577b50628d1d0967015c301671f151e207312e
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Tue Dec 19 19:48:57 UTC 2017:
/bin/sh: 1: Syntax error: "(" unexpected. Check snapshot logs for details. (ElasticBeanstalk::ExternalInvocationError)
caused by: 5503805a9751c924331669930c577b50628d1d0967015c301671f151e207312e
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Tue Dec 19 19:48:57 UTC 2017:
/bin/sh: 1: Syntax error: "(" unexpected. Check snapshot logs for details. (Executor::NonZeroExitStatus)
My Dockerrun.aws.json file
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "path/to/image:latest",
"Update": "true"
},
"Authentication": {
"Bucket": "mybucket",
"Key": "dockercfg"
},
"Ports": [
{
"ContainerPort": "80"
}
],
"Volumes": []
}
Dockerfile
FROM node:latest
WORKDIR /app
ADD package.json /app
RUN npm i --production
ADD . /app
RUN node ./node_modules/grunt/bin/grunt prod
CMD npm start
EXPOSE 80
I appreciate any help I can get on this issue. I am hitting a wall on this.
Thanks.
Running Kubernetes on CoreOS on an AWS EC2 instance, I am unable to execute apiserver via a hyperkube Docker container successfully. The problem is that the etcd server refuses connections due to a bad certificate.
What happens is this:
$ docker run -v /etc/ssl/etcd:/etc/ssl/etcd:ro gcr.io/google_containers/hyperkube:v1.1.2 /hyperkube apiserver --bind-address=0.0.0.0 --insecure-bind-address=127.0.0.1 --etcd-servers=https://172.31.29.111:2379 --allow-privileged=true --service-cluster-ip-range=10.3.0.0/24 --secure-port=443 --advertise-address=172.31.29.111 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --tls-cert-file=/etc/ssl/etcd/master1-master-client.pem --tls-private-key-file=/etc/ssl/etcd/master1-master-client-key.pem --client-ca-file=/etc/ssl/etcd/ca.pem --kubelet-certificate-authority=/etc/ssl/etcd/ca.pem --kubelet-client-certificate=/etc/ssl/etcd/master1-master-client.pem --kubelet-client-key=/etc/ssl/etcd/master1-master-client-key.pem --kubelet-https=true
I0227 17:07:34.117098 1 plugins.go:71] No cloud provider specified.
I0227 17:07:34.549806 1 master.go:368] Node port range unspecified. Defaulting to 30000-32767.
[restful] 2016/02/27 17:07:34 log.go:30: [restful/swagger] listing is available at https://172.31.29.111:443/swaggerapi/
[restful] 2016/02/27 17:07:34 log.go:30: [restful/swagger] https://172.31.29.111:443/swaggerui/ is mapped to folder /swagger-ui/
E0227 17:07:34.659701 1 cacher.go:149] unexpected ListAndWatch error: pkg/storage/cacher.go:115: Failed to list *api.Pod: 501: All the given peers are not reachable (failed to propose on members [https://172.31.29.111:2379] twice [last error: Get https://172.31.29.111:2379/v2/keys/registry/pods?quorum=false&recursive=true&sorted=true: remote error: bad certificate]) [0]
The certificate should be good though. If I execute an interactive shell within that Docker image, I can get the etcd URL via curl without any issues. So, what is going wrong in this case and how do I fix it?
I found I could solve this by using --etcd-config instead of --etcd-servers:
docker run -p 443:443 -v /etc/kubernetes:/etc/kubernetes:ro -v /etc/ssl/etcd:/etc/ssl/etcd:ro gcr.io/google_containers/hyperkube:v1.1.2 /hyperkube apiserver --bind-address=0.0.0.0 --insecure-bind-address=127.0.0.1 --etcd-config=/etc/kubernetes/etcd.client.conf --allow-privileged=true --service-cluster-ip-range=10.3.0.0/24 --secure-port=443 --advertise-address=172.31.29.111 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --kubelet-certificate-authority=/etc/ssl/etcd/ca.pem --kubelet-client-certificate=/etc/ssl/etcd/master1-master-client.pem --kubelet-client-key=/etc/ssl/etcd/master1-master-client-key.pem --client-ca-file=/etc/ssl/etcd/ca.pem --tls-cert-file=/etc/ssl/etcd/master1-master-client.pem --tls-private-key-file=/etc/ssl/etcd/master1-master-client-key.pem
etcd.client.conf:
{
"cluster": {
"machines": [ "https://172.31.29.111:2379" ]
},
"config": {
"certFile": "/etc/ssl/etcd/master1-master-client.pem",
"keyFile": "/etc/ssl/etcd/master1-master-client-key.pem"
}
}
I'm trying to deploy my application with docker and elastic beanstalk. My Dockerrun.aws.json file looks like
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "jvans/maven_weekly",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "5000"
}],
"Volumes": [
{
"HostDirectory": "/Users/jamesvanneman/Code/maven_weekly/maven_weekly",
"ContainerDirectory": "/maven_weekly"
}
],
"Logging": "/var/log/nginx"
}
I created this application with eb create and when I run eb deploy I get
Docker container quit unexpectedly after launch: Docker container quit
unexpectedly on Mon Sep 21 01:15:12 UTC 2015:. Check snapshot logs for details.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
In var/log/eb-activity.log I see the following errors:
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Mon Sep 21 01:08:52 UTC 2015:. Check snapshot logs for details. (ElasticBeanstalk::ExternalInvocationError)
caused by: 83ea9b7f9a069eeb8351fef7aaedb8374f7dfe300a5e0aaeba0fe17600583175
[2015-09-21T01:08:52.205Z] INFO [2246] - [Application deployment/StartupStage1/AppDeployEnactHook/00run.sh] : Activity failed.
So it seems like there's a problem with a startup script. If i ssh into the container and try to run it manually I don't really get any extra help from error messages.
eb ssh
sudo /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Mon Sep 21 01:34:52 UTC 2015:. Check snapshot logs for details.
Msg: Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Mon Sep 21 01:34:52 UTC 2015:. Check snapshot logs for details.
Are snapshot logs something different than what's in var/log/eb-activity.log? Any Idea what is going on/how I can debug this further?
Docker dumps are stored in the host box at /var/log/eb-docker/containers/.
Go there and you'll find the docker startup crash log that should indicate the root cause of your problem.
You want to look at
/var/log/eb-docker/containers/eb-current-app/unexpected-quit.log
in the bundle downloaded by eb logs --all or using eb ssh. This log file will have the stdout and stderr of your application before it crashed.
In my case, unexpected-quit.log was empty and the other log files in /var/log/ didn't help.
So I found out why the container was crashing by manually running the image on the server.
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ nano Dockerfile # or clone the entire project
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ sudo docker build -t test .
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ sudo docker run --rm test
Exception in thread "main" …
I fixed that exception, redeployed a new version and now the container starts without issues.
In my case I did an eb ssh to connect into the ec2 instance and looked at my server logs. It was an application error. Nothing to do with AWS. Fixed it. Tried again and it's working!
I don't know why but I cannot seem to figure out why this is happening. I can build and run the docker image locally.
Recent Events:
2015-05-25 12:57:07 UTC+1000 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.
2015-05-25 12:57:07 UTC+1000 INFO New application version was deployed to running EC2 instances.
2015-05-25 12:57:04 UTC+1000 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2015-05-25 12:57:04 UTC+1000 ERROR [Instance: i-4775ec9b] Command failed on instance. Return code: 1 Output: (TRUNCATED)... run Docker container: vel="fatal" msg="Error response from daemon: Cannot start container 02c057b331bf3a3d912bf064f1dca3e00c95746b5748c3c4a28a5c6b452ff335: [8] System error: exec: \"bin/app\": permission denied" . Check snapshot logs for details. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/04run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2015-05-25 12:57:03 UTC+1000 ERROR Failed to run Docker container: vel="fatal" msg="Error response from daemon: Cannot start container 02c057b331bf3a3d912bf064f1dca3e00c95746b5748c3c4a28a5c6b452ff335: [8] System error: exec: \"bin/app\": permission denied" . Check snapshot logs for details.
Dockerfile:
FROM java:8u45-jre
MAINTAINER Terence Munro <terry#zenkey.com.au>
ADD ["opt", "/opt"]
WORKDIR /opt/docker
RUN ["chown", "-R", "daemon:daemon", "."]
USER daemon
ENTRYPOINT ["bin/app"]
EXPOSE 9000
Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": "9000"
}
],
"Volumes": []
}
Additional logs as attachment at: https://forums.aws.amazon.com/thread.jspa?threadID=181270
Any help is extremely appreciated.
#nick-humrich suggestion of trying eb local run worked. So using eb deploy ended up working.
I had previously been uploading through the web interface.
Initially using eb deploy was giving me a ERROR: TypeError :: data must be a byte string but I found this issue which was resolved by uninstalling pyopenssl.
So I don't know why the web interface was giving me permission denied perhaps something to do with the zip file?
But anyway I'm able to deploy now thank you.
I had a similar problem running Docker on Elastic Beanstalk. When I pointed CMD in the Dockerfile to a shell script (/path/to/my_script.sh), the EB deployment would fail with
/path/to/my_script.sh: Permission denied.
Apparently, even though I had run RUN chmod +x /path/to/my_script.sh during the Docker build, by the time the image was run, the permissions had been changed. Eventually, to make it work I settled on:
CMD ["/bin/bash","-c","chmod +x /path/to/my_script.sh && /path/to/my_script.sh"]