I am trying to spin up a cluster via AWS Cloudera Director. Manager installs fine however upon tailing the installation I find this error.
[2016-06-06 17:16:24] ERROR [pipeline-thread-31] - c.c.l.p.DatabasePipelineRunner: Pipeline '4e04f8e6-5dfc-4603-b58b-9474e054bca6' failed
Any ideas?
Thanks in advance.
That error is a high-level error. Earlier in the log should be an error indicating the root cause of the failure.
Related
We have updated emr version for emr-5.30.0.Since then we are getting error in bootstrap.
"Terminated with bootstrap error"
If i change version back to emr-5.29.0 it work fine.I am not able to find reason for bootstrap error.
We are creating EMR cluster from step function.
We have changed version emr-5.29.0 to emr-5.30.0 as we are adding managed autoscalling and it supports only after 5.29.0
I checked logs but could not find any proper error message. Please suggest some pointers to troubleshoot this.
EMR version changes many thing including different applications you select to include like #Snighdhajyoti mentioned for example in emr 5.29.0 spark had version 2.4.4 and in emr 5.30.0 spark has version 2.4.5. You can see the basic list of application changes here.
But the point is, there can be some application or package that you install or configure in bootstrap scipt manually which might be conflicting with the other updated package.
For logs, bootstrap logs dont appear in cluster logs but are in stderr logs for your bootstrap action like below
s3://doc-example-bucket/cluster-id/node/instance-id/bootstrap-actions/
This link provides some more guidance how can you dig down the error, for example
If you can't determine why the script failed after reviewing the
stderr logs, modify your script to provide additional debug
information. For example, set the -ex parameters in the bash script.
This allows you to view the bash script flow in the bootstrap action
log files.
Note: If the failed bootstrap action isn't a bootstrap action that you
created (for example, if you created six bootstrap actions and the
error message is "bootstrap action 7 failed with non-zero exit code"),
it indicates that Amazon EMR couldn't install applications or start
services. This problem is rare. To resolve this issue, try launching
the cluster again.
I am trying to install CDH on GCP using Cloudera Altus director but it fails in the last step to configure cluster and throws an error of "Bootstrap failed" with error description as "Insufficient number of instances available after allocation".
I have followed all the steps mentioned in the following URL,but unable to deploy/install it.
https://www.cloudera.com/documentation/director/latest/topics/director_get_started_gcp.html
Tried to search for this error, but no solution found.
Can anybody help in this case?
I am trying to get my RoR app up and running on Elastic Beanstalk and am struggling to get the rgeo gem working. The error I am getting on the web server is:
I, [2015-09-28T11:26:54.982049 #21789] INFO -- : Completed 500 Internal Server Error in 5ms (ActiveRecord: 2.6ms)
F, [2015-09-28T11:26:54.983523 #21789] FATAL -- :
NoMethodError (undefined method `point' for nil:NilClass):
lib/app/weather_service.rb:61:in `block in get_location'
....
That error happens when the code accesses a model attribute that is backed by a POINT data type in the database. The error is typically due to a missing dependency, namely geos as described in this thread. So I connected to the underlying EC2 instance, installed geos and re-installed the rgeo gem. That resolved the issue in rails console:
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ cd /var/app/current
[ec2-user#ip-xxx-xxx-xxx-xxx current]$ rails c
Loading production environment (Rails 4.2.4)
irb(main):001:0> RGeo::Geos.supported?
=> true
That did not, however, resolve the error in the web server. I'm pretty sure I don't clearly understand the Elastic Beanstalk environment and maybe making direct changes to the underlying EC2 instance won't make dependencies available to the application instance. I do understand that I would need to either add the dependencies to either a custom AMI or ebextensions for future deployments, but I wanted to make sure I have the dependencies working before going through that process. Any guidance would be appreciated.
In case anyone else runs into this same issue, I did find a solution. These are the steps I followed:
Created an EC2 instance using AMI ami-bddbc48d
Installed the PostGIS dependencies using this helpful script https://gist.github.com/whyvez/8d19096712ea44ba66b0
Created a custom AMI from that instance
Updated my EBS environment configuration to use that new custom AMI
Voila! Problem solved
I've been dealing with this issue since yesterday. All working until 4 PM and suddenly after that this error keeps coming.
Creating application version archive "882a".
Uploading: [##################################################] 100% Done...
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Command execution completed on all instances successfully.
INFO: New application version was deployed to running EC2 instances.
ERROR: Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.
ERROR: Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.
No logs has been logged after that. Even there are no logs coming in AWS management console. I ssh the instance and checked all the logs /var/log/, they all have logs only until 4 PM.
So this is possibly a dead end I'm stuck with. I tried rebuilding the environment but no luck. The forum discusses this, but all troubleshooting is happening behind the screens, no possible solutions provided.
Any idea on how to resolve this?
I was following this tutorial for installing Cascading to EMR:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/CreateCascading.html
But it failed because of bootstrap action installing the cascading-sdk. The corresponding logs is here: http://pastebin.com/jybHssTQ. As seen from the logs, it's failed because of apt-get not found. Seriously?
I also checked the sdk installation script, and found option to disable installing screen with --no-screen. It is still failed, with different error http://pastebin.com/T6CvA2H1
And now it is because of permission denied. What?
It's official guide, but I can't seem to run it. Any idea?
Rather than changing the script first, try a different EMR AMI version.
AMI versions up until 2.4.8 use Debian OS, where apt-get will work, but this runs Hadoop 1.x. AMI versions 3.0.x run Hadoop 2.2 and use Amazon Linux, which uses Yum.
See Below:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/ami-versions-supported.html
Also, try to add the "--tmpdir" option to get around the "Permission Denied" error.