I killed the hiveserver2 process (after finding the PID with ps aux|grep -i hiveserver2) on my EMR cluster with one master and two workers. Before killing hiveserver2 I was able to browse and query Hive on my browser via HUE. I tried restarting with hive --service hiveserver2 but then I can't connect from HUE anymore and it either hangs or says that it can't connect to the <publicDNS>:10000
My use case is that I want to modify the hive configuration of my EMR cluster without shutting down the cluster. Is that possible at all?
initctl list
status hive-server2
sudo restart hive-server2
sudo stop hive-server2
sudo start hive-server2
How do I restart a service in Amazon EMR?
Hive configurations can be added before you launch your cluster, not after you have the cluster ready. You can add them as configuration settings in bootstrap step.
E.G. You can add your configurations in hive-site.xml using following syntax (in java):
Map<String,String> hiveProperties = new HashMap<String,String>();
hiveProperties.put("hive.vectorized.execution.enabled","true");
hiveProperties.put("hive.vectorized.execution.reduce.enabled","true");
hiveProperties.put("hive.execution.engine","Tez");
hiveProperties.put("hive.auto.convert.join","true");
hiveProperties.put("hive.exec.parallel","true");
Configuration myHiveConfig = new Configuration()
.withClassification("hive-site")
.withProperties(hiveProperties);
List <Application> apps = new ArrayList<Application>();
apps.add(new Application().withName("Hadoop"));
apps.add(new Application().withName("Hive"));
apps.add(new Application().withName("Spark"));
//apps.add(new Application().withName("Pig"));
//apps.add(new Application().withName("Zeppelin-Sandbox"));
RunJobFlowRequest request = new RunJobFlowRequest()
.withName("abc")
.withReleaseLabel(emrVersion) //"emr-4.3.0"
.withServiceRole("EMR_DefaultRole")
.withConfigurations(myHiveConfig)
.withInstances(
new JobFlowInstancesConfig()
.withInstanceCount(numberofInstances)
.withKeepJobFlowAliveWhenNoSteps(true)
.withTerminationProtected(false)
.withMasterInstanceType(mserverType)
.withSlaveInstanceType(sserverType)
)
.withApplications(apps)
.withJobFlowRole("EMR_EC2_DefaultRole")
.withSteps(generalSteps);
More details in link below:
http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html
Related
I want to connect to a document db which has TLS enabled .I could do that from a lambda function with the rds-combined-ca-bundle.pem copied with lambda code .I could not do the same with databricks as all the node of cluster should have this file when spark try to connect it always time out.I tried to create the init scripts by following below link
https://learn.microsoft.com/en-us/azure/databricks/kb/python/import-custom-ca-cert
However it does not help either .Let me know if any one has any clue on this kind of use case .
Note:I can connect to TLS disabled document-db from same databricks instance .
If you are experiencing connection time out errors when using an init script to import the rds-combined-ca-bundle.pem file on your Spark cluster, try the following steps:
Make sure that the rds-combined-ca-bundle.pem file is available on the driver node of your Spark cluster. The init script will only be executed on the driver node. You will encounter connection time out errors otherwise.
Use the --conf option when starting the spark-shell or spark-submit command to specify the location of the rds-combined-ca-bundle.pem file on the driver node. To specify the location of the rds-combined-ca-bundle.pem file, run:
spark-shell --conf spark.mongodb.ssl.caFile=path/to/rds-combined-ca-bundle.pem
Check the Spark cluster logs whether the init script is being executed correctly or if its encountering any errors.
I'm setting up a new dataproc server and using initilization-action to run a custom script. The script runs fine on 2 datanodes but not executing on master node.
Tried looking for logs under /var/log/dataprog-initilization-*.log but unable to find the file in the master node.
Has anyone else faced this issue before?
Thanks in advance!!
gcloud command:
gcloud dataproc clusters create test-cluster \
--region=us-central1 --zone=us-central1-a \
--master-machine-type=n1-standard-4 --master-boot-disk-size=200 \
--initialization-actions=gs://dp_init_data/init2.sh --initialization-action-timeout="2m" \
--num-workers=2 --worker-machine-type=n1-standard-8 --worker-boot-disk-size=200
DataNode error log:
2019-07-11 03:29:22,123 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool BP-268987178-10.32.1.248-1562675355441 (Datanode Uuid 71664f82-1d23-4184-b19b-28f86b01a251) service to exp-gcp-kerberos-m.c.exp-cdh-prod.internal/10.32.1.248:8051 Datanode denied communication with namenode because the host is not in the include-list: DatanodeRegistration(10.32.1.60:9866, datanodeUuid=71664f82-1d23-4184-b19b-28f86b01a251, infoPort=0, infoSecurePort=9865, ipcPort=9867, storageInfo=lv=-57;cid=CID-aee57974-1706-4b8c-9654-97da47ad0464;nsid=128710770;c=1562675355441)
According to your DataNode error log, seems you are expecting the init action to be run first on master, then workers. But init actions are run in parallel, you have to add logic to sync between master and workers. I think you can simply add some wait in workers, or if you want something more reliable, write a flag file in GCS when master init is done, check that file in workers.
I'm running emr-5.12.0, with Amazon 2.8.3, Hive 2.3.2, Hue 4.1.0, Livy 0.4.0, Spark 2.2.1 and Zeppelin 0.7.3 on 1 m4.large as my master node and 1 m4.large as core node.
I am trying to execute a bootstrap action that configures some parts of the cluster. One of these includes the line:
sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/alternatives/zeppelin-conf/interpreter.json
It makes sure that the Zeppelin uses python3.4 instead of python2.7. It works fine if I execute this in the terminal after SSH'ing to the master node, but it fails when I submit it as a Custom JAR step on the AWS Web interface. I get the following error:
ed: can't read /etc/alternatives/zeppelin-conf/interpreter.json
: No such file or directory
Command exiting with ret '2'
The same thing happens if I use
sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/zeppelin-conf/interpreter.json
Obviously I could just change it from the Zeppelin UI, but I would like to include it in the bootstrap action.
Thanks!
It turns out that a bootstrap action submitted throug the AWS EMR web interface is submitted as a regular EMR step, so it's only run on the master node. This can be seen if you click the 'AWS CLI export' in the cluster web interface. The intended bootstrap action is listed as a regular step.
Using the command line to launch a cluster with a bootstrap action bypasses this problem, so I've just used that.
Edit: Looking back at the web interface, it's pretty clear that I was adding regular steps instead of bootstrap actions. My bad!
I have configure cassandra-cluster locally and it works fine, following the same steps I configure cassandra-cluster on AWS on a ubuntu-server instance.
It works fine, but if I stop cassandra service from one node:
sudo service cassandra stop
And then I start it, this node never connect to the cluster again.
And it fails throwing the next error:
* could not access pidfile for Cassandra
My cassandra version is 3.7, so if I access to /etc/init.d/cassandra, so the cmd_patt is the next:
CMD_PATT="Dcassandra-pidfile=.*cassandra.pid"
Cassandra version: 3.7
Host: ubuntu server 14.04 (AWS).
You have to remove /var/run/cassandra folder hence it has wrong permissions:
sudo rm -rf /var/run/cassandra
Or you can fix permissions manually:
sudo chmod 750 /var/run/cassandra
Then start Cassandra as service:
sudo service cassandra start
Some explanations
Instructions of file permissions you can find here.
It is safe to delete that folder because it recreates with right permissions and content. But do not delete it once it works correct. It may result in loss of data or incorrect behavior.
chmod 750 decrypts as rwxr-x--- permissions. It allows read-write-execute to the user, read-execute to the group and nothing to others. For Cassandra, it is enough to set permissions so.
Stop cassandra service:
sudo service cassandra stop
Remove the default dataset:
sudo rm -rf /var/lib/cassandra/data/system/*
Start cassandra service:
sudo service cassandra start
I've just created EMR cluster and trying to create my first Impala table. Getting this error: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore. Any suggestion please? I did everything as documented by Amazon.
[ip-10-72-69-85.ec2.internal:21000] > connect localhost;
Connected to localhost:21000
Server version: impalad version 1.2.1 RELEASE (build d0bf3eae1df0f437bb4d0e44649293756ccdc76c)
[localhost:21000] > show tables;
Query: show tables
ERROR: AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.
[localhost:21000] >
I had the same error - after many troubles I've found the simple solution:
A. Check impala-state-store and impala-catalog daemons are running:
sudo service impala-state-store status
sudo service impala-catalog status
If not running - check the logs and be sure to activate them.
B.If they are running - simply type in your impala-shell:
invalidate metadata;
This command will update your catalog from the state store.
Then, you are ready to start!
Run the following command in the said order and reopen the Impala browser
sudo /etc/init.d/hive-metastore start
sudo /etc/init.d/impala-state-store start
And
sudo /etc/init.d/impala-catalog start
sudo /etc/init.d/impala-server start
I actually found the solution to this problem might be to just wait. I had this problem and had restarted everything impala with no luck. I even tried stopping all impala services and starting them in the recommended order (statestore first). Nothing helped but then after an amount of time of being left it started to work. I'm not sure what that time is but it was more than 5 minutes and less than an hour.
I would first recommend you check the logs at /mnt/var/log/apps. The error is likely related to the state-store, which can be restarted with the command below.
sudo service impala-state-store restart
I ran into the same error. The tutorial skipped a couple steps. Once in an impala-shell, create a database, then use the database, then create a table.