How to properly configure HDFS high availability using Zookeeper? - hdfs

Question

I'm a student of big data. I'm coming to you today for a question about the high availability of HDFS using Zookeeper. I am aware that there has already been bcp of topic dealing with this subject, I have read a lot of them already. It's already been 15 days that I've been browsing the forums without finding what I'm looking for (maybe I'm not looking in the right place too ;-) )
I have followed the procedure three times here: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html.
I may have done everything right, but when I kill one of my namenodes, none of them take over.
My architecture is as follows:
- 5 VM
- VM 1,3 and 5 are namenodes
- VMs 1 to 5 are datanodes.
I launched my journalnodes, I started my DFSZKFailoverController, I formatted my first namenode, I copied with -bootstrapStandby the configuration of my first namenode to the 2 others and I started my cluster.
Despite all this and no obvious problems in the ZKFC and namenode logs, I can't get a namenode to take over a dying namenode.
Does anyone have any idea how to help me?
Many thanks for your help :)
zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=2
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/zookeeper/zoo
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
admin.serverPort=7979
server.1=10.10.10.15:2888:3888
server.2=10.10.10.16:2888:3888
server.3=10.10.10.17:2888:3888
server.4=10.10.10.18:2888:3888
server.5=10.10.10.19:2888:3888
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- default configuration -->
<property>
<name>fs.default.name</name>
<value>hdfs://my-cluster</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<!-- zookeeper configuration -->
<property>
<name>ha.zookeeper.quorum</name>
<value>10.10.10.15:2181,10.10.10.16:2181,10.10.10.17:2181,10.10.10.18:2181,10.10.10.19:2181</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- cluster configuration -->
<property>
<name>dfs.nameservices</name>
<value>my-cluster</value>
</property>
<!-- namenode configuration -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hdfs/data/nameNode</value>
</property>
<!-- datanode configuration -->
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hdfs/data/dataNode</value>
</property>
<!-- secondary namenode configuration -->
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/home/hdfs/data/secondaryNameNode</value>
</property>
<!-- replication factor -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- webhdfs connector -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<!-- high-availability configuration -->
<property>
<name>dfs.ha.namenodes.my-cluster</name>
<value>nn1,nn2,nn3</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn1</name>
<value>10.10.10.15:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn2</name>
<value>10.10.10.19:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn3</name>
<value>10.10.10.17:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn1</name>
<value>10.10.10.15:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn2</name>
<value>10.10.10.19:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn3</name>
<value>10.10.10.17:9870</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://10.10.10.15:8485;10.10.10.19:8485;10.10.10.17:8485/my-cluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hdfs/data/journalNode</value>
</property>
<!-- failover configuration -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.my-cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hdfsuser/.ssh/id_rsa</value>
</property>
</configuration>
dfs.service
[Unit]
Description=Hadoop DFS namenode and datanode
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
Requires=network-online.target
[Service]
User=hdfsuser
Group=hdfsgroup
Type=simple
ExecStart=/apps/hadoop/sbin/start-dfs.sh
ExecStop=/apps/hadoop/sbin/stop-dfs.sh
RemainAfterExit=yes
Restart=on-failure
StartLimitInterval=350
StartLimitBurst=10
[Install]
WantedBy=multi-user.target
hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log
(before i crash a namenode)
2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.1
STARTUP_MSG: classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hado$STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG: java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoo$2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 wat$2020-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.$2020-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log
(after i crash a namenode)
2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.1
STARTUP_MSG: classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/commons-net-$STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG: java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/$2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.$2020-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, i$2020-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
2020-04-09 13:38:59,910 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.io.EOFException: End of File Exception between local host is: "node15-hdfs-spark-master/10.10.10.15"; destination host is: "hdfs-0":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:791)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1850)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1183)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079)
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,938 WARN org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Can't get local NN thread dump due to Connexion refusée (Connection refused)
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hdfs-0/10.10.10.15:9000 and marking that fencing is necessary
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ZooKeeper: Session: 0x50000a3038f0000 closed
2020-04-09 13:38:59,947 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x50000a3038f0000
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x50000a3038f0000
2020-04-09 13:39:01,951 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=10$2020-04-09 13:39:01,953 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 8 more
2020-04-09 13:39:03,956 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=10$2020-04-09 13:39:03,958 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 8 more
...
...
...

Answer 1

The problem with my configuration finally came from two commands that hadn't been installed when I installed the hadoop cluster:
first the nc command: fixed by installing the nmap package from yum
then the command fuser: fixed by installing the psmisc package from yum
yum install -y nmap.x86_64
yum install -y psmisc.x86_64
Hopefully it will help other of you soon.