Elasticserch Master not discoverd exception - Version 2.3.0 - amazon-web-services

Question

This is the first time I am working with elasticsearch.
The following is my environment/configuration.
I have 3 EC2 Ubuntu 14.04 instances.
I have download and extracted elasticsearch-2.3.0.tar.gz.
I have changed elasticsearch.yml file under elasticsearch/config in each of the instance.
I have made the following changes in each of the elasticsearch.yml file.
3.1. EC2 Instance number 1 ( my client node)
cluster.name: MyCluster
node.name: Client
node.master: false
node.data: false
path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["aa.aa.aa.aa" , "aa.aa.aaa.aa" , "aaa.a.aa.aa"]
In the above bracket I have provide IP of all my 3 instances.
3.2. EC2 Instance number 2 ( my Master node)
cluster.name: MyCluster
node.name: Master
node.master: true
node.data: true
path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["aa.aa.aa.aa" , "aa.aa.aaa.aa" , "aaa.a.aa.aa"]
In the above bracket I have provide IP of all my 3 instances.
Note that I have made node.data: true (according to this link)
3.3. EC2 Instance number 3 ( my data node)
cluster.name: MyCluster
node.name: slave_1
node.master: false
node.data: true
path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["aa.aa.aa.aa" , "aa.aa.aaa.aa" , "aaa.a.aa.aa"]
In the above bracket I have provide IP of all my 3 instances.
After this configuration I run elasticsearch service on each instance starting from data node then master node and client node in the end.
If I check the node status using curl http://localhost:9200, I am getting json which states that the node is running.
But when I check the cluster health using curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' I am getting the following error on my client instance.
I hope I am clear with my question and I am going in the right direction.
Thankyou

Answer 1

Elasticsearch 2.0+ defaults to binding all sockets to localhost. This means, by default, nothing outside of that machine can talk to it.
This is explicitly for security purposes and simple development setups. Locally, it works great, but you need to configure it for your environment when it gets more serious. This is also why you can talk to the node via localhost. Basically, you want this when you want more than one node across other machines using the network settings. This works with ES 2.3+:
network:
bind_host: [ _local_, _global_ ]
publish_host: _global_
Then other nodes can talk to the public IP, but you still have localhost to simplify working with the node locally (e.g., you--the human--never have to know the IP when SSHed into a box).
As you are in EC2 with Elasticsearch 2.0+, I recommend that you install the cloud-aws plugin (future readers beware: this plugin is being broken into 3 separate plugins in ES 5.x!).
$ bin/plugin install cloud-aws
With that installed, you get a bit more awareness out of your EC2 instances. With this great power, you can add more detail to your ES configurations:
# Guarantee that the plugin is installed
plugin.mandatory: cloud-aws
# Discovery / AWS EC2 Settings
discovery
type: ec2
ec2:
availability_zones: [ "us-east-1a", "us-east-1b" ]
groups: [ "my_security_group1", "my_security_group2" ]
# The keys here need to be replaced with your keys
cloud:
aws
access_key: AKVAIQBF2RECL7FJWGJQ
secret_key: vExyMThREXeRMm/b/LRzEB8jWwvzQeXgjqMX+6br
region: us-east-1
node.auto_attributes: true
# Bind to the network on whatever IP you want to allow connections on.
# You _should_ only want to allow connections from within the network
# so you only need to bind to the private IP
node.host: _ec2:privateIp_
# You can bind to all hosts that are possible to communicate with the
# node but advertise it to other nodes via the private IP (less
# relevant because of the type of discovery used, but not a bad idea).
#node:
# bind_host: [ _local_, _ec2:privateIp_, _ec2:publicIp_, _ec2:publicDns_ ]
# publish_host: _ec2:privateIp_
This will allow them to talk by binding the IP address to what is expected. If you want to be able to SSH into those machines and communicate with ES over localhost (you probably do for debugging), then you will want the version commented out with _local_ as a bind_host in that list.