ElasticSearch cloud-aws plugin not able to join cluster - amazon-web-services

Question

So i have been attempting to use the ElasticSearch "cloud-aws" plugin to join elasticsearch nodes to my single master. I have been though a few online guides and tried a few settings from various sources but I still can't get the new nodes to join the existing master.
I have configured IAMS roles and tags for EC2 and this is my elasticsearch.yml file on one node (the others are similar):
node.name: Thor
node.client: "true"
network.host: localhost
cloud.aws.access_key: foobar
cloud.aws.secret_key: barfoo
cloud.aws.region: eu-west-1
discovery.type: ec2
discovery.ec2.tag.elasticsearch: Ubuntu-ElasticNode
The logging from elasticsearch is poor and even in DEBUG mode not much is offered up.
[2016-03-15 23:01:05,440][INFO ][node ] [Thor] version[2.2.0], pid[1550], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-03-15 23:01:05,447][INFO ][node ] [Thor] initializing ...
[2016-03-15 23:01:06,685][INFO ][plugins ] [Thor] modules [lang-expression, lang-groovy], plugins [cloud-aws], sites []
[2016-03-15 23:01:10,016][INFO ][node ] [Thor] initialized
[2016-03-15 23:01:10,017][INFO ][node ] [Thor] starting ...
[2016-03-15 23:01:10,106][INFO ][transport ] [Thor] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2016-03-15 23:01:10,115][INFO ][discovery ] [Thor] elasticsearch/9PmYq5tXQcaPUPqDh4VTSQ
[2016-03-15 23:01:40,116][WARN ][discovery ] [Thor] waited for 30s and no initial state was set by the discovery
[2016-03-15 23:01:40,155][INFO ][http ] [Thor] publish_address {localhost/127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2016-03-15 23:01:40,155][INFO ][node ] [Thor] started
[2016-03-15 23:54:40,863][DEBUG][action.admin.cluster.health] [Thor] no known master node, scheduling a retry
[2016-03-15 23:55:10,864][DEBUG][action.admin.cluster.health] [Thor] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])
[2016-03-15 23:55:10,874][INFO ][rest.suppressed ] /_cluster/health Params: {pretty=}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:205)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:794)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I have the port range 9200 - 9400 open between the elasticsearch servers but the log seems to indicate that the discovery is still timing out. I set "discovery.ec2.tag.*" to speed up the discovery process but this hasn't helped.
Does anyone have any idea how this plugin needs to be configured? I have read a few guides and a lot use even less configuration options than I and are still able to join nodes to the master.

Answer 1

I'm running ElasticSearch 2.2. Here's an example of my working config:
plugin.mandatory: cloud-aws
cluster.name: mynewcluster
cloud.aws.access_key: mykey
cloud.aws.secret_key: mysecret
cloud.aws.region: us-east-1
discovery.type: ec2
discovery.ec2.tag.elasticsearch: mynewcluster
I think you need to look at your network config. Specificallynetwork.host. From the docs:
Elasticsearch binds to localhost only by default. This is sufficient for you to run a local development server (or even a development cluster, if you start multiple nodes on the same machine), but you will need to configure some basic network settings in order to run a real production cluster across multiple servers.
I don't have the network.host config in my elasticsearch.yml. Which leads me to suggest taking it out altogether. However, since the docs say that it binds to localhost by default I would also suggest that you try to set it to the public hostname or IP of the node.
All of this assumes that you correctly set up IAM, Security Groups according to: https://github.com/elastic/elasticsearch-cloud-aws

Answer 2

So after having this chat in the es forums: https://discuss.elastic.co/t/cloud-aws-plugin-not-able-to-join-cluster/44703/3
I decided to rebuild the node cleanly as i suspected a Java downgrade from 8 to 7 to allow the cloud-aws plugin to work may be causing the issue and i had tried to many failed fixes. I also (from advise in the link provided) installed the marvel-agent and license plugins but I haven't seen any one else do this to get discovery working so I am not sure this is a requirement. I also made sure to hold es package upgrades because the marvel plugin did a bit of complaining when es upgraded (although the plugin could also have been upgraded so just a personal preference really).
Discovery is now working very well.