I have set up a cluster with 3 masters for the time being on AWS.
Here are the three /etc/elasticsearch/elasticsearch.yml files
1.master1
cluster.name: es-staging
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: _ec2:privateIp_
discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
discovery.ec2.host_type: private_ip
discovery.zen.hosts_provider: ec2
http.port: 9200
discovery.zen.minimum_master_nodes: 2
node.master: true
s3.client.default.endpoint: s3-eu-west-1.amazonaws.com
transport.tcp.port: 9300
node.name: elastic-master1-stg
action.auto_create_index: true
2.master2
cluster.name: es-staging
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: _ec2:privateIp_
discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
discovery.ec2.host_type: private_ip
discovery.zen.hosts_provider: ec2
http.port: 9200
discovery.zen.minimum_master_nodes: 2
node.master: true
s3.client.default.endpoint: s3-eu-west-1.amazonaws.com
transport.tcp.port: 9300
node.name: elastic-master2-stg
action.auto_create_index: true
3.master3
cluster.name: es-staging
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: _ec2:privateIp_
discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
discovery.ec2.host_type: private_ip
discovery.zen.hosts_provider: ec2
http.port: 9200
discovery.zen.minimum_master_nodes: 2
node.master: true
s3.client.default.endpoint: s3-eu-west-1.amazonaws.com
transport.tcp.port: 9300
node.name: elastic-master3-stg
action.auto_create_index: true
However, when on say master1:
curl -XGET http://10.11.11.118:9200/_cluster/health
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
I have installed the discovery-ec2 plugin
Turns out it needed a role attached to the instances with the following policy
{
"Statement": [
{
"Action": [
"ec2:DescribeInstances"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
Related
I am configuring Kubernetes based on aws ec2.
I use elasticsearch's packetbeat to get the geometric of clients accessing the service.
Istio is used as the service mesh of Kubernetes, and CLB is used for the load balancer.
I want to know the client ip accessing the service and the domain address the client accesses here.
my packetbeat.yml
setup.dashboards.enabled: true
setup.template.enabled: true
setup.template.settings:
index.number_of_shards: 2
packetbeat.interfaces.device: eth0
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.auto_promices_mode: true
packetbeat.interfaces.with_vlans: true
packetbeat.protocols:
- type: dhcpv4
ports: [67, 68]
- type: dns
ports: [53]
include_authorities: true
include_additionals: true
- type: http
ports: [80,5601,8081,8002,5000, 8000, 8080, 9200]
send_request: true
send_response: true
send_header: ["User-Agent", "Cookie", "Set-Cookie"]
real_ip_header: "X-Forwarded-For"
- type: mysql
ports: [3306, 3307]
- type: memcache
ports: [11211]
- type: redis
ports: [6379]
- type: pgsql
ports: [5432]
- type: thrift
ports: [9090]
- type: mongodb
ports: [27017]
- type: cassandra
ports: [9042]
- type: tls
ports: [443, 993, 995, 5223, 8443, 8883,8883, 9243, 15021, 15443, 32440]
send_request: true
send_response: true
send_all_headers: true
include_body_for: ["text/html", "application/json"]
packetbeat.procs.enabled: true
packetbeat.flows:
timeout: 30s
period: 10s
fields: ["server.domain"]
processors:
- include_fields:
fields:
- source.ip
- server.domain
- add_dokcer_metadata:
- add_host_metadata:
- add_cloud_metadata:
- add_kubernetes_metadata:
host: ${HOSTNAME}
indexers:
- ip_port:
matchers:
- field_format:
format: '%{[ip]}:%{[port]}'
# with version 7 of Packetbeat use the following line instead of the one above.
#format: '%{[destination.ip]}:%{[destination.port]}'
output.elasticsearch:
hosts: ${ELASTICSEARCH_ADDRESS}
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
pipeline: geoip-info
setup.kibana:
host: 'https://myhost:443'
my CLB listener
CLB has enabled proxy protocol.
But the packet beat doesn't bring me the data I want.
search for tls log
"client": {
"port": 1196,
"ip": "10.0.0.83"
},
"network": {
"community_id": "1:+ARNMwsOGxkBkrmWfCVawtA1GKo=",
"protocol": "tls",
"transport": "tcp",
"type": "ipv4",
"direction": "egress"
},
"destination": {
"port": 8443,
"ip": "10.0.1.77",
"domain": "my host domain"
},
search for flow.final: true
"event": {
"duration": 1051434189423,
"kind": "event",
"start": "2022-10-28T05:25:14.171Z",
"action": "network_flow",
"end": "2022-10-28T05:42:45.605Z",
"category": [
"network_traffic",
"network"
],
"type": [
"connection"
],
"dataset": "flow"
},
"source": {
"geo": {
"continent_name": "Asia",
"region_iso_code": "KR",
"city_name": "Has-si",
"country_iso_code": "KR",
"country_name": "South Korea",
"region_name": "Gg",
"location": {
"lon": 126.8168,
"lat": 37.2072
}
},
"port": 50305,
"bytes": 24174,
"ip": "my real ip address",
"packets": 166
},
I can find out if I search separately, but there are no two points of contact.
I would like to see the log of the above two combined.
The domain the client accesses + real client ip.
please help me..
Hello i use this playbook for create EC2 instance
i use this documentation to make my playbook:
https://docs.ansible.com/ansible/latest/collections/amazon/aws/ec2_module.html#parameter-instance_tags
###Example pour launch une instance###
###ansible-playbook --extra-vars "hosts=ansible-test"###
- hosts: localhost
##Init variable##
vars:
keypair: "AWS-KEYS-WEBSERVER001"
instance_type: t2.micro
hosts: "{{ hosts }}"
groups: "Web"
image: "ami-0ea4a063871686f37"
tasks:
- name: startup new instance
community.aws.ec2_instance:
key_name: "{{ keypair }}"
security_group: "Web"
instance_type: "{{ instance_type }}"
name: "fermela"
image_id: "{{ image }}"
wait: true
region: "eu-west-3"
network:
assign_public_ip: true
vpc_subnet_id: "subnet-0c82e6027833af6cc"
register: ec2
- debug :
var: ec2
my playbook works and the debug give me output like :
ok: [localhost] => {
"ec2": {
"changed": false,
"changes": [],
"failed": false,
"instance_ids": [
"i-0ecb077aafd7fda1c"
],
"instances": [
{
"ami_launch_index": 0,
"architecture": "x86_64",
"block_device_mappings": [
{
"device_name": "/dev/xvda",
"ebs": {
"attach_time": "2021-02-23T00:34:53+00:00",
"delete_on_termination": true,
"status": "attached",
"volume_id": "vol-0ad6a503c6cfa7f97"
}
}
],
I would like manipulate this output for only dispaly ip_public
Can someone help me plz ?
I already try debug with var: ec2.public_ip but doesn't work
ec2_instance returns a list of instances, in this case you have just one instance. Try as below:
- debug :
var: ec2.instacnes[0].public_ip_address
You can't do this as in your output there is no public ip address.
Try this
use ansible_facts to register
- debug :
ansible_facts['ansible_ec2_public_ipv4']
Only those variables can be accessed that are shown in debug after registering.
So I'm using Ansible on my MBP to try create key_pair and create/provision EC2 instances. Playbook runs fine with no error but when I check AWS console there is no new key and no new instance... Ping to supposedly created Public IP times out so I am thinking something failed. Ansible definitely hit AWS since if I disable the AWS access key then Ansible errors out, and not using the Ansible created key in the second task also fails so a key must have been created, just not uploaded to AWS?
Can you spot anything I did wrong?
Playbook yaml content:
- name: Create a sandbox instance
hosts: localhost
gather_facts: False
vars:
instance_type: t2.micro
image: ami-d1315fb1
region: us-west-1
tasks:
- name: Generate key
ec2_key:
name: ansible_key
region: "{{ region }}"
aws_access_key: #my_key
aws_secret_key: #my_key
state: present
- name: Launch instance
ec2:
key_name: ansible_key
group: default
instance_type: "{{ instance_type }}"
image: "{{ image }}"
wait: true
region: "{{ region }}"
aws_access_key: #my_key
aws_secret_key: #my_key
register: ec2
- name: Print all ec2 variables
debug: var=ec2
Playbook runs fine with output being:
PLAY [Create a sandbox instance] ***********************************************
TASK [Generate key] ************************************************************
ok: [localhost]
TASK [Launch instance] *********************************************************
changed: [localhost]
TASK [Print all ec2 variables] *************************************************
ok: [localhost] => {
"ec2": {
"changed": true,
"instance_ids": [
"i-0898f09f8d3798961"
],
"instances": [
{
"ami_launch_index": "0",
"architecture": "x86_64",
"block_device_mapping": {
"/dev/sda1": {
"delete_on_termination": true,
"status": "attached",
"volume_id": "vol-04e9c4c4f5d85e60d"
}
},
"dns_name": "ec2-54-215-253-115.us-west1.compute.amazonaws.com",
"ebs_optimized": false,
"groups": {
"sg-778b5711": "default"
},
"hypervisor": "xen",
"id": "i-0898f09f8d3798961",
"image_id": "ami-d1315fb1",
"instance_type": "t2.micro",
"kernel": null,
"key_name": "ansible_key",
"launch_time": "2017-08-16T16:57:09.000Z",
"placement": "us-west-1b",
"private_dns_name": "ip-172-31-29-166.us-west1.compute.internal",
"private_ip": "172.31.29.166",
"public_dns_name": "ec2-54-215-253-115.us-west1.compute.amazonaws.com",
"public_ip": "54.215.253.115",
"ramdisk": null,
"region": "us-west-1",
"root_device_name": "/dev/sda1",
"root_device_type": "ebs",
"state": "running",
"state_code": 16,
"tags": {},
"tenancy": "default",
"virtualization_type": "hvm"
}
],
"tagged_instances": []
}
}
PLAY RECAP *********************************************************************
localhost : ok=3 changed=1 unreachable=0 failed=0
Here are the few things:
- sure, you have selected the N.California(us-west-1) region from the console
- For private part of the key to store inside the .ssh under your username, do the following steps:
- name: Create an EC2 key
ec2_key:
name: "ansible_key"
region: "us-west-1"
aws_access_key: #my_key
aws_secret_key: #my_ke
register: ec2_key
- name: save private key
copy:
content: "{{ ec2_key.key.private_key }}"
dest: "/Users/{{lookup('env', 'USER')}}/.ssh/aws-private.pem"
mode: 0600
when: ec2_key.changed
Note: Run this playbook from the scratch to create new key and save it into your ~/.ssh directory.
I am unable to form a ES cluster between 2 master nodes in EC2 instances. Following is the elasticsearch.yml for the nodes.
Node1:
bootstrap.memory_lock: true
cloud.aws.protocol: http
cloud.aws.proxy.host: <Proxy addr>
cloud.aws.proxy.port: <proxy port>
cloud.aws.region: us-east
cluster.name: production-test
discovery.ec2.availability_zones: us-east-1a,us-east-1b,us-east-1d,us-east-1e
discovery.zen.ping_timeout: 30s
discovery.ec2.tag.Name: <ec2-tag name>
discovery.zen.hosts_provider: ec2
#discovery.type: ec2
#discovery.zen.ping.multicast.enabled: false
http.port: 9205
#network.host: _eth0_, _local_, _ec2_
network.host: <private ip_addr>
#network.bind_host: <private ip_addr>
#network.publish_host: <private ip_addr>
node.data: true
node.master: true
plugin.mandatory: discovery-ec2, repository-s3
transport.tcp.port: 9305
#discovery.zen.ping.unicast.hosts: ["<private ip_addr of node1>","<private ip_addr of node2>"]
discovery.zen.ping.unicast.hosts: ["<private ip_addr of node1>:9305", "<private ip_addr of node2>:9305"]
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
node.name: nodetest1
path.data: /var/lib/elasticsearch/
#path.data: /data/elasticsearch/data/production
path.logs: /var/log/elasticsearch/
path.conf: /etc/elasticsearch
Node 2:
bootstrap.memory_lock: true
cloud.aws.protocol: http
cloud.aws.proxy.host: <Proxy addr>
cloud.aws.proxy.port: <Procy port>
cloud.aws.region: us-east
cluster.name: production-test
discovery.ec2.availability_zones: us-east-1a,us-east-1b,us-east-1d,us-east-1e
discovery.zen.ping_timeout: 30s
discovery.ec2.tag.Name: <ec2-instance tag name>
discovery.zen.hosts_provider: ec2
#discovery.type: ec2
#discovery.zen.ping.multicast.enabled: false
http.port: 9205
#network.host: _eth0_, _local_, _ec2_
network.host: <private ip_addr>
#network.bind_host: <private ip_addr>
#network.publish_host: <private ip_addr>
node.data: true
node.master: true
plugin.mandatory: discovery-ec2, repository-s3
transport.tcp.port: 9305
discovery.zen.ping.unicast.hosts: ["<private ip_addr of node1>:9305","<private ip_addr of node2>:9305"]
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
node.name: nodetest2
#Paths to log, conf, data directories
When both the nodes are started, the following is the log data on both the nodes:
[INFO ][o.e.b.BootstrapCheck ] [nodetest1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[WARN ][o.e.n.Node ] [nodetest1] timed out while waiting for initial discovery state - timeout: 30s
[INFO ][o.e.h.HttpServer ] [nodetest1] publish_address {<private ip_addr of node1>:9205}, bound_addresses {<private ip_addr of node1>:9205}
[INFO ][o.e.n.Node ] [nodetest1] started
[INFO ][o.e.d.z.ZenDiscovery ] [nodetest1] failed to send join request to master [{nodetest}{YcGzQ-4CQtmuuxUGMQJroA}{yuxHmvGPTeK-iw59VTj4ZA}{<private ip_addr of node2>}{<private ip_addr of node2>:9305}{aws_availability_zone=us-east-1d}], reason [RemoteTransportException[[nodetest][<private ip_addr of node2>:9305][internal:discovery/zen/join]]; nested: NotMasterException[Node [{nodetest}{YcGzQ-4CQtmuuxUGMQJroA}{yuxHmvGPTeK-iw59VTj4ZA}{<private ip_addr of node2>}{<private ip_addr of node2>:9305}{aws_availability_zone=us-east-1d}] not master for join request]; ], tried [3] times
I have searched many similar issues and tried to apply the fixes but i still have the same result. Is there any fault in the elasticsearch.yml file?
curl -XGET <private ip_addr>:9205/_cat/master
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
The two node instances are running ES-5.1.1 and are in the same security-group and iam role.
Any suggestions are highly appreciated.
Thanks,
My objective is to run a 6 node cluster on three instances in EC2.
I am placing one master-only and one data-only node on each instance (using the elastic ansible playbook).
The master nodes from each of the three instances all find each other without issue using EC2 discovery and form a cluster of three and elect a master.
The data nodes from the same instances fail on startup with the error below.
What have I tried
- switching data nodes to explicit zen.unicast discovery via hostnames works
- I can telnet on port 9301 from instance A->B without issue
REFERENCE:
java version - OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
es version - 2.1.0
data node elasticseach.yml
bootstrap.mlockall: false
cloud.aws.region: us-east
cluster.name: my-cluster
discovery.ec2.groups: stage-elasticsearch
discovery.ec2.host_type: private_dns
discovery.ec2.ping_timeout: 30s
discovery.type: ec2
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
gateway.expected_nodes: 4
http.port: 9201
network.host: _ec2:privateDns_
node.data: true
node.master: false
transport.tcp.port: 9301
node.name: ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1
master node elasticsearch.yml
bootstrap.mlockall: false
cloud.aws.region: us-east
cluster.name: my-cluster
discovery.ec2.groups: stage-elasticsearch
discovery.ec2.host_type: private_dns
discovery.ec2.ping_timeout: 30s
discovery.type: ec2
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
gateway.expected_nodes: 4
http.port: 9200
network.host: _ec2:privateDns_
node.data: false
node.master: true
transport.tcp.port: 9300
node.name: ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-master
Errors from datanode startup:
[2016-03-02 15:45:06,246][INFO ][node ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] initializing ...
[2016-03-02 15:45:06,679][INFO ][plugins ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] loaded [cloud-aws], sites [head]
[2016-03-02 15:45:06,710][INFO ][env ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [11.5gb], net total_space [14.6gb], spins? [no], types [ext4]
[2016-03-02 15:45:09,597][INFO ][node ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] initialized
[2016-03-02 15:45:09,597][INFO ][node ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] starting ...
[2016-03-02 15:45:09,678][INFO ][transport ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] publish_address {ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1/xxx-xxx-xx-xxx:9301}, bound_addresses {xxx-xxx-xx-xxx:9301}
[2016-03-02 15:45:09,687][INFO ][discovery ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] my-cluster/PNI6WAmzSYGgZcX2HsqenA
[2016-03-02 15:45:09,701][WARN ][com.amazonaws.jmx.SdkMBeanRegistrySupport]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
at java.security.AccessController.checkPermission(AccessController.java:559)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.registerMBean(MBeans.java:50)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:355)
at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:879)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-03-02 15:45:09,703][WARN ][com.amazonaws.metrics.AwsSdkMetrics]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
at java.security.AccessController.checkPermission(AccessController.java:559)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.isRegistered(MBeans.java:98)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.isMBeanRegistered(SdkMBeanRegistrySupport.java:46)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:361)
at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:879)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-03-02 15:45:39,688][WARN ][discovery ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] waited for 30s and no initial state was set by the discovery
[2016-03-02 15:45:39,698][INFO ][http ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] publish_address {ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1/xxx-xxx-xx-xxx:9201}, bound_addresses {xxx-xxx-xx-xxx:9201}
[2016-03-02 15:45:39,699][INFO ][node ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] started
I fixed this by removing the explicit setting of transport.tcp.port: 9300 and using the default of letting it pick any ports in the range 9300-9399.
The warnings from the AwsSdkMetrics remain but are NOT an issue as Val stated.
This is not actually an error.
See this issue where this has been reported. It just seems the plugin is logging too much.
If you modify your logging.yml config file as suggested in that issue with this, then you'll be fine:
# aws will try to do some sketchy JMX stuff, but its not needed.
com.amazonaws.jmx.SdkMBeanRegistrySupport: ERROR
com.amazonaws.metrics.AwsSdkMetrics: ERROR