How to talk to aws elasticsearch service using elastic java client? - amazon-web-services

Question

I have set up a elasticsearch server using AWS elasticsearch service (Not EC2). It gave me an endpoint https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com/ and if I click this endpoint(Note that there is no port specified) I can get the expected
{
status: 200,
name: "Mastermind",
cluster_name: "xxxx",
version: {
number: "1.5.2",
build_hash: "yyyyyy",
build_timestamp: "2015-04-27T09:21:06Z",
build_snapshot: false,
lucene_version: "4.10.4"
},
tagline: "You Know, for Search"
}
The question is how do I get this through the elasticsearch java client without a port number? The sample code I get is
Client client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300));
If I use this code and just replace "host1" with my endpoint, I'll get "NoNodeAvailableException"
ps:
The java client version I'm using is 2.0.0.
Edit
I finally decided to go with Jest, a 3rd party REST client. But what Brooks answered below is also very helpful - AWS do use port 80 for http and 443 for https. The blocker for me was the firewall I guess.
Edit2
The AWS ES service documentation explicitly says:
The service supports HTTP on port 80, but does not support TCP transport.

Answer 1

Believe it or not, AWS doesn't launch Elasticsearch using 9200 and 9300. It's launched via plain old port 80.
So, to demonstrate, try this...
curl -XPOST "http://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:80/myIndex/myType" -d '["name":"Edmond"}'
Or
curl -XPOST "https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:443/myIndex/myType" -d '["name":"Edmond"}'
It should respond with:
{"_index":"myIndex","_type":"myType","_id":"SOME_ID_#","_version":1,"created":true}
Check in Kibana and you'll see it's there.
So, then in your code, it should be:
Client client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("xxx-xxxxxxxx.us-west-2.es.amazonaws.com"), 80));
Unfortunately, I don't off-hand know how to transmit encrypted via SSL/HTTPS using the transport client. You could try using regular REST calls instead using JERSEY.
Finally, make sure your Elasticsearch access policy is configured properly. Something along the lines of:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:yyyyyyy:domain/myDomain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:yyyyyyyyy:domain/myDomain"
}
]
}
NOTE: The above access policy is completely wide open and is not recommended for anything remotely close to production. Just so you know....

Answer 2

Managed elastic search service in AWS does not provide the port for the transport protocol until now.
This question has been answered here ,
Elastic Transport client on AWS Managed ElasticSearch1
There is also a discussion in the AWS forum regarding the transport protocol. Here is the link
What is the port for the transport protocol ?

Answer 3

After lot of search i found an example which used a GET request, so I made minor changes to it for allowing POST requests so that complex queries can be submitted via POST body. The implementation is available at https://github.com/dy10/aws-elasticsearch-query-java
Apart from properly configuring access to you AWS ES (i.e. dont open it to Public), make sure to use https (the above code uses http; just replace http with https in the code and it will work).
Another useful looking but partial implementation is at https://github.com/aws/aws-sdk-java/issues/861