AWS Elasticsearch Service node tries to connect to unknown destination - amazon-web-services

Question

I have an AWS Elasticsearch Service Cluster running with the following specs:
version 7.9.1, service software release R20210426-P2
3 nodes, no dedicated master nodes
instance type: t3.medium.elasticsearch
I enabled the application log via Cloudwatch and noticed that every second the following message gets logged:
[2021-08-19T04:41:42,118][WARN ][r.suppressed] [NODE_ID] path: __PATH__ params: {redirectEndpoint=metrics}
java.net.ConnectException: Connection refused (Connection refused)
Complete Stacktrace:
[2021-08-17T09:30:42,333][WARN ][r.suppressed ] [NODE_ID] path: __PATH__ params: {redirectEndpoint=metrics}
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at org.conscrypt.AbstractConscryptSocket.connect(AbstractConscryptSocket.java:155) ~[conscrypt-openjdk-uber-1.4.0.jar:1.4.0]
at sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1071) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1069) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1068) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1507) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1505) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1504) ~[?:?]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) ~[?:?]
__AMAZON_INTERNAL__
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94) ~[elasticsearch-7.9.1.jar:7.9.1]
__AMAZON_INTERNAL__
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) [elasticsearch-7.9.1.jar:7.9.1]
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:264) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) [jetty-http-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) [jetty-io-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) [jetty-io-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.12.v20130726.jar:8.1.12.v20130726]
at java.lang.Thread.run(Thread.java:834) [?:?]
It seems like the node tries to connect to a server. At first I thought that one node can't reach another node in the cluster but even my development environment with just a single node logs this message every second.
I don't know if this is either an error caused by AWS ES / misconfiguration by myself or a general issue of the OpenDistro.
I would really appreciate if someone could help me understand this error message and/or could get me on the right track to solve this issue.
Any help is appreciated.
Thanks in advance!

Answer 1

Okay so interestingly this issue stopped occurring after I upgraded the instance type to m6g.large.elasticsearch due to high jvm pressure. However my development and staging single-node clusters (using a t3.small.elasticsearch instance) are still logging this exception.
I'm not 100% sure if upgrading your instance type is the key to solve this issue. For now I will accept my own answer. If I have new insights I'll let you know.