Getting below exception while trying to delete an object in amazon S3. Can someone throw some light on what is causing this and how to resolve this issue?
Looking at the exception it looks like client side exception. Is it because that particular thread unable to get a connection from connection pool?
Using Below Maven Dependency
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.1.4</version>
</dependency>
Exception Details:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to bucketname.s3.region.amazonaws.com.cn:443 [bucketname.s3.region.amazonaws.com.cn/52.82.188.56] failed: connect timed out at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.handleThrownException(RetryableStage.java:140) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:96) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:44) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:51) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:33) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:79) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.s3.DefaultS3Client.deleteObject(DefaultS3Client.java:868) atcom.abc.magnet.objectstore.impl.ObjectStoreImpl.deleteFile(ObjectStoreImpl.java:114) at com.abc.magnet.job.PurgeVideos.purgeVideos(PurgeVideos.java:83) atcom.abc.magnet.job.PurgeVideos.run(PurgeVideos.java:59) at com.abc.magnet.Main.init(Main.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1737) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:576) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:846) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:863) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546) at com.abc.magnet.Main.main(Main.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at pie.spark.orchestra.driver.app.SparkDriverWorker.invokeMainMethodWithProxyUserCheck(SparkDriverWorker.java:162) at pie.spark.orchestra.driver.app.SparkDriverWorker.runJob(SparkDriverWorker.java:132) at pie.spark.orchestra.driver.app.SparkDriverWorker.run(SparkDriverWorker.java:70) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to bucketname.s3.region.amazonaws.com.cn:443 [bucketname.s3.region.amazonaws.com.cn/52.82.188.56] failed: connect timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:80) at com.sun.proxy.$Proxy64.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:207) at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:95) at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:188) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:66) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:51) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:35) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:63) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:115) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:88) ... 54 more Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory.connectSocket(SdkTlsSocketFactory.java:113) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 83 more
Related
could you please help me with the Amazon Athena to HBase connection.
I work with respect to the documentation while trying to setup data source. The connector successfully connects to the HBase instance on the Amazon EMR cluster, fetches table structure... Both columns and rows ids natively have strings as a values, but I can not prevent connector (or maybe Athena) to cast data types to some different ones: I have BIGINT or DOUBLE inside Athena table structure. Then I put to AWS Glue corresponding table structure: I have appropriate database and table inside, that equals to the names inside HBase.
But connector can't connect to AWS Glue with the following error:
com.amazonaws.SdkClientException: Unable to execute HTTP request:
Connect to glue.us-east-2.amazonaws.com:443 [glue.us-east-2.amazonaws.com/3.143.132.217, glue.us-east-2.amazonaws.com/18.188.134.211, glue.us-east-2.amazonaws.com/3.12.12.20, glue.us-east-2.amazonaws.com/3.23.23.240, glue.us-east-2.amazonaws.com/3.23.109.251, glue.us-east-2.amazonaws.com/3.129.154.43] failed: connect timed out
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[task/:?]
at com.amazonaws.services.glue.AWSGlueClient.doInvoke(AWSGlueClient.java:8202) ~[task/:?]
at com.amazonaws.services.glue.AWSGlueClient.invoke(AWSGlueClient.java:8169) ~[task/:?]
at com.amazonaws.services.glue.AWSGlueClient.invoke(AWSGlueClient.java:8158) ~[task/:?]
at com.amazonaws.services.glue.AWSGlueClient.executeGetTable(AWSGlueClient.java:4774) ~[task/:?]
at com.amazonaws.services.glue.AWSGlueClient.getTable(AWSGlueClient.java:4745) ~[task/:?]
at com.amazonaws.athena.connector.lambda.handlers.GlueMetadataHandler.doGetTable(GlueMetadataHandler.java:347) ~[task/:?]
at com.amazonaws.athena.connectors.hbase.HbaseMetadataHandler.doGetTable(HbaseMetadataHandler.java:206) [task/:?]
at com.amazonaws.athena.connector.lambda.handlers.MetadataHandler.doHandleRequest(MetadataHandler.java:256) [task/:?]
at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:132) [task/:?]
at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:100) [task/:?]
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:902) [LambdaSandboxJava-1.0.jar:?]
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:340) [LambdaSandboxJava-1.0.jar:?]
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:63) [LambdaSandboxJava-1.0.jar:?]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_302]
at java.lang.Class.forName(Class.java:348) [?:1.8.0_302]
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:150) [LambdaJavaRTEntry-1.0.jar:?]
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to glue.us-east-2.amazonaws.com:443 [glue.us-east-2.amazonaws.com/3.143.132.217, glue.us-east-2.amazonaws.com/18.188.134.211, glue.us-east-2.amazonaws.com/3.12.12.20, glue.us-east-2.amazonaws.com/3.23.23.240, glue.us-east-2.amazonaws.com/3.23.109.251, glue.us-east-2.amazonaws.com/3.129.154.43] failed: connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[task/:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[task/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_302]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_302]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_302]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_302]
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[task/:?]
at com.amazonaws.http.conn.$Proxy18.connect(Unknown Source) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[task/:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[task/:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[task/:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[task/:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[task/:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[task/:?]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[task/:?]
... 23 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_302]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_302]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_302]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_302]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_302]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_302]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[task/:?]
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142) ~[task/:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[task/:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[task/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_302]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_302]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_302]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_302]
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[task/:?]
at com.amazonaws.http.conn.$Proxy18.connect(Unknown Source) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[task/:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[task/:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[task/:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[task/:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[task/:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[task/:?]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330) ~[task/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[task/:?]
I have an AWS Elasticsearch Service Cluster running with the following specs:
version 7.9.1, service software release R20210426-P2
3 nodes, no dedicated master nodes
instance type: t3.medium.elasticsearch
I enabled the application log via Cloudwatch and noticed that every second the following message gets logged:
[2021-08-19T04:41:42,118][WARN ][r.suppressed] [NODE_ID] path: __PATH__ params: {redirectEndpoint=metrics}
java.net.ConnectException: Connection refused (Connection refused)
Complete Stacktrace:
[2021-08-17T09:30:42,333][WARN ][r.suppressed ] [NODE_ID] path: __PATH__ params: {redirectEndpoint=metrics}
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at org.conscrypt.AbstractConscryptSocket.connect(AbstractConscryptSocket.java:155) ~[conscrypt-openjdk-uber-1.4.0.jar:1.4.0]
at sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1071) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1069) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1068) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1507) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1505) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1504) ~[?:?]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) ~[?:?]
__AMAZON_INTERNAL__
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94) ~[elasticsearch-7.9.1.jar:7.9.1]
__AMAZON_INTERNAL__
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) [elasticsearch-7.9.1.jar:7.9.1]
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:264) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) [jetty-http-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) [jetty-io-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) [jetty-io-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.12.v20130726.jar:8.1.12.v20130726]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.12.v20130726.jar:8.1.12.v20130726]
at java.lang.Thread.run(Thread.java:834) [?:?]
It seems like the node tries to connect to a server. At first I thought that one node can't reach another node in the cluster but even my development environment with just a single node logs this message every second.
I don't know if this is either an error caused by AWS ES / misconfiguration by myself or a general issue of the OpenDistro.
I would really appreciate if someone could help me understand this error message and/or could get me on the right track to solve this issue.
Any help is appreciated.
Thanks in advance!
Okay so interestingly this issue stopped occurring after I upgraded the instance type to m6g.large.elasticsearch due to high jvm pressure. However my development and staging single-node clusters (using a t3.small.elasticsearch instance) are still logging this exception.
I'm not 100% sure if upgrading your instance type is the key to solve this issue. For now I will accept my own answer. If I have new insights I'll let you know.
ERROR {org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager} - Cannot create connection to LDAP server. Connection URL: ldap://xx.xxx.xx.x:xxx Error message: Error occurred while obtaining LDAP connection, LDAP connection circuit breaker state set to: open
INFO {org.wso2.carbon.user.core.common.UserStoreDeploymentManager} - Realm configuration of tenant:-1234 modified with D:\wso2_identity_server\wso2is-5.10.0\wso2is-5.10.0\bin..\repository\deployment\server\userstores\testing.xml
INFO {org.wso2.carbon.user.core.common.UserStoreDeploymentManager} - User store: contata.in of tenant:-1234 is removed from realm chain.
INFO {org.wso2.carbon.user.core.common.AbstractUserStoreManager} - User store: testing of tenant:-1234 is removed from user store chain.
ERROR {org.wso2.carbon.user.core.common.AbstractUserStoreManager} - Cannot create org.wso2.carbon.user.core.ldap.ActiveDirectoryUserStoreManager java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.createSecondaryUserStoreManager(AbstractUserStoreManager.java:8726)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.addSecondaryUserStoreManager(AbstractUserStoreManager.java:8798)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager$2.run(AbstractUserStoreManager.java:194)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.callSecure(AbstractUserStoreManager.java:191)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.addSecondaryUserStoreManager(AbstractUserStoreManager.java:8784)
at org.wso2.carbon.user.core.common.UserStoreDeploymentManager.deploy(UserStoreDeploymentManager.java:71)
at org.wso2.carbon.identity.user.store.configuration.deployer.UserStoreConfigurationDeployer.deploy(UserStoreConfigurationDeployer.java:169)
at org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
at org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:807)
at org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:144)
at org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:59)
at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:67)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(CarbonDeploymentSchedulerTask.java:93)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at org.wso2.carbon.user.core.ldap.LDAPConnectionContext.(LDAPConnectionContext.java:122)
at org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager.(ReadOnlyLDAPUserStoreManager.java:227)
at org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager.(ReadWriteLDAPUserStoreManager.java:120)
at org.wso2.carbon.user.core.ldap.ActiveDirectoryUserStoreManager.(ActiveDirectoryUserStoreManager.java:115)
... 33 more
It seems to be a connectivity problem. Maybe the ldap server is not visible, or port is incorrect. Check it with telnet. If you could detail a little bit more the error and the is version you are using.
I tried to run wso2mb-3.1.0/samples/JmsMessageListenerClient but I get below error on server start. Am I missing some config?
org.wso2.andes.AMQConnectionFailureException: Unable to Connect
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:475)
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:328)
at org.wso2.andes.client.AMQConnectionFactory.createTopicConnection(AMQConnectionFactory.java:443)
at com.wickes.common.jms.service.wso2mq.Wso2MessageListener.registerSubscribers(Wso2MessageListener.java:41)
at com.wickes.product.web.Controller.get(Controller.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:132)
at org.wso2.msf4j.internal.MSF4JMessageProcessor.dispatchMethod(MSF4JMessageProcessor.java:130)
at org.wso2.msf4j.internal.MSF4JMessageProcessor.receive(MSF4JMessageProcessor.java:72)
at org.wso2.carbon.transport.http.netty.listener.WorkerPoolDispatchingSourceHandler.lambda$publishToWorkerPool$12(WorkerPoolDispatchingSourceHandler.java:125)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
javax.jms.JMSException: Error creating connection: Unable to Connect
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:338)
at org.wso2.andes.client.AMQConnectionFactory.createTopicConnection(AMQConnectionFactory.java:443)
at com.wickes.common.jms.service.wso2mq.Wso2MessageListener.registerSubscribers(Wso2MessageListener.java:41)
at com.wickes.product.web.Controller.get(Controller.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:132)
at org.wso2.msf4j.internal.MSF4JMessageProcessor.dispatchMethod(MSF4JMessageProcessor.java:130)
at org.wso2.msf4j.internal.MSF4JMessageProcessor.receive(MSF4JMessageProcessor.java:72)
at org.wso2.carbon.transport.http.netty.listener.WorkerPoolDispatchingSourceHandler.lambda$publishToWorkerPool$12(WorkerPoolDispatchingSourceHandler.java:125)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.andes.AMQConnectionFailureException: Unable to Connect
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:475)
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:328)
... 14 more
2017-02-21 16:03 ERROR AMQStateManager:162 - Notifying Waiters([org.wso2.andes.client.state.StateWaiter#520acd8f]) for error:Exception thrown against AMQConnection:
Host: localhost
Port: 5672
Virtual Host: carbon
Client ID: carbon
Active session count: 0: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted.
2017-02-21 16:03 ERROR AMQProtocolHandler:485 - Exception processing frame
org.wso2.andes.framing.AMQProtocolClassException: Protocol class 1 was expected; received 3
at org.wso2.andes.framing.ProtocolInitiation.checkVersion(ProtocolInitiation.java:190)
at org.wso2.andes.client.protocol.AMQProtocolHandler$2.run(AMQProtocolHandler.java:476)
at org.wso2.andes.pool.Job.processAll(Job.java:109)
at org.wso2.andes.pool.Job.run(Job.java:157)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-02-21 16:03 ERROR AMQConnection:1307 - Throwable Received but no listener set: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted.
2017-02-21 16:03 ERROR AMQConnection:1307 - Throwable Received but no listener set: org.wso2.andes.framing.AMQProtocolClassException: Protocol class 1 was expected; received 3
Thanks,
Manish
At the first access to the Main - Entitlement - Policy section I get the following error:
TID[-1234] [IS] [2015-05-07 11:11:03,046] ERROR
{org.apache.axis2.rpc.receivers.RPCMessageReceiver} - Exception
occurred while trying to invoke service method getAllPolicies
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
java.lang.reflect.Method.invoke(Method.java:611)
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
org.apache.axis2.transport.local.LocalTransportReceiver.processMessage(LocalTransportReceiver.java:169)
org.apache.axis2.transport.local.LocalTransportReceiver.processMessage(LocalTransportReceiver.java:82)
org.wso2.carbon.core.transports.local.CarbonLocalTransportSender.finalizeSendWithToAddress(CarbonLocalTransportSender.java:45)
org.apache.axis2.transport.local.LocalTransportSender.invoke(LocalTransportSender.java:77)
org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
Similar to report: see at this forum
Thanks in advance for assistance how fix this error.