I am running a springboot app in docker using amazon ecs. The problem is the springboot application stops at connecting to datasource.
2022-04-13 01:35:25.473 INFO 1 --- [ main] com.query.QueryServiceApplication : Starting QueryServiceApplication v0.0.1-SNAPSHOT using Java 11.0.14.1 on 0ab5b5efda39 with PID 1 (/query-service.jar started by root in /)
2022-04-13 01:35:25.487 INFO 1 --- [ main] com.query.QueryServiceApplication : No active profile set, falling back to 1 default profile: "default"
2022-04-13 01:35:28.185 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2022-04-13 01:35:28.231 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 16 ms. Found 0 JPA repository interfaces.
2022-04-13 01:35:30.933 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-04-13 01:35:30.979 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-04-13 01:35:30.980 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.58]
2022-04-13 01:35:31.164 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-04-13 01:35:31.165 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 5477 ms
2022-04-13 01:35:32.967 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
After sometime i get the exception
java.sql.SQLNonTransientConnectionException: Got timeout reading communication packets
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) ~[mysql-connector-java-8.0.28.jar!/:8.0.28]
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.28.jar!/:8.0.28]
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) ~[mysql-connector-java-8.0.28.jar!/:8.0.28]
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:449) ~[mysql-connector-java-8.0.28.jar!/:8.0.28]
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) ~[mysql-connector-java-8.0.28.jar!/:8.0.28]
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) ~[mysql-connector-java-8.0.28.jar!/:8.0.28]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-4.0.3.jar!/:na]
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:159) ~[spring-jdbc-5.3.16.jar!/:5.3.16]
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:117) ~[spring-jdbc-5.3.16.jar!/:5.3.16]
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-5.3.16.jar!/:5.3.16]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:330) ~[spring-jdbc-5.3.16.jar!/:5.3.16]
at org.springframework.boot.jdbc.EmbeddedDatabaseConnection.isEmbedded(EmbeddedDatabaseConnection.java:162) ~[spring-boot-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateDefaultDdlAutoProvider.getDefaultDdlAuto(HibernateDefaultDdlAutoProvider.java:42) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration.lambda$getVendorProperties$1(HibernateJpaConfiguration.java:130) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateSettings.getDdlAuto(HibernateSettings.java:41) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties.determineDdlAuto(HibernateProperties.java:143) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties.getAdditionalProperties(HibernateProperties.java:103) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties.determineHibernateProperties(HibernateProperties.java:95) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration.getVendorProperties(HibernateJpaConfiguration.java:132) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at org.springframework.boot.autoconfigure.orm.jpa.JpaBaseConfiguration.entityManagerFactory(JpaBaseConfiguration.java:132) ~[spring-boot-autoconfigure-2.6.4.jar!/:2.6.4]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.16.jar!/:5.3.16]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1154) ~[spring-context-5.3.16.jar!/:5.3.16]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:908) ~[spring-context-5.3.16.jar!/:5.3.16]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.16.jar!/:5.3.16]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.4.jar!/:2.6.4]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) ~[spring-boot-2.6.4.jar!/:2.6.4]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) ~[spring-boot-2.6.4.jar!/:2.6.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-2.6.4.jar!/:2.6.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312) ~[spring-boot-2.6.4.jar!/:2.6.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) ~[spring-boot-2.6.4.jar!/:2.6.4]
at com.query.QueryServiceApplication.main(QueryServiceApplication.java:10) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[query-service.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[query-service.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[query-service.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[query-service.jar:0.0.1-SNAPSHOT]
I check for security groups as instructed in post link. I made the RDS publicly accessible and allowed access from all IP's and all protocols in the security group. Both my RDS and ECS are in same VPC and same security group. I am not really sure what else to do here ? Will there be an issue connecting since its a docker container ?
I made the RDS publicly accessible
That could actually be a problem. The ECS task may be resolving the RDS endpoint to an Internet IP address, and then trying to connect to the database over the Internet, instead of keeping the traffic inside the VPC. If the ECS task doesn't have a public IP address, or it isn't running in a subnet with a route to a NAT gateway, then it wouldn't be able to connect.
Both my RDS and ECS are in same VPC and same security group.
Being in the same security group does not confer any connectivity by default. This is actually bad practice because it doesn't allow you to restrict traffic on specific ports, since you are mixing different services that use different ports in the same security group. Since you allowed access from all IPs and protocols this shouldn't be the cause of your current issue however.
Some steps to debug this would be to use ECS Exec, or spin up an EC2 instance in the same VPC subnet (with the same security group), and see if the RDS endpoint resolves to an internal or external IP, and see if you can connect with the MySQL command line.
Related
I have been trying to replicate this aws function example to deploy a lambda with spring cloud function and graalvm.
The function works locally and starts up without a problem.
❯ ./demo
2021-05-10 15:56:17.964 INFO 10764 --- [ main] o.s.nativex.NativeListener
: This application is bootstrapped with code generated with Spring AOT
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.4.5)
2021-05-10 16:42:42.607 INFO 12792 --- [ main] com.example.demo.DemoApplication : Starting DemoApplication using Java 11.0.11 on loyalty-C02F70R8MD6R with PID 12792 (/Users/user/Workspace/learning/demo/target/demo started by user in /Users/user/Workspace/learning/demo/target)
2021-05-10 16:42:42.607 INFO 12792 --- [ main] com.example.demo.DemoApplication : No active profile set, falling back to default profiles: default
2021-05-10 16:42:42.637 INFO 12792 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
May 10, 2021 4:42:42 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8080"]
May 10, 2021 4:42:42 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
May 10, 2021 4:42:42 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet engine: [Apache Tomcat/9.0.45]
May 10, 2021 4:42:42 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring embedded WebApplicationContext
2021-05-10 16:42:42.640 INFO 12792 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 32 ms
2021-05-10 16:42:42.650 INFO 12792 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
May 10, 2021 4:42:42 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-8080"]
2021-05-10 16:42:42.658 INFO 12792 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2021-05-10 16:42:42.658 INFO 12792 --- [ main] com.example.demo.DemoApplication : Started DemoApplication in 0.071 seconds (JVM running for 0.072)
However, when I deployed it to aws it failed with the following exception
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cloud.function.adapter.aws.CustomRuntimeEventLoop': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.function.adapter.aws.CustomRuntimeEventLoop]: No default constructor found; nested exception is java.lang.NoSuchMethodException: org.springframework.cloud.function.adapter.aws.CustomRuntimeEventLoop.<init>()
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1316) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1214) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[na:na]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[na:na]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[na:na]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[na:na]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[na:na]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[na:na]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:782) ~[demo:na]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:774) ~[demo:na]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) ~[demo:na]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:339) ~[demo:na]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1340) ~[demo:na]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) ~[demo:na]
at com.example.demo.DemoApplication.main(DemoApplication.java:10) ~[demo:na]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.function.adapter.aws.CustomRuntimeEventLoop]: No default constructor found; nested exception is java.lang.NoSuchMethodException: org.springframework.cloud.function.adapter.aws.CustomRuntimeEventLoop.<init>()
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:83) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1308) ~[na:na]
... 18 common frames omitted
Caused by: java.lang.NoSuchMethodException: org.springframework.cloud.function.adapter.aws.CustomRuntimeEventLoop.<init>()
at java.lang.Class.getConstructor0(DynamicHub.java:3349) ~[na:na]
at java.lang.Class.getDeclaredConstructor(DynamicHub.java:2553) ~[na:na]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:78) ~[na:na]
... 19 common frames omitted
I inspected the CustomRuntimeEventLoop and found that class is annotate with
#Configuration
#ConditionalOnProperty("AWS_LAMBDA_RUNTIME_API")
Which means it, only, gets initialised in the AWS environment. And this is why the function did not fail locally. I tried to initialised the class at build time by adding the following snippet to my spring boot application
#NativeHint(initialization = {#InitializationHint(types = {CustomRuntimeEventLoop.class}, initTime = BUILD)})
But this still did not affect the build or runtime behaviour.
Any idea how can I initialise this in the build time?
Have you seem this example - https://github.com/spring-projects-experimental/spring-native/tree/main/samples/cloud-function-aws?
Also, there were issues with that new CustomRuntimeEventLoop that have already been addressed, so consider using the latest snapshot.
**
[2020-09-16 13:25:32,090] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://10.1.31.61:9711. org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: Cannot borrow client for ssl://10.1.31.61:9711. at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:145) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) [org.wso2.carbon.databridge.agent_5.2.24.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_261] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_261] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261] Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: Error while trying to login to data receiver :/10.1.31.61:9711 at org.wso2.carbon.databridge.agent.endpoint.binary.BinaryDataEndpoint.login(BinaryDataEndpoint.java:50) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:139) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] ... 6 more Caused by: org.wso2.carbon.databridge.commons.exception.AuthenticationException: java.lang.IllegalStateException: Cannot find suitable mapped attribute for local claim http://wso2.org/claims/userid at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_261] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_261] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_261] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_261] at org.wso2.carbon.databridge.agent.endpoint.binary.BinaryEventSender.processResponse(BinaryEventSender.java:163) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] at org.wso2.carbon.databridge.agent.endpoint.binary.BinaryDataEndpoint.login(BinaryDataEndpoint.java:44) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:139) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] ... 6 more [2020-09-16 13:25:32,169] INFO - QpidServiceComponent Activating Andes Message Broker Engine... [Broker] BRK-1001 : Startup : Version: 0.11 Build: 90784:90849 [Broker] MNG-1001 : Startup [Broker] MNG-1004 : Ready : Using the platform JMX Agent [Broker] BRK-1002 : Starting : Listening on TCP port 5672 [2020-09-16 13:25:32,878] INFO - listening [Broker] BRK-1002 : Starting : Listening on TCP port 5672 [Broker] BRK-1002 : Starting : Listening on TCP/SSL port 8672 [2020-09-16 13:25:32,924] INFO - listening [Broker] BRK-1002 : Starting : Listening on TCP/SSL port 8672 [Broker] BRK-1004 : Qpid Broker Ready Caused by: org.wso2.carbon.databridge.commons.exception.AuthenticationException: java.lang.IllegalStateException: Cannot find suitable mapped attribute for local claim http://wso2.org/claims/userid at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_261] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_261] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_261] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_261] at org.wso2.carbon.databridge.agent.endpoint.binary.BinaryEventSender.processResponse(BinaryEventSender.java:163) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] at org.wso2.carbon.databridge.agent.endpoint.binary.BinaryDataEndpoint.login(BinaryDataEndpoint.java:44) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:139) ~[org.wso2.carbon.databridge.agent_5.2.24.jar:?]
**
Please check the following generic configs:
If it is distributed setup check TM is up and running
Check TM's port is correct and 7611 and 7711 is open
CHeck throttling related credentials are correct according to TM
Based on the logs, there are changes to the claim attribute mapping. Looks like claim http://wso2.org/claims/userid is mapped to non-existing attributes in the user store. Please check and set to correct available attribute,
This question about "aws sub-generator" (not "aws-containers sub-generator").
Question 1:
Is this generator supposed to update "application-prod.yml"? I can see the information I provided while running the sub-generator including the database name in ".yo-rc.json". But database user-name and password are neither there nor in the "application-prod.yml".
Question 2:
I generated a simple JHipster project with some entities. It was working fine locally on my PC. I used the "aws sub-generator" and everything went well. It created the Elastic Beanstalk application, an EC2 instance, the S3 bucket, the RDS database, the security groups...Here is the event log of the Elastic Beanstalk:
Successfully launched environment: alirezatodoapplicationapril-env
Environment health has been set to GREEN
Created CloudWatch alarm named: awseb-e-i2vaqsmdbh-stack-AWSEBCloudwatchAlarmHigh-18WDVCSH132VM
Created CloudWatch alarm named: awseb-e-i2vaqsmdbh-stack-AWSEBCloudwatchAlarmLow-EA2GT2HFN77U
Created Auto Scaling group policy named: arn:aws:autoscaling:us-east-1:827968211260:scalingPolicy:5cf0c635-40b7-4008-9ebe-57823a3f9b3c:autoScalingGroupName/awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingGroup-1SLCXWGKICYGM:policyName/awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingScaleDownPolicy-JHTI8C9VQTF1
Created Auto Scaling group policy named: arn:aws:autoscaling:us-east-1:827968211260:scalingPolicy:5e6f86c4-79f7-4504-a6d2-8a8a9c638ef5:autoScalingGroupName/awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingGroup-1SLCXWGKICYGM:policyName/awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingScaleUpPolicy-E1W03HEXTE6R
Waiting for EC2 instances to launch. This may take a few minutes.
Created Auto Scaling group named: awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingGroup-1SLCXWGKICYGM
Added EC2 instance 'i-074488994cedde8c7' to Auto Scaling Group 'awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingGroup-1SLCXWGKICYGM'.
Adding instance 'i-074488994cedde8c7' to your environment.
Created Auto Scaling launch configuration named: awseb-e-i2vaqsmdbh-stack-AWSEBAutoScalingLaunchConfiguration-1F3VDL1LKSD1B
Created load balancer named: awseb-e-i-AWSEBLoa-18MWGKA8O6FHP
Created security group named: awseb-e-i2vaqsmdbh-stack-AWSEBSecurityGroup-1BS14HUVJLFUD
Created security group named: sg-874777cd
Using elasticbeanstalk-us-east-1-827968211260 as Amazon S3 storage bucket for environment data.
createEnvironment is starting.
But, now when I use the link of the Beanstalk application it gives me a 404 error. What is missed? Here is the log of application startup (catalina.out):
██╗ ██╗ ██╗ ████████╗ ███████╗ ██████╗ ████████╗ ████████╗ ███████╗
██║ ██║ ██║ ╚══██╔══╝ ██╔═══██╗ ██╔════╝ ╚══██╔══╝ ██╔═════╝ ██╔═══██╗
██║ ████████║ ██║ ███████╔╝ ╚█████╗ ██║ ██████╗ ███████╔╝
██╗ ██║ ██╔═══██║ ██║ ██╔════╝ ╚═══██╗ ██║ ██╔═══╝ ██╔══██║
╚██████╔╝ ██║ ██║ ████████╗ ██║ ██████╔╝ ██║ ████████╗ ██║ ╚██╗
╚═════╝ ╚═╝ ╚═╝ ╚═══════╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══════╝ ╚═╝ ╚═╝
:: JHipster 🤓 :: Running Spring Boot 2.0.3.RELEASE ::
:: https://www.jhipster.tech ::
2018-07-31 20:18:22.679 ERROR 3398 --- [ost-startStop-1] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:136)
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:94)
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:633)
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:373)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:325)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:155)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:135)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:87)
at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:172)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5204)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1141)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8761/config/ToDo/prod/master": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:732)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:218)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:96)
... 20 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:723)
... 24 common frames omitted
31-Jul-2018 20:18:22.688 SEVERE [localhost-startStop-1] org.apache.catalina.core.ContainerBase.addChildInternal ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1141)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:136)
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:94)
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:633)
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:373)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:325)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:155)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:135)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:87)
at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:172)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5204)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 10 more
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8761/config/ToDo/prod/master": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:732)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:218)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:96)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:723)
... 24 more
31-Jul-2018 20:18:22.690 SEVERE [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Error deploying web application directory [/var/lib/tomcat8/webapps/ROOT]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[]]
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1141)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
31-Jul-2018 20:18:22.691 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat8/webapps/ROOT] has finished in [16,376] ms
31-Jul-2018 20:18:22.695 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
31-Jul-2018 20:18:22.727 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
31-Jul-2018 20:18:22.742 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 16499 ms
The JHipster AWS ElasticBeanstalk option only works with monoliths and won't deploy the JHipster Registry if that option is chosen when generating the project.
If you want to disable the JHipster Registry feature, you can set two config variables in your project. In bootstrap-prod.yml, set spring.cloud.config.enabled: false and disable Eureka in application-prod.yml with eureka.client.enabled: false.
In regards to the database connection settings, those are set as environment variables when creating the ElasticBeanstalk environment which Spring Boot uses instead of the hardcoded config in application-prod.yml
I am using AWS ECS to deploy Eureka in my Cluster to zones inside us-east-1 region. ECS dynamically deploys to any region and I cannot predetermine the IP or domain the EC2 instance will be, hence I use DNS.
I am using DNS as illustrated here https://github.com/Netflix/eureka/wiki/Deploying-Eureka-Servers-in-EC2. Below are my configurations:
eureka:
instance:
healthCheckUrlPath: /manage/health
client:
region: us-east-1
availabilityZones:
us-east-1: us-east-1a,us-east-1c
eurekaServerPort: 8761
useDnsForFetchingServiceUrls: true
eurekaServerDNSName: eureka.mydomain.com
eurekaServerURLContext: eureka
registerWithEureka: true
fetchRegistry: true
cloud:
aws:
credentials:
accessKey: AWS_KEY
secretKey: AWS_KEY_SECRET
region:
static: us-east-1
The user with AWS_KEY has this policy attached:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:AllocateAddress",
"ec2:AssociateAddress",
"ec2:DescribeAddresses",
"ec2:DisassociateAddress"
],
"Sid": "Stmt1375723773000",
"Resource": [
"*"
],
"Effect": "Allow"
}
]
}
and configured the EurekaInstanceConfigBean configured as:
#Bean
#Profile("!default")
public EurekaInstanceConfigBean eurekaInstanceConfig(InetUtils inetUtils) {
EurekaInstanceConfigBean config = new EurekaInstanceConfigBean(inetUtils);
AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
info.getMetadata().put(AmazonInfo.MetaDataKey.publicHostname.getName(), info.get(AmazonInfo.MetaDataKey.publicIpv4));
config.setHostname(info.get(AmazonInfo.MetaDataKey.publicHostname));
config.setIpAddress(info.get(AmazonInfo.MetaDataKey.publicIpv4));
config.setNonSecurePort(port);
config.setDataCenterInfo(info);
return config;
}
GOOD THING: Eureka recognise my Route 53 configured eureka.mydomain.com DNS EIPs and it tries to bind, the (available and unassigned) EIP in zone us-east-1c, to the instance where my eureka server is deployed
PROBLEM: I get the following logs and Unauthorized error as below when booting my app:
...................................
.................................
2017-04-10 16:07:42.141 DEBUG 5 --- [ main] c.n.d.s.r.a.DnsTxtRecordClusterResolver : Resolved txt.us-east-1.eureka.mydomain.com to [AwsEndpoint{ serviceUrl=
'http://ec2-34.200.47.82.compute-1.amazonaws.com:8761/eureka', region='us-east-1', zone='us-east-1c'}]
2017-04-10 16:07:42.141 DEBUG 5 --- [ main] c.n.d.s.r.a.ZoneAffinityClusterResolver : Local zone=us-east-1c; resolved to: [AwsEndpoint{ serviceUrl='http://ec2-3
4.200.47.82.compute-1.amazonaws.com:8761/eureka', region='us-east-1', zone='us-east-1c'}]
2017-04-10 16:07:42.204 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Disable delta property : false
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Single vip registry refresh property : null
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Force full registry fetch : false
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Application is null : false
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Registered Applications size is zero : true
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Application version is -1: true
2017-04-10 16:07:42.211 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
2017-04-10 16:07:42.213 DEBUG 5 --- [ main] c.n.d.s.t.d.SessionedEurekaHttpClient : Ending a session and starting anew
2017-04-10 16:07:42.222 DEBUG 5 --- [ main] n.d.s.t.j.AbstractJerseyEurekaHttpClient : Created client for url: http://ec2-34.200.47.82.compute-1.amazonaws.com:87
61/eureka
2017-04-10 16:07:42.313 DEBUG 5 --- [ main] c.n.d.shared.MonitoredConnectionManager : Get connection: {}->http://ec2-34.200.47.82.compute-1.amazonaws.com:8761,
timeout = 5000
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : [{}->http://ec2-34.200.47.82.compute-1.amazonaws.com:8761] total kept aliv
e: 0, total issued: 0, total allocated: 0 out of 200
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : No free connections [{}->http://ec2-34.200.47.82.compute-1.amazonaws.com:8
761][null]
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Available capacity: 50 out of 50 [{}->http://ec2-34.200.47.82.compute-1.am
azonaws.com:8761][null]
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Creating new connection [{}->http://ec2-34.200.47.82.compute-1.amazonaws.c
om:8761]
2017-04-10 16:07:42.330 DEBUG 5 --- [ main] c.n.d.shared.MonitoredConnectionManager : Released connection is not reusable.
2017-04-10 16:07:42.331 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Releasing connection [{}->http://ec2-34.200.47.82.compute-1.amazonaws.com:
8761][null]
2017-04-10 16:07:42.331 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Notifying no-one, there are no waiting threads
2017-04-10 16:07:42.331 DEBUG 5 --- [ main] n.d.s.t.j.AbstractJerseyEurekaHttpClient : Jersey HTTP GET http://ec2-34.200.47.82.compute-1.amazonaws.com:8761/eurek
a/apps/?; statusCode=N/A
2017-04-10 16:07:42.345 ERROR 5 --- [ main] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution
....................
....................
2017-04-10 16:07:49.455 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : This client will talk to the following serviceUrls in order : [http://ec2-
34.206.31.211.compute-1.amazonaws.com:8761/eureka/]
2017-04-10 16:07:49.455 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The region url to be looked up is txt.us-east-1.eureka.mydomain.com :
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The zoneName mapped to region us-east-1 is us-east-1c
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : Checking if the instance zone us-east-1c is the same as the zone from DNS
us-east-1c
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The zone index from the list [us-east-1c] that matches the instance zone u
s-east-1c is 0
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The zone url to be looked up is txt.us-east-1c.eureka.mydomain.com :
2017-04-10 16:07:49.457 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The eureka url for the dns name txt.us-east-1c.eureka.mydomain.com is e
c2-34.200.47.82.compute-1.amazonaws.com
2017-04-10 16:07:49.457 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The EC2 url is http://ec2-34.200.47.82.compute-1.amazonaws.com:8761/eureka
/
2017-04-10 16:07:49.457 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : This client will talk to the following serviceUrls in order : [http://ec2-
34.200.47.82.compute-1.amazonaws.com:8761/eureka/]
**2017-04-10 16:07:49.527 ERROR 5 --- [ Thread-11] com.netflix.eureka.aws.EIPManager : Failed to bind elastic IP: 34.200.47.82 to i-0bc1018ccdcc69148
com.amazonaws.AmazonServiceException: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request I
D: f9b2dec4-6d79-4da2-bbac-061416bde000)**
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:12003) ~[aws-java-sdk-ec2-1.11.18.jar!/:na]
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11973) ~[aws-java-sdk-ec2-1.11.18.jar!/:na]
at com.amazonaws.services.ec2.AmazonEC2Client.describeAddresses(AmazonEC2Client.java:4716) ~[aws-java-sdk-ec2-1.11.18.jar!/:na]
at com.netflix.eureka.aws.EIPManager.bindEIP(EIPManager.java:202) [eureka-core-1.4.12.jar!/:1.4.12]
at com.netflix.eureka.aws.EIPManager.handleEIPBinding(EIPManager.java:136) [eureka-core-1.4.12.jar!/:1.4.12]
at com.netflix.eureka.aws.EIPManager.start(EIPManager.java:105) [eureka-core-1.4.12.jar!/:1.4.12]
at com.netflix.eureka.aws.AwsBinderDelegate.start(AwsBinderDelegate.java:42) [eureka-core-1.4.12.jar!/:1.4.12]
at org.springframework.cloud.netflix.eureka.server.EurekaServerBootstrap.initEurekaServerContext(EurekaServerBootstrap.java:145) [spring-cloud-netflix-eureka-serve
r-1.2.6.RELEASE.jar!/:1.2.6.RELEASE]
at org.springframework.cloud.netflix.eureka.server.EurekaServerBootstrap.contextInitialized(EurekaServerBootstrap.java:81) [spring-cloud-netflix-eureka-server-1.2.
6.RELEASE.jar!/:1.2.6.RELEASE]
at org.springframework.cloud.netflix.eureka.server.EurekaServerInitializerConfiguration$1.run(EurekaServerInitializerConfiguration.java:70) [spring-cloud-netflix-e
ureka-server-1.2.6.RELEASE.jar!/:1.2.6.RELEASE]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
2017-04-10 16:07:49.527 INFO 5 --- [ Thread-11] com.netflix.eureka.aws.EIPManager : No EIP is free to be associated with this instance. Candidate EIPs are: [3
4.200.47.82]
......................................
........................................
........................................
QUESTION: I have attached the policy to allow Eureka to bind the Elastic IP to the instance where it is deployed but WHY am I getting a You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation and how can I fix this? As it stands, I have spend more than a day Googling and still the same error :(
I tried the netflix way of configuring eureka like below but to no avail :(:
eureka:
awsAccessId: AWS_KEY
awsSecretKey:AWS_KEY_SECRET
asgName: EIPAccessPolicyGroup
So I finally got a solution and had help from #DirkLachowski and #spencergibb on this post. Thanks a lot guys. So I only had to change this:
eureka:
awsAccessId: AWS_KEY
awsSecretKey:AWS_KEY_SECRET
asgName: EIPAccessPolicyGroup
To this:
eureka:
server:
aWSAccessId: AWS_KEY
aWSSecretKey: AWS_SECRET_KEY
asgName: EC2ContainerService_AUTO_SCALING_GROUP_CREATED_BY_ECS_FOR_MY_CLUSTER
So each eureka server bind an unused/free EIP that I put on my TXT DNS records to the EC2 instance where my eureka server is running :)
I am trying to setup 3 Kafka brokers on ubuntu EC2 machines. But I am getting ConnectException while starting zookeeper. All the ports in the security group of my ec2 intsances are already open.
Below is the stack trace:
[2016-03-03 07:37:12,040] ERROR Exception while listening (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.BindException: Cannot assign requested address
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:330)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:507)
...
...
...
[2016-03-03 07:23:46,093] WARN Cannot open channel to 2 at election address /52.36.XXX.181:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
Below is the configuration of:
zookeeper.properties:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=52.37.XX.70:2888:3888
server.2=52.36.XXX.181:2888:3888
server.3=52.37.XX.42:2888:3888
initLimit=5
syncLimit=2
server.properties:
broker.id=1
port=9092
host.name=52.37.XX.70
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=52.37.XX.70:2181,52.36.XXX.181:2181,52.37.XX.42:2181
zookeeper.connection.timeout.ms=6000
I have added server's public IP in /etc/hosts of the instances. My modified /etc/hosts is as:
127.0.0.1 localhost localhost.localdomain ip-10-131-X-217
127.0.0.1 localhost localhost.localdomain 52.37.XX.70
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
The unique myid entries in /tmp/zookeper/myid are also entered correctly.
I have followed all the steps mentioned in: How to create a MultiNode - MultiBroker Cluster for Kafka on AWS
The issue was because I was using the Public IP of the server. Instead of that, use of Public DNS of ec2 instances fixed the issue.