WSO2 User Store Database Initialization Configuration - wso2

Question

I am running WSO2 Identity Server 5.7.0. I am deploying a new instance of it in a different environment with a different set of backing databases. In the process of the WSO2 startup, it appears the User Manager Core bundle is failing to start.
The error is:
[2019-07-10 18:30:16,596] ERROR {org.wso2.carbon.user.core.util.DatabaseUtil} - Database Error - The connection attempt failed.
org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:454)
at org.postgresql.Driver.connect(Driver.java:256)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:730)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:664)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:200)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at org.wso2.carbon.user.core.claim.dao.ClaimDAO.getDialectCount(ClaimDAO.java:158)
at org.wso2.carbon.user.core.common.DefaultRealm.populateProfileAndClaimMaps(DefaultRealm.java:442)
at org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:124)
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:264)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:102)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:115)
at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:72)
at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390)
at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:70)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
... 33 more
[2019-07-10 18:30:16,599] ERROR {org.wso2.carbon.user.core.internal.Activator} - Cannot start User Manager Core bundle
org.wso2.carbon.user.core.UserStoreException: Cannot initialize the realm.
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:274)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:102)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:115)
at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:72)
at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390)
at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
Caused by: org.wso2.carbon.user.core.UserStoreException: Database Error - The connection attempt failed.
at org.wso2.carbon.user.core.claim.dao.ClaimDAO.getDialectCount(ClaimDAO.java:168)
at org.wso2.carbon.user.core.common.DefaultRealm.populateProfileAndClaimMaps(DefaultRealm.java:442)
at org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:124)
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:264)
... 19 more
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:454)
at org.postgresql.Driver.connect(Driver.java:256)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:730)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:664)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:200)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at org.wso2.carbon.user.core.claim.dao.ClaimDAO.getDialectCount(ClaimDAO.java:158)
... 22 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:70)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
... 33 more
It appears that the connections to the databases are initializing successfully before this error appears, so I'm not sure what the issue is.
Here are the logs regarding the datasources:
[2019-07-10 17:48:59,688] DEBUG {org.wso2.carbon.ndatasource.core.internal.DataSourceServiceComponent} - Initializing system data sources...
[2019-07-10 17:48:59,834] DEBUG {org.wso2.carbon.ndatasource.core.DataSourceRepository} - Adding data source: WSO2_CARBON_DB
[2019-07-10 17:48:59,835] DEBUG {org.wso2.carbon.ndatasource.core.DataSourceRepository} - Registering data source: WSO2_CARBON_DB
[2019-07-10 17:49:00,089] DEBUG {org.wso2.carbon.context.internal.CarbonContextDataHolder} - Loading JNDI Initial Context Factory: org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory
[2019-07-10 17:49:00,106] DEBUG {org.wso2.carbon.context.internal.CarbonContextDataHolder} - Unload Tenant Task: org.wso2.carbon.context.internal.CarbonContextDataHolder$CarbonInitialJNDIContext$ContextCleanup
Task was registered.
[2019-07-10 17:49:00,190] DEBUG {org.wso2.carbon.ndatasource.core.DataSourceRepository} - Adding data source: WSO2_METRICS_DB
[2019-07-10 17:49:00,191] DEBUG {org.wso2.carbon.ndatasource.core.DataSourceRepository} - Registering data source: WSO2_METRICS_DB
[2019-07-10 17:49:00,255] DEBUG {org.wso2.carbon.context.internal.CarbonContextDataHolder} - Loading JNDI Initial Context Factory: org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory
[2019-07-10 17:49:00,323] DEBUG {org.wso2.carbon.ndatasource.core.DataSourceRepository} - Adding data source: BPS_DS
[2019-07-10 17:49:00,324] DEBUG {org.wso2.carbon.ndatasource.core.DataSourceRepository} - Registering data source: BPS_DS
[2019-07-10 17:49:00,385] DEBUG {org.wso2.carbon.context.internal.CarbonContextDataHolder} - Loading JNDI Initial Context Factory: org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory
[2019-07-10 17:49:00,391] DEBUG {org.wso2.carbon.ndatasource.core.internal.DataSourceServiceComponent} - System data sources successfully initialized
[2019-07-10 17:49:00,392] DEBUG {org.wso2.carbon.ndatasource.core.internal.DataSourceServiceComponent} - ServerConfigurationService acquired
[2019-07-10 17:49:00,393] DEBUG {org.wso2.carbon.ndatasource.core.internal.DataSourceServiceComponent} - DataSourceServiceComponent activated
My question is which configurations are relevant to the step of the process that is failing. Is this an issue with the datasources connection, the LDAP connection, or something else?
Thanks for your help.
Edit:
master-datasources.xml:
<?xml version="1.0" encoding="UTF-8"?>
<datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">
<providers>
<provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
</providers>
<datasources>
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>${master.datasources.url}</url>
<username>${master.datasources.username}</username>
<password>password</password>
<driverClassName>org.postgresql.Driver</driverClassName>
<maxActive>80</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
<testOnBorrow>true</testOnBorrow>
<defaultAutoCommit>true</defaultAutoCommit>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
</datasources>
</datasources-configuration>
user-mgt.xml:
<UserManager xmlns:svns="http://org.wso2.securevault/configuration">
<Realm>
<Configuration>
<AddAdmin>false</AddAdmin>
<AdminRole>admin</AdminRole>
<AdminUser>
<UserName>${user.mgt.admin.username}</UserName>
<Password>placeholder</Password>
</AdminUser>
<!-- By default users in this role sees the registry root-->
<EveryOneRoleName>everyone</EveryOneRoleName>
<Property name="isCascadeDeleteEnabled">true</Property>
<Property name="initializeNewClaimManager">false</Property>
<Property name="dataSource">jdbc/WSO2CarbonDB</Property>
</Configuration>
<UserStoreManager class="org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager">
<Property name="TenantManager">org.wso2.carbon.user.core.tenant.CommonHybridLDAPTenantManager</Property>
<Property name="ConnectionURL">${user.mgt.connection.url}</Property>
<Property name="ConnectionName">${user.mgt.connection.name}</Property>
<Property name="ConnectionPassword">password</Property>
<Property name="AnonymousBind">false</Property>
<Property name="UserSearchBase">${user.mgt.search.base}</Property>
<Property name="UserEntryObjectClass">customer</Property>
<Property name="UserNameAttribute">cn</Property>
<Property name="UserNameSearchFilter">(&(objectClass=top)(cn=?))</Property>
<Property name="UserNameListFilter">(objectClass=top)</Property>
<Property name="UserDNPattern"/>
<Property name="DisplayNameAttribute"/>
<Property name="ReadGroups">true</Property>
<Property name="WriteGroups">true</Property>
<Property name="GroupSearchBase">ou=Groups,dc=cse-renaissance,dc=equifax,dc=com</Property>
<Property name="GroupEntryObjectClass">groupOfNames</Property>
<Property name="GroupNameAttribute">cn</Property>
<Property name="GroupNameSearchFilter">(&(objectClass=groupOfNames)(cn=?))</Property>
<Property name="GroupNameListFilter">(objectClass=groupOfNames)</Property>
<Property name="RoleDNPattern">cn={0},ou=Groups,dc=cse-renaissance,dc=equifax,dc=com</Property>
<Property name="MembershipAttribute">member</Property>
<Property name="BackLinksEnabled">false</Property>
<Property name="UsernameJavaRegEx">[a-zA-Z0-9._\-|//]{3,30}$</Property>
<Property name="UsernameJavaScriptRegEx">^[\S]{3,30}$</Property>
<Property name="UsernameJavaRegExViolationErrorMsg">Username pattern policy violated</Property>
<Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>
<Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>
<Property name="PasswordJavaRegExViolationErrorMsg">Password length should be within 5 to 30 characters</Property>
<Property name="RolenameJavaRegEx">[a-zA-Z0-9._\-|//]{3,30}$</Property>
<Property name="RolenameJavaScriptRegEx">^[\S]{3,30}$</Property>
<Property name="SCIMEnabled">false</Property>
<Property name="IsBulkImportSupported">false</Property>
<Property name="EmptyRolesAllowed">true</Property>
<Property name="PasswordHashMethod">PLAIN_TEXT</Property>
<Property name="MultiAttributeSeparator">,</Property>
<Property name="MaxUserNameListLength">100</Property>
<Property name="MaxRoleNameListLength">100</Property>
<Property name="kdcEnabled">false</Property>
<Property name="defaultRealmName">WSO2.ORG</Property>
<Property name="UserRolesCacheEnabled">true</Property>
<Property name="ConnectionPoolingEnabled">false</Property>
<Property name="LDAPConnectionTimeout">5000</Property>
<Property name="ReadTimeout"/>
<Property name="RetryAttempts"/>
</UserStoreManager>
<AuthorizationManager class="org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager">
<Property name="AdminRoleManagementPermissions">/permission</Property>
<Property name="AuthorizationCacheEnabled">true</Property>
<Property name="GetAllRolesOfUserEnabled">false</Property>
</AuthorizationManager>
</Realm>
</UserManager>
The ${user.mgt.connection.url} has the form ldap://DOMAIN:389.
The datasources urls are of the form jbdc:postgresql://DOMAIN:5432/master.
I have verified that I am able to make a connection to the databases and this LDAP instance from the machine this is running on, so it is not a firewall issue.
I have also been able to start up WSO2 with this configuration with a different LDAP and set of databases.
I'm looking for more clarification about which database/LDAP instance that WSO2 is trying to connect to during the realm initialization, so that I can identify which part of the config the issue is with.

Answer 1

Have a look "dataSource" property in user-mgt.xml and you will find JNDI name of the relevant data source search for the same name in master-datasource.xml there you will get corresponding DB connection parameters. You can verify connection by using some external database client. Either LDAP or JDBC you need to have healthy database connection. If you can attache user-mgt.xml and master-datasource.xml after removing sensitive data like DB password exact problem can be tracked.