WSO2-AM: API calls in store fail after restart - wso2

Question

We've setup the most recent versions of
API Manager
Identity Server
Data Analytics Server
in separate pods on a kubernetes cluster (we've used the existing scripts as a basis for creating the docker images), all sharing relevant MySQL databases hosted in another pod on the same cluster. After a lot of configuring everything works as expected, we can add users/roles, create/publish/invoke APIs and collect analytics on these calls. The last issue we're facing however is, that once an API got published, it's possible to invoke it in the store. The problem appears once the API-manager pod gets restarted (for example by scaling the relevant deployment down/up, not by using the carbon dashboard), once the API-Manager is up and running again after such a (hard) restart, the published APIs are still visible in the store, however, if one tries to make a call to one of the (previously working) endpoints, an 404 message is returned and the API-manager logs the following error:
INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS =
Message dispatched to the main sequence. Invalid URL, RESOURCE =
/"context"/"endpoint being called"
where "context" is the context set for a given API while publishing it and "endpoint being called" the REST endpoint of the service which should receive the call. Further, if we try to make a change to the API in the publisher (i.e. re-publish it), there seems to be a 50/50 chance that it works and fixes all issues till the next restart of the pod or an error is thrown when clicking the "Next: Implement" button on the first step, which has been recorded here already (although it says that it has been resolved in a previous version already).
The remaining setup is the following:
Users/Roles
User: prod, used for creating and publishing an API in the publisher
Roles: prod-role (empty role with no priviledges), internal/creator,
internal/publisher
User: cons, used for invoking an API in the store
Roles: cons-role (empty role, no priviledges, just for providing
access to the API in the store), internal/subscriber
Databases
WSO2_CARBON_DB referees to a database on the MySQL pod, used only by the API-Manager
WSO2REG_DB is supposed to be the shared registry database between the three modules (am, das, is)
excerpt from master-datasources.xml from the am:
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used by the registry</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://mysql-apimdb:3306/amcarbondb?autoReconnect=true&useSSL=false</url>
<username>wso2carbon</username>
<password>abc</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2AM_DB</name>
<description>The datasource used for API Manager database</description>
<jndiConfig>
<name>jdbc/WSO2AM_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://mysql-apimdb:3306/apimgtdb?autoReconnect=true&useSSL=false</url>
<username>wso2carbon</username>
<password>abc</password>
<defaultAutoCommit>false</defaultAutoCommit>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2UM_DB</name>
<description>The datasource used by user manager</description>
<jndiConfig>
<name>jdbc/WSO2UM_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://mysql-apimdb:3306/userdb?autoReconnect=true&useSSL=false</url>
<username>wso2carbon</username>
<password>abc</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2REG_DB</name>
<description>The datasource used for registry</description>
<jndiConfig>
<name>jdbc/WSO2REG_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://mysql-apimdb:3306/regdb?autoReconnect=true&useSSL=false</url>
<username>wso2carbon</username>
<password>abc</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
All of these databases get populated upon usage and are initialized using the DBscripts from the different products for the initial setup. Further, the registry.xml is configured as follows (showing just the changes, the remainder of the file is as it is configured by default):
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<enableCache>false</enableCache>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>
<dbConfig name="govregistry">
<dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9943/registry">
<id>gov</id>
<dbConfig>govregistry</dbConfig>
<cacheId>wso2carbon#jdbc:mysql://mysql-apimdb:3306/regdb</cacheId>
<readOnly>false</readOnly>
<enableCache>false</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/config</targetPath>
</mount>
In the end it seems to be an issue with some data being lost upon restart as the memory on the am pod is not persistent, since we've moved all databases to a persistent storage on another pod however, this shouldn't be a problem -- unless we missed something. Is there another registry/data source that need to be added/changed? All other default data sources not shown above (STATS, MB_STORE, METRICS) are setup to use the MySQL server as well, so unless there is something else than the data sources provided in the datasource folder, all data should remain on the corresponding MySQL databases.

Answer 1

I guess you haven't used any persistent storage for API Manager. When you create an API, the API artifact is created in the file system and stored in wso2am-2.1.0/repository/deployment/server/synapse-configs/de‌​fault/api location. You need to have a persistent storage for 'wso2am-2.1.0/repository/deployment/server/'. In your case, during the restart, you don't have relevant data as you have a fresh pack.

Answer 2

this can be fixed by move to persistent volume following folders:
as was suggested:
<WSO2_CARBON_HOME>/repository/deployment/server
but, In case of multitenant installation, need to make persistent also:
<WSO2_CARBON_HOME>/repository/tenants