In wso2am/repository/conf/datasources/*.xml, I can see 5 datasources :
<name>WSO2_CARBON_DB</name>
<name>WSO2AM_DB</name>
<name>WSO2AM_STATS_DB</name>
<name>WSO2_MB_STORE_DB</name>
<name>WSO2_METRICS_DB</name>
But in wso2am/dbscripts, I can find only 4 scripts for 4 databases (no script for WSO2AM_STATS_DB).
Is WSO2AM_DB supposed to stay an H2 database in production ? or should it point to an existing database ?
You don't need to create tables in WSO2AM_STATS_DB manually. Just creating a database is enough. Tables are automatically created by analytics scripts.
Table creation of the statistics database is handled by the Analytics
scripts when you configure APIM Analytics, so you will create the
statistics database in this step but will not specify a source script.
Ref: https://docs.wso2.com/display/CLUSTER44x/Clustering+API+Manager+2.0.0
Related
We have noticed that if a table is empty in SQL Server, the empty table does not come via DMS. Only after inserting a record it starts to show up.
Just checking, is there a way to get the schema only from DMS?
Thanks
You can use Schema conversion tool for moving DB objects and Schema. Its a free tool by AWS and can be installed on On-Prem server or on EC2. It gives a good report before you can actually migrate the DB schema and other DB objects. It shows how many Tables, SP's Funcs etc can be directly migrated and shows possible solutions too.
I'm installing WSO2IS 5.10.0 and I am creating five PostgreSQL databases per the column titled Recommended Database Structure in this document:
https://is.docs.wso2.com/en/next/setup/setting-up-separate-databases-for-clustering/
Actually it's six databases if you count the CARBON_DB. The five PostgreSQL databases are named as follows: SHARED_DB, USERSTORE_DB, IDENTITY_DB, CONSENT_MGT_DB and BPS_DB. I already have them configured in the deployment.toml file. I've created the databases in PostgreSQL and I have to manually execute the SQL files against each database in order to create the schema for each database. Based on the document in the link, I have figured out which SQL files to execute for four of the databases. However, I have no idea what SQL files I need to execute to create the USERSTORE_DB schema. It's got to be one of the files under the dbscripts directory but I just don't know which one(s). Can anybody help me on this one?
The CARBON_DB contains product-specific data. And by default that stores in the embedded h2 database. There is no requirement to point that DB to the PostgreSQL database. Hence you need to worry only about these databases SHARED_DB, USERSTORE_DB, IDENTITY_DB, CONSENT_MGT_DB and BPS_DB.
As per your next question, You can find the DB scripts related to USER_DB(USERSTORE_DB) in /dbscripts/postgresql.sql file. This file has tables starting with the name UM_. These tables are the user management tables. You can use those table sql scripts to create tables in USERSTORE_DB.
Refer the following doc for more information
[1]https://is.docs.wso2.com/en/5.10.0/administer/user-management-related-tables/
I have configured API Manager 2.0.0 & API Manager Analytics Pack to use MySQL databases.
For each server, there exists a WSO2AM_STATS_DB. I have given these differing names on my MySQL server. I have also pointed my datasources in master-datasources.xml(for APIM) & stats-datasources.xml(for Analytics) to the relevant databases.
I couldn't find any relevant schema(dbscripts) for these databases in their respective packs.
On running, the Analytics database is populated but the APIM database isn't and throws an exception. The Analytics database not only gets the schema but also the invocation details of my API.
I am unable to get the stats on my dashboard though.
Previously, I (unwittingly) configured the h2-repository stats database to be the same for both servers (due to the folder structure) and was able to get all the statistics on my dashboard in the publisher.
Other configurations I have tried :
On the MySQL Server, pointed it to the same database (the Analytics one with the schema) but with no results on my dashboard (after waiting for a while).
Both datasources (WSO2AM_STATS_DB) in 2 servers should be pointed to the same database. There are no database scripts for this. Tables are created automatically.
By default in both servers, Stats DB path comes like this. (note ../ part)
<url>jdbc:h2:../tmpStatDB/WSO2AM_STATS_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000;AUTO_SERVER=TRUE</url>
So if you extract both servers to the same directory as mentioned in this doc, both datasources will be pointing to the same database (inside tmpStatDB) like this.
/parent_dir
|__wso2am-2.0.0/
|__wso2am-analytics-2.0.0/
|__tmpStatDB/
So, what happens here is, wso2am-analytics writes stats data to shared database, then apim reads it and shows data on its databases.
I'm performing WSO2 API manager + Analytics 2.0 POC now. When i change datasource from H2 to Oracle, in wso2am-2.0.1-SNAPSHOT, there are 2 data source config files:
master-datasources.xml & metrics-datasources.xml, according Installing and configuring the databases, there should be WSO2AM_DB, WSO2UM_DB and the WSO2REG_DB datasource configurations, but i just find WSO2_CARBON_DB & WSO2AM_DB, so my questions are
Is WSO2_CARBON_DB = WSO2UM_DB + WSO2REG_DB?
for WSO2_METRICS_DB, according Enabling Metrics and Storage Types, if we enable JDBC storage, can we store all components metrics information in one shared db or it needs one db per component(local)?
What's WSO2_MB_STORE_DB used for? from the scripts, it's for Message Store and Andes Context Store. Can we keep to use H2 in prod. cluster env.?
When i config wso2am-analytics-2.0.0-SNAPSHOT, i have below questions:
Can we share WSO2_CARBON_DB setting for both APIMGRT related components and analytics? or it's better to not share?
For WSO2AM_STATS_DB, is analytics resposible to aggregate and write to it, APIMGRT responsible to read? Which APIMGRT components need to read it?
For analytics related store, it supports RDBMS, Cassandra, HBase, but it does not support mongodb, right?
for GEO_LOCATION_DATA, What's this used for? Can we just use H2 in prod. env.?
APIM:
1) In default pack, yes. But in a production environment, it is recommended to separate them as WSO2_CARBON_DB, WSO2UM_DB and WSO2REG_DB (Please note you need WSO2_CARBON_DB too, to store local data. And this can be an h2 database)
2) You can have a shared DB
3) WSO2_MB_STORE_DB is required only if you use Advanced Throttling. Tables for this are created by APIM itself. So you don't need to run any scripts on it.
APIM Analytics:
1) You can share WSO2UM_DB and WSO2REG_DB. But don't share (local) WSO2_CARBON_DB.
2) Store and Publisher
3) See WSO2 DAS with MongoDB
4) GEO_LOCATION_DATA is used for Geolocation Based Statistics. H2 is not recommended.
I was able to clone one of my SQL Server database using the DMS. It copied clustered indexes, primary key definition etc along with the data.
However, it didn't not move/copy other constraints (identity, foreign key definition, default values) or any indexes.
I have generated / scripted out the indexes, default constraints and foreign keys, executed successfully. But is there a way to turn on the IDENTITY on respective columns ?
Figured out there is no way i can do this with AWS DMS as it do not import secondary/foreign keys, Indexes and Identity columns as well. You need to do it manually yourself by generating a script from SSMS or writing your own script.
Check this FAQ from Amazon:
Q. Does AWS Database Migration Service migrate the database schema for me?
To quickly migrate a database schema to your target instance you can rely on the Basic Schema Copy feature of AWS Database Migration Service. Basic Schema Copy will automatically create tables and primary keys in the target instance if the target does not already contain tables with the same names. Basic Schema Copy is great for doing a test migration, or when you are migrating databases heterogeneously e.g. Oracle to MySQL or SQL Server to Oracle. Basic Schema Copy will not migrate secondary indexes, foreign keys or stored procedures. When you need to use a more customizable schema migration process (e.g. when you are migrating your production database and need to move your stored procedures and secondary database objects), you can use the AWS Schema Conversion Tool for heterogeneous migrations, or use the schema export tools native to the source engine, if you are doing homogenous migrations like (1) SQL Server Management Studio's Import and Export Wizard, (2) Oracle's SQL Developer Database Export tool or script the export using the dbms_metadata package, (3) MySQL's Workbench Migration Wizard.