I have configured API Manager 2.0.0 & API Manager Analytics Pack to use MySQL databases.
For each server, there exists a WSO2AM_STATS_DB. I have given these differing names on my MySQL server. I have also pointed my datasources in master-datasources.xml(for APIM) & stats-datasources.xml(for Analytics) to the relevant databases.
I couldn't find any relevant schema(dbscripts) for these databases in their respective packs.
On running, the Analytics database is populated but the APIM database isn't and throws an exception. The Analytics database not only gets the schema but also the invocation details of my API.
I am unable to get the stats on my dashboard though.
Previously, I (unwittingly) configured the h2-repository stats database to be the same for both servers (due to the folder structure) and was able to get all the statistics on my dashboard in the publisher.
Other configurations I have tried :
On the MySQL Server, pointed it to the same database (the Analytics one with the schema) but with no results on my dashboard (after waiting for a while).
Both datasources (WSO2AM_STATS_DB) in 2 servers should be pointed to the same database. There are no database scripts for this. Tables are created automatically.
By default in both servers, Stats DB path comes like this. (note ../ part)
<url>jdbc:h2:../tmpStatDB/WSO2AM_STATS_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000;AUTO_SERVER=TRUE</url>
So if you extract both servers to the same directory as mentioned in this doc, both datasources will be pointing to the same database (inside tmpStatDB) like this.
/parent_dir
|__wso2am-2.0.0/
|__wso2am-analytics-2.0.0/
|__tmpStatDB/
So, what happens here is, wso2am-analytics writes stats data to shared database, then apim reads it and shows data on its databases.
Related
We have noticed that if a table is empty in SQL Server, the empty table does not come via DMS. Only after inserting a record it starts to show up.
Just checking, is there a way to get the schema only from DMS?
Thanks
You can use Schema conversion tool for moving DB objects and Schema. Its a free tool by AWS and can be installed on On-Prem server or on EC2. It gives a good report before you can actually migrate the DB schema and other DB objects. It shows how many Tables, SP's Funcs etc can be directly migrated and shows possible solutions too.
In wso2am/repository/conf/datasources/*.xml, I can see 5 datasources :
<name>WSO2_CARBON_DB</name>
<name>WSO2AM_DB</name>
<name>WSO2AM_STATS_DB</name>
<name>WSO2_MB_STORE_DB</name>
<name>WSO2_METRICS_DB</name>
But in wso2am/dbscripts, I can find only 4 scripts for 4 databases (no script for WSO2AM_STATS_DB).
Is WSO2AM_DB supposed to stay an H2 database in production ? or should it point to an existing database ?
You don't need to create tables in WSO2AM_STATS_DB manually. Just creating a database is enough. Tables are automatically created by analytics scripts.
Table creation of the statistics database is handled by the Analytics
scripts when you configure APIM Analytics, so you will create the
statistics database in this step but will not specify a source script.
Ref: https://docs.wso2.com/display/CLUSTER44x/Clustering+API+Manager+2.0.0
I'm performing WSO2 API manager + Analytics 2.0 POC now. When i change datasource from H2 to Oracle, in wso2am-2.0.1-SNAPSHOT, there are 2 data source config files:
master-datasources.xml & metrics-datasources.xml, according Installing and configuring the databases, there should be WSO2AM_DB, WSO2UM_DB and the WSO2REG_DB datasource configurations, but i just find WSO2_CARBON_DB & WSO2AM_DB, so my questions are
Is WSO2_CARBON_DB = WSO2UM_DB + WSO2REG_DB?
for WSO2_METRICS_DB, according Enabling Metrics and Storage Types, if we enable JDBC storage, can we store all components metrics information in one shared db or it needs one db per component(local)?
What's WSO2_MB_STORE_DB used for? from the scripts, it's for Message Store and Andes Context Store. Can we keep to use H2 in prod. cluster env.?
When i config wso2am-analytics-2.0.0-SNAPSHOT, i have below questions:
Can we share WSO2_CARBON_DB setting for both APIMGRT related components and analytics? or it's better to not share?
For WSO2AM_STATS_DB, is analytics resposible to aggregate and write to it, APIMGRT responsible to read? Which APIMGRT components need to read it?
For analytics related store, it supports RDBMS, Cassandra, HBase, but it does not support mongodb, right?
for GEO_LOCATION_DATA, What's this used for? Can we just use H2 in prod. env.?
APIM:
1) In default pack, yes. But in a production environment, it is recommended to separate them as WSO2_CARBON_DB, WSO2UM_DB and WSO2REG_DB (Please note you need WSO2_CARBON_DB too, to store local data. And this can be an h2 database)
2) You can have a shared DB
3) WSO2_MB_STORE_DB is required only if you use Advanced Throttling. Tables for this are created by APIM itself. So you don't need to run any scripts on it.
APIM Analytics:
1) You can share WSO2UM_DB and WSO2REG_DB. But don't share (local) WSO2_CARBON_DB.
2) Store and Publisher
3) See WSO2 DAS with MongoDB
4) GEO_LOCATION_DATA is used for Geolocation Based Statistics. H2 is not recommended.
Can anyone explain to me what state the data should be in for a healthy sitecore instance in each database?
for example:
We currently have an issue with publishing in a 2 server setup.
Our staging server hosts the SQL instance and the authoring / staging instance of sitecore.
We then have a second server to host just the production website for our corp site.
When I look in the master database the PublishQueue table is full of entries and the same table in the web database is empty.
Is this correct?
No amount of hitting publish buttons is changing that at the moment.
How do I determine what the state of an item is in both staging and production environments without having to write an application on top of the sitecore API which I really don't have time for?
This is a normal behavior for the Publish Queue of the Web Database to be blank. The reason is because changes are made on the Master database which will add an entry in the Publish Queue.
After publishing, the item will not be removed from the Publish Queue table. It is the job of the CleanupPublishQueue to cleanup the publish queue table.
In general, tables WILL be different between the two databases as they are used for different purposes. Your master database is generally connected to by authors and the publishing logic, while the web database is generally used as a holding place for the latest published version of content that should be visible.
In terms of debugging publishing, from the Sitecore desktop, you can swap between 'master' and 'web' databases in the lower right corner and use the Content Editor to examine any individual item. This is useful for spot checking individual items have been published successfully.
If an item is missing from 'web', or the wrong version is in 'web', you should examine the following:
Publishing Restrictions on the item: Is there a restriction applied to the item or version that prevents it from publishing at this time?
Workflow state: Is the item/version in the final approved workflow state? You can use the workbox to do a quick check for items needing approval.
Connection strings: Is your staging system connection strings setup to connect to the correct 'web' used by the production delivery server?
The Database table [PublishQueue] is a table where all save and other mutations are stored. This table is used by a Incremental Publish. Sitecore get all the items from the PublishQueue table that were modified more recently than the last incremental publish date. The PublishQueue tabel is not used by a full publish
So it is okay that this Table contain a lot of records on the Master. The web database has the same database scheme. (not the same data, web contain only one version of a item, optimize for performance) The PublishQueue on the web is Empty this is normal.
To Know the state of an item compair the master version with the web version, there can be more than 1 webdatabase, The master database do not know the state/version of the web database
I was trying to create a report from master database in analytics reports. (Stimulsoft Report Designer)
As it explains in the reports cookbook, I have created a "mrt file" (Report UI) and a report definition item in Engagement analytics.
I have configured the datasource item as query item
(/sitecore/system/Settings/Analytics/Reports SQL Queries/Visit Pages).
It worked.
But then I tried with a query using the master database, in the SQL query item I specifically mentioned the database as 'testProjectMaster' to point to master database. It did not work!
Then I figured out that in "/sitecore/system/Settings/Analytics/Reports SQL Queries/Visit Pages" item and other query items, it does not specify the database, that means by default sitecore queries the analytics database.
Is this a limitaion in sitecore, cant we query the master databse for reports? Are there any good resources to follow on creating reports?
I suggest taking the SQL from the Visit Pages report and running it in SQL Server Management Studio. There, you will be able to quickly see what's preventing your query from running. If I had to venture a guess, I would suspect that your SQL user account does not have db_datareader access to the master database.
The default SQL queries provided by Sitecore assume that the DMS is configured as the default database in the connection string. This, however, does not prevent you from querying other databases or doing cross-database joins like so:
SELECT TOP 100 * FROM Pages
INNER JOIN Sitecore_Master.dbo.Items AS MasterItems ON Pages.ItemId = MasterItems.ID
A word of caution.. from my experience, this can really slow down your reports as it does not take advantage of indexing and creating indexed views doesn't work across multiple databases.