It is easy to use each adapter by command connect.
for example:
!connect jdbc:calcite:model=target/test-classes/model.json admin admin
but I have not found in documentation how to do queries involving heterogeneous sources.
I would like to do a query involving entities that are in the two sources, CSV and MySQL.
You have to add two schemas in your in model.json file, one that targets CSV source and another one that targets JDBC MySQL source.
If the sources contain tables that are somehow connected, then you can create queries like this:
SELECT csv_source.table1.field1, mysql_source.table2.field2,
csv_source.table1.joint_field, mysql_source.table2.joint_field
FROM csv_source.table1
JOIN mysql_source.table2
ON joint_field;
Related
I'm trying to ingest data from different tables with in same database using Data fusion Multiple database tables plugin to bigquery tables using multiple big query tables sink. I write 3 different custom SQL and add them inside the plugin section which is under "Data Section Mode" > "Custom SQL Statements".
The problem is When I preview or deploy and run the pipeline I get the error "BigQuery Multi Table has no outputs. Please check that the sink calls addOutput at some point."
What I try to figure out this problem;
Run custom SQL on database and worked properly.
Create pipelines that are specific for custom SQLs but it's like 1 table ingestion from sql server to bigquery table as sink. it worked properly.
Try different Data Section Mode under multiple database tables plugin that is Table Allow List , works but it's just insert all data with no option to transform any column or filtering. Did that one to see if plugin can reach the database and able to read data ,it can read.
Data Pipeline - Multiple Database Tables Plugin Config - 1
Data Pipeline - Multiple Database Tables Plugin Config - 2
As a conclusion I would like to ingest data from one database with multiple tables with in one data pipeline. If possible I would like to do it with writing custom sqls for each tables.
Open for any advice and try.
Thank you.
I tried to connect to ODBC with PowerBi using this string connection
Driver={Client Access ODBC Driver
(32-bit)};System=xxxxx.xxx.xxxxx;libraries=XXXXXX;naming=system;transaction
isolation=read committed;
Connection is done but i cannot see the right tables, i see 3 folders
EXPLOIT, INSTAL, QGPL
With different tables inside that are not tables when I connect with squirrel client, for example.
I know there are few elements to understand. Someone has any ideas?
UPDATE
I found out this three catalogs (EXPLOIT, INSTAL and QGPL) also in Squirrel, but I cannot see all others catalogs that I see in Squirrel. Could be any limited views? The user is always the same.
Seems that the default library list for your server does not include QSYS2. You can access most of the DB2 for i catalog files in QSYS2. Try this:
Select * from qsys2.systables;
That should show you all the tables your heart desires, as well as the schema (library) that the table resides in.
BTW, I don't think EXPLOIT, INSTAL, and QGPL are catalogs. They are likely libraries, well QGPL is definitely a system library supplied by IBM. The other two seem to be something provided by some 3rd party app.
I'm installing WSO2IS 5.10.0 and I am creating five PostgreSQL databases per the column titled Recommended Database Structure in this document:
https://is.docs.wso2.com/en/next/setup/setting-up-separate-databases-for-clustering/
Actually it's six databases if you count the CARBON_DB. The five PostgreSQL databases are named as follows: SHARED_DB, USERSTORE_DB, IDENTITY_DB, CONSENT_MGT_DB and BPS_DB. I already have them configured in the deployment.toml file. I've created the databases in PostgreSQL and I have to manually execute the SQL files against each database in order to create the schema for each database. Based on the document in the link, I have figured out which SQL files to execute for four of the databases. However, I have no idea what SQL files I need to execute to create the USERSTORE_DB schema. It's got to be one of the files under the dbscripts directory but I just don't know which one(s). Can anybody help me on this one?
The CARBON_DB contains product-specific data. And by default that stores in the embedded h2 database. There is no requirement to point that DB to the PostgreSQL database. Hence you need to worry only about these databases SHARED_DB, USERSTORE_DB, IDENTITY_DB, CONSENT_MGT_DB and BPS_DB.
As per your next question, You can find the DB scripts related to USER_DB(USERSTORE_DB) in /dbscripts/postgresql.sql file. This file has tables starting with the name UM_. These tables are the user management tables. You can use those table sql scripts to create tables in USERSTORE_DB.
Refer the following doc for more information
[1]https://is.docs.wso2.com/en/5.10.0/administer/user-management-related-tables/
I see in the tutorials how to import data using Python. However, is it possible to manipulate a table or create a new one using Python? For example, I import data using Sharepoint. I can't wrap SQL in Python because the databases are only accessible through intranet, which PowerBI is not part of. Therefor, I need to input the data using one of PowerBI's connectors, but I'd like to manipulate the tables using Pandas. Is this possible?
You can use Python scripts to manipulate existing data. This can be done in the "Transform" tab under the Edit queries section. There is an option to select "Run Python Script". Once you select this option a dialog box will open and you can write the Python script you want. If you run into any issues you can refer to the following video:
https://www.youtube.com/watch?v=pF_JZk_ghCM
Hope this helps.
I am trying to use django's database routers to shard my database, but I am not able to find a solution for that.
I'd like to define two databases, create the same table in both and then save the even rows in one db, the odd ones in the other one. The examples in the documentation show how to write to a master db and to read from the readonly slaves, which is not what I want, because I don't want to store the whole dataset in both dbs.
Do know any webpage explaining what I am trying to do?
Thank you
PS: I am using Postgresql and I know there are tools to achieve the same goal at DB level. My goal is to study if it can also be done in django and to explore if there are some advantages by doing this.