Hi i have a scenerio i have a database dump which i want to import in my new rails web applications database i have used activerecord etl gem but now the demand is that use kettle etl for importing the data. i have no idea of kettle can someone help me or link me the tutorials from where by following it i can do my job?
thanks in advance :)
Actually it's very easy. Just use Table Input to read from current MySQL Table, do some transformation, pipe it to Table Output. In the Table Output, point it to target MySQL connection, fill in the table name and click "Generate SQL" button to execute the DML generated.
For example you can see my article for a sample Excel to MySQL, but the operation can easily converted for MysQL-MySQL migration. (The site is in Indonesian - you can translate it using Google Translator)
http://pentaho.phi-integration.com/kettle/export-excel-ke-mysql
Related
I'm trying to ingest data from different tables with in same database using Data fusion Multiple database tables plugin to bigquery tables using multiple big query tables sink. I write 3 different custom SQL and add them inside the plugin section which is under "Data Section Mode" > "Custom SQL Statements".
The problem is When I preview or deploy and run the pipeline I get the error "BigQuery Multi Table has no outputs. Please check that the sink calls addOutput at some point."
What I try to figure out this problem;
Run custom SQL on database and worked properly.
Create pipelines that are specific for custom SQLs but it's like 1 table ingestion from sql server to bigquery table as sink. it worked properly.
Try different Data Section Mode under multiple database tables plugin that is Table Allow List , works but it's just insert all data with no option to transform any column or filtering. Did that one to see if plugin can reach the database and able to read data ,it can read.
Data Pipeline - Multiple Database Tables Plugin Config - 1
Data Pipeline - Multiple Database Tables Plugin Config - 2
As a conclusion I would like to ingest data from one database with multiple tables with in one data pipeline. If possible I would like to do it with writing custom sqls for each tables.
Open for any advice and try.
Thank you.
I see in the tutorials how to import data using Python. However, is it possible to manipulate a table or create a new one using Python? For example, I import data using Sharepoint. I can't wrap SQL in Python because the databases are only accessible through intranet, which PowerBI is not part of. Therefor, I need to input the data using one of PowerBI's connectors, but I'd like to manipulate the tables using Pandas. Is this possible?
You can use Python scripts to manipulate existing data. This can be done in the "Transform" tab under the Edit queries section. There is an option to select "Run Python Script". Once you select this option a dialog box will open and you can write the Python script you want. If you run into any issues you can refer to the following video:
https://www.youtube.com/watch?v=pF_JZk_ghCM
Hope this helps.
I am new in informatica cloud. I have list of queries ready in my table. Like below.
Now I want to take one by one query from this table which work as a source query and whatever results return which I need to load into target. All tables were already created in source and target.
I just need to copy the data based on dynamic queries which kept in my one of sql tables.
If anyone have any idea then please share your toughs with me. It great helps to me.
The source connection will be the connector to your source database and the Source Type will be query. From there it depends how you are managing your variables. See thread on Informatica Network for links to multiple examples.
Read the table like normally you would do in the cloud. Then pass each of the record into the sql transformation for execution. configure where the sql transformation has to execute and it will run the queries in the database you want.
you can use a SQL task to run dynamic SQL queries.
link to using SQL task approach: https://www.datastackpros.com/2019/12/informatica-cloud-incremental-load_14.html
I have C++ code, and from it I need to access the DB and make a query in table (with name NECE_TABLE, which has 2 columns - IntID and Status).
Here I need to get "status" column value from DB table (NECE_TABLE) using the IntID from C++ code.
Any help will be greatly helpful. Thanks in advance
Your question is very vague, but in summary you need to:
Use an appropriate client library supported by your database to connect to that database using some user credentials with appropriate permissions for SELECTing from your table
Execute a SQL select to fetch the data you want
There's some confusion as to which database you're using.
If you're using Oracle, you can use the OCCI client library to connect to the database and execute SQL statements. See section 2 of the linked document, where it describes connecting to a database and executing SQL queries.
Take a look at this link - it's a simple tutorial on how to get started with MySQL and C++. You say you are using vanilla SQL in your tags, but the two should be compatible as long as you stick to the more basic queries.
Can I use a web service as a data source for creating Excel pivot tables?
Currently, the soure data for the pivot table is being exported from our SQL db to a CSV file. Then, the CSV file is loaded into a worksheet. From there, a pivot table is created in the same workbook.
Customers login to a website, click some links, and an excel file (with data and pivot table) is generated. This is a public app so the preference is to not connect directly to the DB.
We control the database and generate the output. We are looking to streamline this process. The SQL db and pivot tables can not / will not change.
See http://www.vertex42.com/News/excel-web-query.html
What format does the "public-facing website" use in making the data available? A data file, a table on a web page? This issue will determine how much of a scraping operation you'll need to do.
You'll still need to write the web service and have it run on a server. A possible alternative is to use Yahoo Pipes to do the conversions for you.