I need to copy some tables in one dashdb database over to separate dashdb database. Normally I would export the CSV file from one and load it into the other using the Web console, however one table in particular has a CLOB column and so we will need to export to an ixf + lob files and then import it. Unfortunately I can't see any easy way to do this as it looks like clpplus can only export to the server that the database is on (which I don't have access to) and I can't see any way to get it to export the lob files. Does anyone know how best to accomplish this?
If the CLOB values are in reality smaller than 32K you can try to transform them into a VARCHAR value as part of the SELECT statement that you provide to EXPORT.
If you really need to export LOB files you can write them to your users home dir inside the dashDB instance and then use the /home REST API to download the files e.g. with curl: https://developer.ibm.com/static/site-id/85/api/dashdb-analytics/
Another option is to export the table with the LOBs to a local machine and then import into another dashDB.
One way to export a dashDB table to a local client is to run the EXPORT command in a DB2 Command Line Processor (CLP) on your client machine. To do so, you need to install the IBM Data Server Runtime Client and then catalog your dashDB databases in the client, like this:
CATALOG TCPIP NODE mydash REMOTE dashdb-txn-small-yp-lon02-99.services.eu-gb.bluemix.net SERVER 50000;
CATALOG DATABASE bludb AS dash1 AT NODE mydash;
CONNECT TO dash1 USER <username> USING <password>;
Now, let's export the table called "mytable" so that the LOB column is written to a separate file:
export to mytable.del of del
lobfile mylobs
modified by lobsinfile
select * from mytable;
This export commands produces the files mytable.del and mylobs.001.lob. The file mytable.del contains pointers into the file mylobs.001.lob that specify the offset and length of each value.
If the LOB data is too large to fit into a single file, then additional files mylobs.002.lob, mylobs.003.lob, etc. will be created.
Note the exported data will be sent from dashDB to your local client in uncompressed form, which may take some time depending on the data volume.
If the .DEL and .LOB files reside on a client machine, such as your laptop or a local server, you can use the IMPORT command to ingest these files into a table with a LOB column. In the CLP you would first connect to the dashDB database that you want to load into.
Let's assume the original table has been exported to the files mytable.del and mylobs.001.lob, and that these files are now located on your client machine in the directory /mydata. Then this command will load the data and LOBs into the target table:
IMPORT FROM /mydata/mytable.del OF DEL
LOBS FROM /mydata
MODIFIED BY LOBSINFILE
INSERT INTO mytable2;
This IMPORT command can be run in a DB2 Command Line Processor on your client machine.
Related
I have a Postgres database on Heroku and I want to have that database hosted on ElephantSQL, but I cannot for the life of me find how to do it. I have a DUMP file downloaded from the Heroku Postgres database, but I cannot find how I put the data into ElephantSQL.
My database was linked to a Django app and I already have my instance on ElephantSQL linked to the a copy of the same Django app, but hosted on Render. I can see on ElephantSQL that the instance has all the tables, so I just need to put the data in there somehow.
Any tips or hints are welcome!
The tables that you already see may be the problem. It may also be that the parents of those tables are the problem. An SQL dump is just a series of SQL commands that run in order to write schema and data back into a new database.
The first SQL commands set up the databases, schemas, and tables. If your tables are already there, your import may be failing because it wants to create new tables that already exist. Sorry without more information on the specific errors you are seeing, it's hard to be more specific. Hopefully one of these 4 options help you. Let me know in the comments how it goes.
For all of the options below, I would suggest backing up your destination server database to be sure that you don't mess up the Django app that currently is working there even without your data.
Option 1: Look for a way to export your source database without database, schema, and table definitions.
Option 2: If you're careful, you can edit your dump file to remove those initial setup commands, and have the dump file start with the commands that only push the data to your tables.
Option 3: (IMPORTANT to backup the destination server first for this one.) Drop the database, schema, and/or tables on the destination server so that nothing is pre-defined, and see if your dump file can reconstruct everything again the way Django needs it on the destination server with your data.
Option 4: It may be a lot of tables, but you can usually export individual table data files with an option to not include table definitions. Then you have to import all the tables as separate imports. A mix of option 3 and 4 may work also where if you can't find an option to not include the definition, drop the tables at the destination and import each table independently.
Here is what worked for me using a Mac terminal:
Create a backup file from Heroku:
pg_dump -h heroku_host -d heroku_database -U heroku_user -p heroku_port -W -Ft > latest.dump
You'll be prompted to input the password from Heroku PostgreSQL. Note that these heroku_xxx are found in the Heroku PostgreSQL credentials.
Restore the backup from Heroku to ElephantSQL:
pg_restore -h elephant_host -d elephant_database -U elephant_user -p 5432 -W -Ft latest.dump
You'll be prompted to input the password from Heroku PostgreSQL. Note that these elephant_xxx are found in the ElephantSQL credentials and that they use the term "server" instead of host.
I want to export output of my select statement in Bigquery. There is "Save Results" option available but it has limit of 16,000 rows.
How can i export more than that?
If your data has more than 16,000 rows you'd need to save the result of your query as a BigQuery Table.
Afterwards, export the data from the table into Google Cloud Storage using any of the available options (such as the Cloud Console, API, bq or client libraries).
Finally, you can use any of the available methods in Google Cloud Storage (Cloud Console, API, gsutil or client libraries) to download the CSV file within your local environment.
Since for the time being you can't export data from a Table directly to a local file.
There is a quite easier step by using the export function. You can perform a simple query to export directly to GCS
EXPORT DATA OPTIONS(
uri='gs://MyBucket/path/to/file*.csv',
format='CSV',
overwrite=true,
header=true,
field_delimiter=',') AS
SELECT * from MyTable WHERE condition.
However, it's greatly possible that several files are generated. And thus you have to get all of them with gsutil.
The advantage of Daniel's solution, is that you can choose to export the table to only one file.
I'm installing WSO2IS 5.10.0 and I am creating five PostgreSQL databases per the column titled Recommended Database Structure in this document:
https://is.docs.wso2.com/en/next/setup/setting-up-separate-databases-for-clustering/
Actually it's six databases if you count the CARBON_DB. The five PostgreSQL databases are named as follows: SHARED_DB, USERSTORE_DB, IDENTITY_DB, CONSENT_MGT_DB and BPS_DB. I already have them configured in the deployment.toml file. I've created the databases in PostgreSQL and I have to manually execute the SQL files against each database in order to create the schema for each database. Based on the document in the link, I have figured out which SQL files to execute for four of the databases. However, I have no idea what SQL files I need to execute to create the USERSTORE_DB schema. It's got to be one of the files under the dbscripts directory but I just don't know which one(s). Can anybody help me on this one?
The CARBON_DB contains product-specific data. And by default that stores in the embedded h2 database. There is no requirement to point that DB to the PostgreSQL database. Hence you need to worry only about these databases SHARED_DB, USERSTORE_DB, IDENTITY_DB, CONSENT_MGT_DB and BPS_DB.
As per your next question, You can find the DB scripts related to USER_DB(USERSTORE_DB) in /dbscripts/postgresql.sql file. This file has tables starting with the name UM_. These tables are the user management tables. You can use those table sql scripts to create tables in USERSTORE_DB.
Refer the following doc for more information
[1]https://is.docs.wso2.com/en/5.10.0/administer/user-management-related-tables/
I am recreating a web app in Django that was running in a server but it was terminated, fortunately, I did a backup of all the code. My problem comes with the database because but I do not know how to transfer all the data from the old db.sqlite3 Django database web app into the new one.
I found a similar question as mine Django: transfer data from one database to another but the user wanted to transfer data from specific columns because their models.pyfrom the old and new databases were slightly different. In my case, my models.py from the old and new databases are the same.
Alternatives
I am using the DB Browser for SQLite to explore the content of the old database and I could add manually each row into the Django administration but this will take me too much time.
I could copy the old db.sqlite and replace it in the new web app because the models.py file remains the same but this solution is not appropriate IMO, this solution is rude and I think it goes against the good practices of Software.
How should I proceed for transferring data from the old database to the new one?
This seems like a one time copy of one db to another. I don't see how this goes against good software practice unless you have to be copying this db frequently. I've done it before when migrating servers and it doesn't cause any issues assuming the two instances of the application are the same build.
I was able to do some minor tricks in order to solve my problem because there is not a straightforward functionality that allows you to transfer all your data from two sqlite databases in Django. Here is the trick I did:
Download the sqlite db browser to explore and export the contents of your old database in a .csv file. Open you database with sqlite db browser and hit on the tables option and you will see all your tables, then do a right click on any of those and hit the export as a csv file option to generate the csv file (name_of_your_csv_file.csv). The other alternative is to use the sqlite3.exe to open your database in cmd or powershell and then doing the export with:
.tables #this lets you explore your available tables
headers on
mode csv
output name_of_your_csv_file.csv
2.There are two choices up to this point: You can either insert all the records at once to your new database or you can drop your existing tables from the new database and then recreate them and import the .csv file. I went for the drop option because there were more than 100 records to migrate.
# sqlite3
# check the structure of your table so you can re-create it
.schema <table_name>
#the result might be something like CREATE TABLE IF NOT EXISTS "web_app_navigator_table" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "ticket" varchar(120) NOT NULL);
#drop the table
drop table web_app_navigator_table
#re-create the table
create table web_app_navigator_table(id integer not null primary key autoincrement, ticket varchar(120) not null);
#do the import of the csv file
.import C:/Users/a/PycharmProjects/apps/navigator/name_of_your_csv_file.csv table_name_goes_here
You might see an error such as csv:1: INSERT failed datatype mismatch but this indicates that the first row of your csv file was not inserted because it contains the headers of the exported data from your old database.
I'm trying to load data to a Kudu table but getting a strange result.
In the Impala console I created an external table from the four HDFS files imported by Sqoop:
drop table if exists hdfs_datedim;
create external table hdfs_datedim
( ... )
row format
delimited fields terminated by ','
location '/user/me/DATEDIM';
A SELECT COUNT(*) tells me there lots of rows present. The data looks good when queried.
I use a standard select into to copy the results
INSERT INTO impala_kudu.DATEDIM
SELECT * FROM hdfs_datedim;
A SELECT COUNT(*) tells me impala_kudu.DATEDIM has four rows (the number of files in HDFS not the number of rows in the table.
Any Ideas?
Currently Sqoop does not support Kudu yet. You can import to HDFS and then use Impala to write data to Kudu.
The data created by sqoop was under the covers was a sequence of poorly formatted csv files. The import failed without an error because of data in the flat file. Watch out for date formats and text strings with delimiters embedded in the string.
If you have the data in HDFS in (csv/avro/parquet) format,then you can use the below command to import the files to Kudu table.
Prerequisites:
Kudu jar with compatible version (1.6 or higher)
spark2-submit --master yarn/local --class org.apache.kudu.spark.tools.ImportExportFiles <path of kudu jar>/kudu-spark2-tools_2.11-1.6.0.jar --operation=import --format=<parquet/avro/csv> --master-addrs=<kudu master host>:<port number> --path=<hdfs path for data> --table-name=impala::<table name>