How does one bulk insert data with Postgres into QuestDB?
The following does not work
CREATE TABLE IF NOT EXISTS employees (employee_id INT, last_name STRING,first_name STRING);
INSERT INTO employees
(employee_id, last_name, first_name)
VALUES
(10, 'Anderson', 'Sarah'),(11, 'Johnson', 'Dale');
For inserting data in bulk, there are a few options. You can use CREATE AS SELECT to bulk insert from an existing table which is closest to your example:
CREATE TABLE employees
AS (SELECT employee_id, last_name, first_name FROM existing_table)
Or you can use prepared statements, there are full working examples in a few languages in the QuestDB Postgres documentation, here is a snippet from the Python example:
# insert 10 records
for x in range(10):
cursor.execute("""
INSERT INTO example_table
VALUES (%s, %s, %s);
""", (dt.datetime.utcnow(), "python example", x))
# commit records
connection.commit()
Or you can bulk import from CSV, i.e.:
curl -F data=#data.csv http://localhost:9000/imp
Related
I'm pretty novice at programming (recently learned functions), and have found myself re-writing the same "insert into mysql table" function (below) from script to script... mainly to just modify these two section - (name,insert_ts) &&& VALUES (%s, %s)
Is there a good way to re-write the below to accept ANY number of values , based on length of a tuple that contains values as well as inserting the column headers based on 'labels' list? VALUES (%s, %s) and this part (name,insert_ts)
list_of_tuples = [] #list of records to be inserted.
#take a list of dictionaries - and create a list of tuples in proper format/order
for dict1 in output:
one_list = []
one_list.extend((dict1['name'],dict1['insert_ts']))
list_of_tuples.append(tuple(one_list))
labels = ['name', 'insert_ts']
#db_write accepts table name as str, labels as str, and output as list of tuples
def db_write(table,labels,output):
local_cursor.executemany(""" INSERT INTO my_table
(name,insert_ts) #this is pulled from 'labels'
VALUES (%s, %s) #number of %s comes from len(labels)
"""
, list_of_tuples)
local_db.commit()
local_db.close()
#print 'done posting!'
Or, is there a better way to accomplish what I'm trying to do, using mysqldb?
Thank you all in advance!
After a bit of experience (3 months, heh), wanted to update everyone on the solution that seems to work pretty well!
Instead of using mysqldb, I spent some time learning how to use SQL Alchemy python package, and would recommend everyone do the same!
SQL Alchemy allows you to:
1) Define a table within python code (used Excel to come up with column names, etc).
2) Most important! You can pass on a dictionary to SQL Alchemy, and as long as dictionary's key names match the table's key names, everything will magically get posted to your SQL table. If you have 60 columns in your sql table, but your dict has only two keys - BAM, SQL Alchemy will take care of everything and post just the two values, and leave the other values in MySQL as blanks. MAGIC!
I intend to create a table in sqlite3 with the column names in a list [item1,...]. I don't want to hard code the column names while creating the table because the number of columns and even names are dynamic. I want the correct syntax to get the column names directly from this list. Thank you. I'm using python.
You can do something like this:
import contextlib
import sqlite3
fields = ["field1", "field2", "field3"]
with contextlib.closing(sqlite3.connect(':memory:')) as conn:
cursor = conn.cursor()
columns = ", ".join("{field} TEXT".format(field=field) for field in fields)
cursor.execute("""
CREATE TABLE IF NOT EXISTS your_table(
id INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE,
{columns}
)
""".format(columns=columns))
conn.commit()
But, you should consider using an ORM, like SqlAlchemy. Here is a tutorial for that: https://www.pythoncentral.io/introductory-tutorial-python-sqlalchemy/
Edit
To go further, your insert statement could be built like this:
stmt = ("INSERT INTO mytable ({fields}) VALUES ({marks})"
.format(fields=", ".join(fields), marks=", ".join("?" * len(fields))))
For instance:
INSERT INTO mytable (field1, field2, field3) VALUES (?, ?, ?)
I am very new to sql and intermediate at python. Using sqlite3, how can I get a print() list of of primary and foreign keys (per table) in my database?
Using Python2.7, SQLite3, PyCharm.
sqlite3.version = 2.6.0
sqlite3.sqlite_version = 3.8.11
Also note: when I set up the database, I enabled FKs as such:
conn = sqlite3.connect(db_file)
conn.execute('pragma foreign_keys=ON')
I tried the following:
conn=sqlite3.connect(db_path)
print(conn.execute("PRAGMA table_info"))
print(conn.execute("PRAGMA foreign_key_list"))
Which returned:
<sqlite3.Cursor object at 0x0000000002FCBDC0>
<sqlite3.Cursor object at 0x0000000002FCBDC0>
I also tried the following, which prints nothing (but I think this may be because it's a dummy database with tables and fields but no records):
conn=sqlite3.connect(db_path)
rows = conn.execute('PRAGMA table_info')
for r in rows:
print r
rows2 = conn.execute('PRAGMA foreign_key_list')
for r2 in rows2:
print r2
Unknown or malformed PRAGMA statements are ignored.
The problem with your PRAGMAs is that the table name is missing. You have to get a list of all tables, and then execute those PRAGMAs for each one:
rows = db.execute("SELECT name FROM sqlite_master WHERE type = 'table'")
tables = [row[0] for row in rows]
def sql_identifier(s):
return '"' + s.replace('"', '""') + '"'
for table in tables:
print("table: " + table)
rows = db.execute("PRAGMA table_info({})".format(sql_identifier(table)))
print(rows.fetchall())
rows = db.execute("PRAGMA foreign_key_list({})".format(sql_identifier(table)))
print(rows.fetchall())
SELECT
name
FROM
sqlite_master
WHERE
type ='table' AND
name NOT LIKE 'sqlite_%';
this sql will show all table in database, for eache table run sql PRAGMA table_info(your_table_name);, you can get the primary key of the table.
Those pictures show what sql result like in my database:
first sql result
second sql result
I have a table 'Person' with columns as 'Person_id as primary key','DOB' and 'place' as follows:
'Person'
Person_id |Name|DOB | place
Another table is "employee" where emp_id is primary key as follows:
'employee'
Person_id |emp_id|dateofjoin
And one more table "Details":
'Details'
emp_id|competency|rating
Now what i want is once i add the 'Person' table details the rest of the two tables as'employe' and 'Details' to get updated also with respect to the new Person added in the Person table. So, how can i have this using sql query? Also i want to clear that i am not very much familiar with database.
I think your after something like this ( for SQL Server ):
Create Procedure dbo.CreateMyEmployee ( #empName varchar(50),
#dob datetime,
#doj datetime,
#place as varchar(100),
#competency varchar(100),
#rating int)
As
Begin
Declare #empId int
Begin Transaction
Begin Try
Insert into Person (Name, DOB, Place)
Values ( #empName, #dob, #place)
Insert into employe (Name, dateofJoin) -- Assuming emp_id is identity columen
Values ( #empName, #doj)
Select #empId = SCOPE_IDENTITY()
Insert Into Details(emp_id, competency, rating)
Values (#empId, #competency, #rating)
Commit transaction
End Try
Begin Catch
Rollback Transaction
SELECT ERROR_NUMBER() AS ErrorNumber, ERROR_MESSAGE() AS ErrorMessage
End Catch
End
Here's how I can do it when MySQL is the backend,
cursor.execute('show tables')
rows = cursor.fetchall()
for row in rows:
cursor.execute('drop table %s; ' % row[0])
But how can I do it when postgresql is the backend?
cursor.execute("""SELECT table_name FROM information_schema.tables WHERE table_schema='public' AND table_type != 'VIEW' AND table_name NOT LIKE 'pg_ts_%%'""")
rows = cursor.fetchall()
for row in rows:
try:
cursor.execute('drop table %s cascade ' % row[0])
print "dropping %s" % row[0]
except:
print "couldn't drop %s" % row[0]
Courtesy of http://www.siafoo.net/snippet/85
You can use select * from pg_tables; get get a list of tables, although you probably want to exclude where schemaname <> 'pg_catalog'...
Based on another one of your recent questions, if you're trying to just drop all your django stuff, but don't have permission to drop the DB, can you just DROP the SCHEMA that Django has everything in?
Also on your drop, use CASCADE.
EDIT: Can you select * from information_schema.tables; ?
EDIT: Your column should be row[2] instead of row[0] and you need to specify which schema to look at with a WHERE schemaname = 'my_django_schema_here' clause.
EDIT: Or just SELECT table_name from pg_tables where schemaname = 'my_django_schema_here'; and row[0]
Documentation says that ./manage.py sqlclear Prints the DROP TABLE SQL statements for the given app name(s).
I use this script to clear the tables, I put it in a script called phoenixdb.sh because it burns the DB down and a new one rises from the ashes. I use this to prevent lots of migrations in the early dev portion of the project.
set -e
python manage.py dbshell <<EOF
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
EOF
python manage.py migrate
This wipes the tables from the db without deleting the db. Your Django user will need to own the schema though which you can setup with:
alter schema public owner to django-db-user-name;
And you might want to change the owner of the db as well
alter database django-db-name owner to django-db-user-name;
\dt is the equivalent command in postgres to list tables. Each row will contain values for (schema, Name, Type, Owner), so you have to use the second (row[1]) value.
Anyway, you solution will break (in MySQL and PostgreSQL) when foreign-key constraints are involved, and if there aren't any, you might get troubles with the sequences. So the best way is in my opinion to simply drop the whole database and call initdb again (which is also the more efficient solution).