I am trying to load Redshift table from stage table.
Whenever I try to insert data from the stage table, Redshift is locking table2 and it is not inserting any data.
Here is the python code:
insertSQL = "insert into table2 select c1,c2,c3,c4 FROM table_name "
load_redshift_table(insertSQL)
def load_redshift_table(insertSQL):
try:
conn = psycopg2.connect(
host="host",port="redshift_port",database="redshift_database",user="redshift_user",password="reshift_password")
cur = conn.cursor()
cur.execute(insertSQL)
conn.commit()
except Exception as e:
print("Error:", e)
finally:
cur.close()
Related
I'm django newbie, I have one fundamental and one technical questions.
I'm using Postgres DB. I used psycopg2 connection/cursor for fetching the data, there was some delay while establishing a connection. I read that, ORM takes care of low level activities such as establishing a connection, etc. If I use django, ORM will takes care of connection challenge ?
1.1. Can I expect same (low level activities) with raw() as well?
objects.raw(sql) returning Query but not records from the table.
I defined Student Model as below
class Student(models.Model):
firstname = models.CharField(max_length=100)
surname = models.CharField(max_length=100)
def __str__(self):
return self.firstname
While creating the view,
def studentList(request):
#posts = Student.objects.all() --> 1. working as expected (fetching all records firstname)
cursor = connection.cursor()
sql = "select * from api_student"
cursor.execute(sql)
posts = cursor.fetchone() --> 2. returning entire record
#posts = Student.objects.raw(sql) --> 3. RETURNING SQL QUERY NOT RECORD FROM TABLE ???
print(posts)
return render(request, 'output.html', {'posts':posts})
output:
<QuerySet [<Student: Anil>]>
<RawQuerySet: select * from api_student> --> this is the challenge, did I miss any
('Anil', 'kumar')
The raw() method takes a raw sql query, executes it, and returns a RawQuerySet instance. You can iterate on RawQuerySet like normal QuerySet and get objects.
sql = "select * from api_student"
student_qs = Student.objects.raw(sql)
for obj in student_qs:
print(obj.pk, obj.firstname, obj.surname)
I have this query in my views.py
cardId = request.POST.get('id')
with connection.cursor() as cursor:
cursor.execute("SELECT category1_id FROM app_card_tbl WHERE id =%s", [cardId])
row = cursor.fetchone()
cursor.execute("SELECT name FROM app_category1_tbl WHERE id =%s",[row])
tab1 = cursor.fetchone()
print(tab1)
And the output of this when print is
('thisisname',)
All i want is the output should looks like below must remove the `(',) sign it is possible?
thisisname
The reason this happens is because each record is put in a tuple, even if it contains a single column. You can print the first item with:
print(tab1[0])
or you can unwrap it when fetching the object:
tab1, = cursor.fetchone()
print(tab1)
I want to create a dynamic Raw Queryset.
By default a raw queryset looks like:
qs = self.raw(
'SELECT DISTINCT ON(P.slug) P.id, P.slug, P.company_id, I.image,
'FROM product AS P ' ....
It is possible to create the queryset using conditions. I'm not referring to database conditions like Where, but python conditions like if
Something like:
'SELECT DISTINCT ON(P.slug) P.id, P.slug, P.company_id, I.image,
'FROM product AS P ' # if var_a Limit 20 else where P.id = var_b
Just use simple python if/else:
query = 'SELECT DISTINCT ON(P.slug) P.id, P.slug, P.company_id, I.image FROM product AS P '
params = []
if var_a:
query += 'LIMIT 20'
else:
query += 'WHERE P.id = %s'
params.append(var_b)
qs = self.raw(query, params)
Here is my script that ran successfully, but still i cannot see the table being created in SQL Server's database.
import _mssql
conn = _mssql.connect(server='MAQSOOD-PC', user='sa', password='123', \
database='TestDB')
conn.execute_non_query('CREATE TABLE persons(id INT, name VARCHAR(100))')
conn.execute_non_query("INSERT INTO persons VALUES(1, 'John Doe')")
conn.execute_non_query("INSERT INTO persons VALUES(2, 'Jane Doe')")
What could be the issue?
I believe the issue is that you need to commit the transaction.
Try this version instead:
import pymssql
conn = pymssql.connect('MAQSOOD-PC', 'sa', '123', 'tempdb')
cur = conn.cursor()
cur.execute('CREATE TABLE persons(id INT, name VARCHAR(100))')
cur.commit()
cur.execute("INSERT INTO persons VALUES (1, 'John Doe')")
cur.execute("INSERT INTO persons VALUES (2, 'Jane Doe')")
cur.commit()
cur.execute('SELECT * FROM persons')
for row in cur:
print(row)
I have been working on an application in Django. To begin with, for simplicity, I had been using sqlite3 for the database.
However, once I moved to PostgreSQL, I've run into a bit of a problem: the primary key does not reset once I clear out a table.
This app is a game that is played over a long time period (weeks). As such, every time a new game starts, all of the data is cleared out of the database and then new, randomized data is added.
I'd like to be able to "start over" with primary keys starting at 1 each time I clean/rebuild the game.
The code still works as-is, but integers are a pretty natural way for describing the objects in my game. I'd like to have each new game start at 1 rather than wherever the last game left off.
How can I reset the primary key counter in PostgreSQL? Keep in mind that I don't need to preserve the data in the table since I am wiping it out anyway.
In your app directory try this:
python manage.py help sqlsequencereset
Pipe it into psql like this to actually run the reset:
python manage.py sqlsequencereset myapp1 myapp2 | psql
Edit: here's an example of the output from this command on one of my tables:
BEGIN;
SELECT setval('"project_row_id_seq"', coalesce(max("id"), 1), max("id") IS NOT null) FROM "project_row";
COMMIT;
As suggested by "Van Gale" you can get the commands to solve your problem running sqlsequencereset.
or
You can execute the SQL query generated by sqlsequencereset from within python in this way (using the default database):
from django.core.management.color import no_style
from django.db import connection
from myapps.models import MyModel1, MyModel2
sequence_sql = connection.ops.sequence_reset_sql(no_style(), [MyModel1, MyModel2])
with connection.cursor() as cursor:
for sql in sequence_sql:
cursor.execute(sql)
I tested this code with Python3.6, Django 2.0 and PostgreSQL 10.
If you perform a raw sql, can do this:
ALTER SEQUENCE youApp_id_seq RESTART WITH 1;
docs:
http://www.postgresql.org/docs/8.2/static/sql-altersequence.html
I view auto-increment primary keys as purely internal identifiers for database records, and I don't like exposing them to users. Granted, it's a common design to use them as part of URLs, but even there slugs or other identifiers feel more appropriate.
If you do not want to have to manually grab the apps you need, or if you have a series of different databases, this command will dynamically gather all connections from settings.py and reset the sequence.
To run use: python manage.py reset_sequences
import psycopg2
from django.conf import settings
from django.core.management.base import BaseCommand
from django.db import connections
def dictfetchall(cursor):
"""Return all rows from a cursor as a dict"""
columns = [col[0] for col in cursor.description]
return [
dict(zip(columns, row))
for row in cursor.fetchall()
]
class Command(BaseCommand):
help = "Resets sequencing errors in Postgres which normally occur due to importing/restoring a DB"
def handle(self, *args, **options):
# loop over all databases in system to figure out the tables that need to be reset
for name_to_use_for_connection, connection_settings in settings.DATABASES.items():
db_name = connection_settings['NAME']
host = connection_settings['HOST']
user = connection_settings['USER']
port = connection_settings['PORT']
password = connection_settings['PASSWORD']
# connect to this specific DB
conn_str = f"host={host} port={port} user={user} password={password}"
conn = psycopg2.connect(conn_str)
conn.autocommit = True
select_all_table_statement = f"""SELECT *
FROM information_schema.tables
WHERE table_schema = 'public'
ORDER BY table_name;
"""
# just a visual representation of where we are
print('-' * 20, db_name)
try:
not_reset_tables = list()
# use the specific name for the DB
with connections[name_to_use_for_connection].cursor() as cursor:
# using the current db as the cursor connection
cursor.execute(select_all_table_statement)
rows = dictfetchall(cursor)
# will loop over table names in the connected DB
for row in rows:
find_pk_statement = f"""
SELECT k.COLUMN_NAME
FROM information_schema.table_constraints t
LEFT JOIN information_schema.key_column_usage k
USING(constraint_name,table_schema,table_name)
WHERE t.constraint_type='PRIMARY KEY'
AND t.table_name='{row['table_name']}';
"""
cursor.execute(find_pk_statement)
pk_column_names = dictfetchall(cursor)
for pk_dict in pk_column_names:
column_name = pk_dict['column_name']
# time to build the reset sequence command for each table
# taken from django: https://docs.djangoproject.com/en/3.0/ref/django-admin/#sqlsequencereset
# example: SELECT setval(pg_get_serial_sequence('"[TABLE]"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "[TABLE]";
try:
reset_statement = f"""SELECT setval(pg_get_serial_sequence('"{row['table_name']}"','{column_name}'),
coalesce(max("{column_name}"), 1), max("{column_name}") IS NOT null) FROM "{row['table_name']}" """
cursor.execute(reset_statement)
return_values = dictfetchall(cursor)
# will be 1 row
for value in return_values:
print(f"Sequence reset to {value['setval']} for {row['table_name']}")
except Exception as ex:
# will only fail if PK is not an integer...
# currently in my system this is from django.contrib.sessions
not_reset_tables.append(f"{row['table_name']} not reset")
except psycopg2.Error as ex:
raise SystemExit(f'Error: {ex}')
conn.close()
print('-' * 5, ' ALL ERRORS ', '-' * 5)
for item_statement in not_reset_tables:
# shows which tables produced errors, so far I have only
# seen this with PK's that are not integers because of the MAX() method
print(item_statement)
# just a visual representation of where we are
print('-' * 20, db_name)
You need to truncate the table.
See http://www.postgresql.org/docs/8.1/static/sql-truncate.html