I have ten records in mysql database and am using fetchall() method
Now I have requirements to display all database result in json using sql queries in django.
When I run the code below, it only shows the first records while the rest is not displayed.
I was wondering why am getting just only one json record despite using fetchall() approach
Here is the code
from django.db import connection
def read(request):
sql = 'SELECT * from crud_posts'
with connection.cursor() as cursor:
cursor.execute(sql)
output = cursor.fetchall()
print(output[0])
items=[]
for row in output:
items.append({'id':row[0], 'title': row[1],'content': row[2]})
jsondata = json.dumps({'items': items})
return HttpResponse(jsondata, content_type='application/json')
You are exiting the for loop after the first iteration...fix your identation:
from django.db import connection
def read(request):
sql = 'SELECT * from crud_posts'
with connection.cursor() as cursor:
cursor.execute(sql)
output = cursor.fetchall()
print(output[0])
items=[]
for row in output:
items.append({'id':row[0], 'title': row[1],'content': row[2]})
jsondata = json.dumps({'items': items})
return HttpResponse(jsondata, content_type='application/json')
I am currently trying to build an application that manages multiple databases. Since the app will be managing data in 30+ databases I am attempting to generate DATABASE_ROUTERS in the settings file. I cannot directly import the db model into the settings file. I get this error:
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
This error makes since. Is there a way I can control the sequence of events so that I have access to the database before all of the settings are established on execution? My goal is to automate database connections pulling relevant data from a DB and generate the DATABASE_ROUTERS and DATABASES within the setting file. Is this even possible? Is there a package that I can download that does exactly this?
If you do not know what I am asking please do not down vote just ask me to elaborate.
I was able to figure out how to query the data I needed from my database and import it into the settings file. I created the script below. Keep in mind this can be improved, this is just something I modified from here. This directly queries data from my test db (sqlite3). I use postgreSQL in production. This script should work, with some modification, with PostgreSQL.
As you can see below I am storing the data in dictionaries that is then stored in a list. I then import that list of dictionaries into my settings file. From there I can loop through the list and create my DATABASE_ROUTERS and DATABASES dynamically from the database. I was also able to generate router Classes in my routers.py file by importing the same list. Please comment below if you need me to elaborate further.
import sqlite3
from sqlite3 import Error
dbs = []
def create_connection(db_file):
""" create a database connection to the SQLite database
specified by the db_file
:param db_file: database file
:return: Connection object or None
"""
try:
conn = sqlite3.connect(db_file)
return conn
except Error as e:
print(e)
return None
def select_all_data(conn):
"""
Query all rows in the table
:param conn: the Connection object
:return:
"""
cur = conn.cursor()
cur.execute("SELECT * FROM fund_table")
rows = cur.fetchall()
for row in rows:
print(row)
def select_name_and_db(conn):
"""
Query table by fund_name and db_name
:param conn: the Connection object
:return:
"""
cur = conn.cursor()
cur.execute("SELECT fund_name, db_name FROM fund_table")
rows = cur.fetchall()
for row in rows:
dbs.append({"fund_name": row[0], "db_name": row[1]})
return dbs
def main():
database = "edb.sqlite3"
# create a database connection
conn = create_connection(database)
with conn:
""" select_all_data(conn) """
select_name_and_db(conn)
main()
Make one function that loads this variables and make it async, so after you app is ready you load it, but im not sure is this will work properly
https://hackernoon.com/asynchronous-python-45df84b82434
Dirty Sollution is make 1 file for each BD and you call your settings based in what BD gonna work...
I am trying to return data that was retrieved from the database into a dictionary
view.py
from django.db import connection
def custom_query(query):
cursor = connection.cursor()
cursor.execute(query)
row = cursor.fetchall()
return row
when I try and change fetchall() to dictfetchall() it says 'psycopg2.extensions.cursor' object has no attribute 'dictfetchall'
I have tried also to add an argument in the cursor method cursor_factory=psycopg2.extras.DictCursor it says cursor() got an unexpected keyword argument 'cursor_factory'
In recent versions you can use NamedTupleCursor, DictCursor, or RealDictCursor:
Import:
from psycopg2.extras import NamedTupleCursor
As the default cursor on the connection:
psycopg2.connect(dsn, cursor_factory=NamedTupleCursor)
Or per query:
with connection.cursor(cursor_factory=NamedTupleCursor) as cursor:
cursor.execute('select * from tbl')
for row in cursor.fetchall():
print(row.col1, row.col2)
I recently imported a lot of data from an old database into a new Postgresql database as the basis for models in a new Django site.
I used the IDs from the old database (as rows in various tables refer to each other), but they aren't all sequential - there are often large gaps.
I've noticed that when I add a new object via the Django app, then it has been using IDs starting from 1, which hasn't been a problem so far as there were no rows with very low IDs.
But once it reaches the first row of legacy data, then postgres obviously complains:
ERROR: duplicate key value violates unique constraint "django_comments_pkey"
DETAIL: Key (id)=(25) already exists.
Looking at the table descriptions I'm guessing I need to reset some kind of sequence on each table:
Table "public.django_comments"
Column | Type | Modifiers
-----------------+--------------------------+--------------------------------------------------------------
id | integer | not null default nextval('django_comments_id_seq'::regclass)
...
What do I need to do to reset that sequence, so that new rows are added with IDs higher than the current maximum ID?
Run sqlsequencereset and it'll print all the reset commands you need.
As suggested by "Dmitry Shevchenko" you can run sqlsequencereset to solve your problem.
or
You can execute the SQL query generated by sqlsequencereset from within python in this way (using the default database):
from django.core.management.color import no_style
from django.db import connection
from myapps.models import MyModel1, MyModel2
sequence_sql = connection.ops.sequence_reset_sql(no_style(), [MyModel1, MyModel2])
with connection.cursor() as cursor:
for sql in sequence_sql:
cursor.execute(sql)
I tested this code with Python3.6, Django 2.0 and PostgreSQL 10.
Here's a short snippet to reset all sequences in Django 1.9+ (based on http://djangosnippets.org/snippets/2774/) and compatible with Python 3:
import os
from io import StringIO
os.environ['DJANGO_COLORS'] = 'nocolor'
from django.core.management import call_command
from django.apps import apps
from django.db import connection
commands = StringIO()
cursor = connection.cursor()
for app in apps.get_app_configs():
label = app.label
call_command('sqlsequencereset', label, stdout=commands)
cursor.execute(commands.getvalue())
So the quickest, easiest and most "Django" way to do this in my opinion is to use the following management command:
python manage.py sqlsequencereset app_name
After this, you'll get something such as:
BEGIN;
SELECT setval(pg_get_serial_sequence('"measurements_quantity"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "measurements.Quantities";
SELECT setval(pg_get_serial_sequence('"measurements.Prefixes"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "measurements.Prefixes";
COMMIT;
The next step is to run this in the python manage.py dbshell management command, so run this and then you'll see the interaction database shell in your terminal:
psql (11.7 (Debian 11.7-0+deb10u1), server 11.5 (Debian 11.5-1.pgdg90+1))
Type "help" for help.
postgres=# BEGIN;
BEGIN
postgres=# SELECT setval(pg_get_serial_sequence('"measurements.Quantities"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "measurements.Quantities";
setval
--------
1
(1 row)
postgres=# SELECT setval(pg_get_serial_sequence('"measurements.Prefixes"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "measurements.Prefixes";
setval
--------
1
(1 row)
postgres=# COMMIT;
COMMIT
postgres=# exit
Simple as that. The python manage.py sqlsequencereset app_name command will give you the SQL you need to run, and you run it in the dbshell.
No writing your own custom SQL or custom code and it will give you what you need in the correct format and db engine of choice.
PostgreSQL Command:
ALTER SEQUENCE app_model_id_seq RESTART WITH 1
select setval('django_comments_id_seq', 12345);
This snippet Run sqlsequencereset on all apps reset all IDs of all Empty Models
Here is a more-or-less completely dynamic solution I just implemented in a management command that has no restriction as to the name of the Primary Key you are attempting to reset as it gathers it based on the connection params you have in settings.
The only sequencing I could not reset included PKs that are not integers, which is apparent in the PK for django.contrib.sessions, but again I have never run into sequencing errors with that so I doubt it is an issue.
Here is the command, run using python manage.py reset_sequences (obviously as long as your file/command is named reset_sequences.py)
import psycopg2
from django.conf import settings
from django.core.management.base import BaseCommand
from django.db import connections
def dictfetchall(cursor):
"""Return all rows from a cursor as a dict"""
columns = [col[0] for col in cursor.description]
return [
dict(zip(columns, row))
for row in cursor.fetchall()
]
class Command(BaseCommand):
help = "Resets sequencing errors in Postgres which normally occur due to importing/restoring a DB"
def handle(self, *args, **options):
# loop over all databases in system to figure out the tables that need to be reset
for name_to_use_for_connection, connection_settings in settings.DATABASES.items():
db_name = connection_settings['NAME']
host = connection_settings['HOST']
user = connection_settings['USER']
port = connection_settings['PORT']
password = connection_settings['PASSWORD']
# connect to this specific DB
conn_str = f"host={host} port={port} user={user} password={password}"
conn = psycopg2.connect(conn_str)
conn.autocommit = True
select_all_table_statement = f"""SELECT *
FROM information_schema.tables
WHERE table_schema = 'public'
ORDER BY table_name;
"""
# just a visual representation of where we are
print('-' * 20, db_name)
try:
not_reset_tables = list()
# use the specific name for the DB
with connections[name_to_use_for_connection].cursor() as cursor:
# using the current db as the cursor connection
cursor.execute(select_all_table_statement)
rows = dictfetchall(cursor)
# will loop over table names in the connected DB
for row in rows:
find_pk_statement = f"""
SELECT k.COLUMN_NAME
FROM information_schema.table_constraints t
LEFT JOIN information_schema.key_column_usage k
USING(constraint_name,table_schema,table_name)
WHERE t.constraint_type='PRIMARY KEY'
AND t.table_name='{row['table_name']}';
"""
cursor.execute(find_pk_statement)
pk_column_names = dictfetchall(cursor)
for pk_dict in pk_column_names:
column_name = pk_dict['column_name']
# time to build the reset sequence command for each table
# taken from django: https://docs.djangoproject.com/en/3.0/ref/django-admin/#sqlsequencereset
# example: SELECT setval(pg_get_serial_sequence('"[TABLE]"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "[TABLE]";
try:
reset_statement = f"""SELECT setval(pg_get_serial_sequence('"{row['table_name']}"','{column_name}'),
coalesce(max("{column_name}"), 1), max("{column_name}") IS NOT null) FROM "{row['table_name']}" """
cursor.execute(reset_statement)
return_values = dictfetchall(cursor)
# will be 1 row
for value in return_values:
print(f"Sequence reset to {value['setval']} for {row['table_name']}")
except Exception as ex:
# will only fail if PK is not an integer...
# currently in my system this is from django.contrib.sessions
not_reset_tables.append(f"{row['table_name']} not reset")
except psycopg2.Error as ex:
raise SystemExit(f'Error: {ex}')
conn.close()
print('-' * 5, ' ALL ERRORS ', '-' * 5)
for item_statement in not_reset_tables:
# shows which tables produced errors, so far I have only
# seen this with PK's that are not integers because of the MAX() method
print(item_statement)
# just a visual representation of where we are
print('-' * 20, db_name)
based on #Paolo Melchiorre I created a custom management command, which populates all the models from chosen apps.
from django.core.management.base import BaseCommand
from django.apps import apps
from django.core.management.color import no_style
from django.db import connection
class Command(BaseCommand):
def handle(self, *args, **kwargs):
self.stdout.write('Reset AutoFields ...')
APPS = ['app1', 'app2']
APPS = [apps.get_app_config(app) for app in APPS]
models = []
for app in APPS:
models.extend(list(app.get_models()))
sequence_sql = connection.ops.sequence_reset_sql(no_style(), models)
with connection.cursor() as cursor:
for sql in sequence_sql:
self.stdout.write(sql)
cursor.execute(sql)
self.stdout.write(self.style.SUCCESS('Reset AutoField complete.'))
tested using python 3.7 and django 2.2.
It seems like it should be easy to run "explain" directly off of a queryset in Django, but I don't see anything obvious for how to do it, and "explain" is a difficult thing to search for in the docs.
Well, there seems to be nothing out there except a toolbar so I wrote my own mixin to give me an explain() method on my querysets:
from django.db import connections
from django.db.models.query import QuerySet
class QuerySetExplainMixin:
def explain(self):
cursor = connections[self.db].cursor()
cursor.execute('explain %s' % str(self.query))
return cursor.fetchall()
QuerySet.__bases__ += (QuerySetExplainMixin,)
Hopefully this is useful to others.
QuerySet.explain(), available in Django 2.1.0 and above, is now the official way to explain queries.
Just a slight modification to guidoism's answer. This prevents getting a ProgrammingError: syntax error at or near ... error caused by the parameters not being correctly escaped in the raw query:
from django.db import connections
from django.db.models.query import QuerySet
class QuerySetExplainMixin:
def explain(self):
cursor = connections[self.db].cursor()
query, params = self.query.sql_with_params()
cursor.execute('explain %s' % query, params)
return '\n'.join(r[0] for r in cursor.fetchall())
QuerySet.__bases__ += (QuerySetExplainMixin,)
To use, simply invoke explain() at the end of your queryset, e.g.:
print SomeModel.objects.filter(...).explain()