Django not showing all mysql database results despite using fetchall method - django

I have ten records in mysql database and am using fetchall() method
Now I have requirements to display all database result in json using sql queries in django.
When I run the code below, it only shows the first records while the rest is not displayed.
I was wondering why am getting just only one json record despite using fetchall() approach
Here is the code
from django.db import connection
def read(request):
sql = 'SELECT * from crud_posts'
with connection.cursor() as cursor:
cursor.execute(sql)
output = cursor.fetchall()
print(output[0])
items=[]
for row in output:
items.append({'id':row[0], 'title': row[1],'content': row[2]})
jsondata = json.dumps({'items': items})
return HttpResponse(jsondata, content_type='application/json')

You are exiting the for loop after the first iteration...fix your identation:
from django.db import connection
def read(request):
sql = 'SELECT * from crud_posts'
with connection.cursor() as cursor:
cursor.execute(sql)
output = cursor.fetchall()
print(output[0])
items=[]
for row in output:
items.append({'id':row[0], 'title': row[1],'content': row[2]})
jsondata = json.dumps({'items': items})
return HttpResponse(jsondata, content_type='application/json')

Related

How do I run a SQL query via REST against QuestDB?

I'm using Python for some tests and I would like to be able to run a SQL query via REST. Is there an easy way to use requests to run queries like:
requests.get('http:myserver:9000/exec' query="select * from my_table")
If you need to use REST via Python, this can be done similar to the following example:
import requests
import json
host = 'http://myserver:9000'
sql_query = "select * from my_table limit 100"
query_params = {'query': sql_query}
try:
response = requests.post(host + '/exec', params=query_params)
json_response = json.loads(response.text)
rows = json_response['dataset']
for row in rows:
print(row)
except requests.exceptions.RequestException as e:
print("Error: %s" % (e))
There is additional documentation for this on the QuestDB REST docs page

How to see what rows were added with Django connection.cursor.execute()?

I have this python code ...
sql = """
INSERT INTO app_table(...cols..)
SELECT DISTINCT ... cols ... FROM app_temptable e
WHERE something = 'Generic' AND NOT EXISTS
(SELECT id FROM app_table u WHERE e.somethingelse = u.somethingelse);
"""
try:
with connection.cursor() as cursor:
cursor.execute(sql)
log.info( '%s rows inserted.' % cursor.rowcount)
for row in cursor.fetchall():
print(row)
except Exception as ex:
err = ex
log.error("sql: %s" % sql)
raise err
... it prints the row count and then throws the exception:
django.db.utils.InterfaceError: not a query
I guess because I wasn't doing a select. But the Django documentation (https://docs.djangoproject.com/en/2.2/topics/db/sql/) says to get the affected rows call
cursor.fetchall().
So how do I get the affected/rows inserted?
I am using a Oracle database.
It says it here:

How to correctly execute query in Django?

I need to execute the SELECT pg_database_size ('mydatabase') query, how do I do this in Django with the Postgres DBMS?
I already tried doing the following
from django.db import connection
cursor = connection.cursor()
size = cursor.execute('''SELECT pg_database_size("mydatabase")''')
But the resulting size is None.
How do I perform this query? The intent is to return the size of the database.
That's almost right, but note that while cursor.execute() will execute the SQL, cursor.fetchall() is needed to return the result. More info in the docs.
Try doing the following:
from django.db import connection
with connection.cursor() as cursor:
cursor.execute('SELECT pg_database_size("mydatabase")')
size = cursor.fetchall()

retrieving data from postgresql database into a dictionary in django

I am trying to return data that was retrieved from the database into a dictionary
view.py
from django.db import connection
def custom_query(query):
cursor = connection.cursor()
cursor.execute(query)
row = cursor.fetchall()
return row
when I try and change fetchall() to dictfetchall() it says 'psycopg2.extensions.cursor' object has no attribute 'dictfetchall'
I have tried also to add an argument in the cursor method cursor_factory=psycopg2.extras.DictCursor it says cursor() got an unexpected keyword argument 'cursor_factory'
In recent versions you can use NamedTupleCursor, DictCursor, or RealDictCursor:
Import:
from psycopg2.extras import NamedTupleCursor
As the default cursor on the connection:
psycopg2.connect(dsn, cursor_factory=NamedTupleCursor)
Or per query:
with connection.cursor(cursor_factory=NamedTupleCursor) as cursor:
cursor.execute('select * from tbl')
for row in cursor.fetchall():
print(row.col1, row.col2)

java.sql.SQLExceptionPyRaisable on the second attempt connecting to Athena using Django

I am using the python module called PyAthenaJDBC in order to query Athena using the provided JDBC driver.
Here is the link : https://pypi.python.org/pypi/PyAthenaJDBC/
I have been facing some persistent issue. I keep getting this java error whenever I use the Athena connection twice in a row.
As a matter of fact, I was able to connect to Athena, show databases, create new tables and even query the content. I am building an application using Django and running its server to use Athena
However, I am obliged to re-run the server in order for the Athena connection to work once again,
Here is a glimpse of the class I have built
import os
import configparser
import pyathenajdbc
#Get aws credentials for the moment
aws_config_file = '~/.aws/config'
Config = configparser.ConfigParser()
Config.read(os.path.expanduser(aws_config_file))
access_key_id = Config['default']['aws_access_key_id']
secret_key_id = Config['default']['aws_secret_access_key']
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
athena_jdbc_driver_path = BASE_DIR + "/lib/static/AthenaJDBC.jar"
log_path = BASE_DIR + "/lib/static/queries.log"
class PyAthenaLoader():
def __init__(self):
pyathenajdbc.ATHENA_JAR = athena_jdbc_driver_path
def connecti(self):
self.conn = pyathenajdbc.connect(
s3_staging_dir="s3://aws-athena-query-results--us-west-2",
access_key=access_key_id,
secret_key=secret_key_id,
#profile_name = "default",
#credential_file = aws_config_file,
region_name="us-west-2",
log_path=log_path,
driver_path=athena_jdbc_driver_path
)
def databases(self):
dbs = self.query("show databases;")
return dbs
def tables(self, database):
tables = self.query("show tables in {0};".format(database))
return tables
def create(self):
self.connecti()
try:
with self.conn.cursor() as cursor:
cursor.execute(
"""CREATE EXTERNAL TABLE IF NOT EXISTS sales4 (
Day_ID date,
Product_Id string,
Store_Id string,
Sales_Units int,
Sales_Cost float,
Currency string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '|',
'field.delim' = '|',
'collection.delimm' = 'undefined',
'mapkey.delim' = 'undefined'
) LOCATION 's3://athena-internship/';
""")
res = cursor.description
finally:
self.conn.close()
return res
def query(self, req):
self.connecti()
try:
with self.conn.cursor() as cursor:
cursor.execute(req)
print(cursor.description)
res = cursor.fetchall()
finally:
self.conn.close()
return res
def info(self):
res = []
for i in dir(pyathenajdbc):
temp = i + ' = ' + str(dic[i])
#print(temp)
res.append(temp)
return res
Example of usage :
def test(request):
athena = jdbc.PyAthenaLoader()
res = athena.query('Select * from sales;')
return render(request, 'test.html', {'data': res})
Works just fine!
However refreshing the page would cause this error :
Error
Note that I am using a local .jar file: I thought that would solve the issue but I was wrong
Even if I remove the path of the JDBC driver and let the module download it from s3, the error persists:
File "/home/tewfikghariani/.virtualenvs/venv/lib/python3.4/site-packages/pyathenajdbc/connection.py", line 69, in init
ATHENA_CONNECTION_STRING.format(region=self.region_name, schema=schema_name), props)
jpype._jexception.java.sql.SQLExceptionPyRaisable:
java.sql.SQLException: No suitable driver found for
jdbc:awsathena://athena.us-west-2.amazonaws.com:443/hive/default/
Furthermore, when I run the module on its own, it works just fine.
When I set multiple connection inside my view before rendering the template, that works just fine as well.
I guess the issue is related to the django view, once one of the views is performing a connection with athena, the next connection is not possible anymore and the error is raised unless I restart the server
Any help? If other details are missing I will provide them immediately.
Update:
After posting the issue in github, the author solved this problem and released a new version that works perfectly.
It was a multi-threading problem with JPype.
Question answered!
ref : https://github.com/laughingman7743/PyAthenaJDBC/pull/8