Django ORM raw delete query not deleting records - django

I am using raw_sql queries for my convenience for keeping my database minimal I am deleting extra records. By this query
#d is from a loop and has values
res=MyModel.objects.raw("DELETE FROM mydb_mymodel WHERE mydb_mymodel.s_type = '%s' and mydb_mymodel.barcode = '%s' and mydb_mymodel.shopcode = '%s' and mydb_mymodel.date = '%s'" ,[d.s_type,d.barcode,d.shopcode,d.date])
It is not deleting records in database but
when I do res.query and run it from postgres console it works!
Yes I can use
MyModel.objects.filter(s_type=d.s_type,barcode=d.barcode,
shopcode=d.shopcode,date=d.date).delete()
but what I am missing in raw_sql?

A .raw(..) is not executed eagerly, it is, just like most Django ORM queries performed lazily. It thus returns a RawQuerySet object with the query in the object. For example:
>>> User.objects.raw('BLA BLA BLA', [])
<RawQuerySet: BLA BLA BLA>
A query like BLA BLA BLA does not make any sense: a database will error on it, but still we retrieve a RawQuerySet.
You can force evaluation by for example iterating over it, and then we get:
>>> list(User.objects.raw('BLA BLA BLA', []))
Traceback (most recent call last):
File "/djangotest/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/djangotest/env/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/djangotest/env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/djangotest/env/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/djangotest/env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/djangotest/env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 412, in _query
rowcount = self._do_query(q)
File "/djangotest/env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 375, in _do_query
db.query(q)
File "/djangotest/env/lib/python3.6/site-packages/MySQLdb/connections.py", line 276, in query
_mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'BLA BLA BLA' at line 1")
So the list(..) forces evaluation, and now the database of course produces an error. However even if it was a valid DELETE query, it would still raise an error, since such query does not return any record.
In order to make DELETE calls, the Django manual specifies that you should use a cursor [Django-doc]:
from django.db import connection
with connection.cursor() as cursor:
cursor.execute(
"DELETE FROM mydb_mymodel WHERE s_type = '%s' AND barcode = '%s' AND shopcode = '%s' AND date = '%s'" ,
[d.s_type,d.barcode,d.shopcode,d.date]
)
But I think it is probably a lot simpler to specify it like:
MyModel.objects.filter(
s_type=d.s_type,
barcode=d.barcode,
shopcode=d.shopcode,
date=d.date
).delete()
This will construct a DELETE query, and serialize the parameters properly. A .delete() query is done eagerly, so the odds of making above discussed mistakes is a lot lower: if the ORM is implemented correctly, then we do not need to worry about that.

try this:
from django.db import connection
def executeQuery(self, sql, params=[]):
with connection.cursor() as cursor:
cursor.execute(
sql,
params
)
use it like:
executeQuery("DELETE FROM mydb_mymodel WHERE mydb_mymodel.s_type = '%s' and mydb_mymodel.barcode = '%s' and mydb_mymodel.shopcode = '%s' and mydb_mymodel.date = '%s'" ,[d.s_type,d.barcode,d.shopcode,d.date])

Related

Is there a workaround for Django's `ModelDoesNotExist` error in a situation like this?

So I get a ModelDoesNotExist error when I run manage.py runserver because in a file in my directory, I make a query to a table which has not been populated yet.
def __init__(self):
self.spotify_object = SocialToken.objects.get(account__provider="spotify")
The above is a class instantiated to perform some sort of authentication and the SocialToken table gets populated only after I login. Now, I was wondering if there was a way to escape the error by triggering this part of the code only after I login? I only use the class in an endpoint, and during that period, the table would have been populated but the fact that it is not populated before running the server is causing a DoesNotExist error. Is there a solution to this?
Traceback
File "C:\Users\Kwaku Biney\Desktop\sparison-1\project\Sparison\views.py", line 4, in <module>
from .authentication import SparisonCacheHandler
File "C:\Users\Kwaku Biney\Desktop\sparison-1\project\Sparison\authentication.py", line 43, in
<module>
cache_handler = SparisonCacheHandler() ,
File "C:\Users\Kwaku Biney\Desktop\sparison-1\project\Sparison\authentication.py", line 25,
in __init__
self.spotify_object = SocialToken.objects.get(account__provider="spotify")
File "C:\Users\Kwaku Biney\Desktop\sparison-1\project\venv\lib\site-
packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\Kwaku Biney\Desktop\sparison-1\project\venv\lib\site-
packages\django\db\models\query.py", line 429, in get
raise self.model.DoesNotExist(
allauth.socialaccount.models.DoesNotExist: SocialToken matching query does not exist.
In my views.py, I import the class which has the query and the error comes up.
There are two ways to avoid the DoesNotExist Error.
A) Use .filter() instead of .get()
Filtering leads to an empty queryset when the search comes up empty.
def __init__(self):
self.spotify_object = SocialToken.objects.filter(account__provider="spotify")
B) Use get_object_or_404() instead of .get()
This is a built-in function by django:
Calls get() on a given model manager, but it raises Http404 instead of the model’s DoesNotExist exception.
Django Documentation
def __init__(self):
self.spotify_object = SocialToken.objects.get_object_or_404(account__provider="spotify")
Hope I could help you. Have a nice day.

Log warning from Selenium on Django [duplicate]

Whenever I try to construct a string based on self.live_server_url, I get python TypeError messages. For example, I've tried the following string constructions (form 1 & 2 below), but I experience the same TypeError. My desired string is the Live Server URL with "/lists" appended. NOTE: the actual test does succeed to create a server and I can manually access the server, and more specifically, I can manually access the exact URL that I'm trying to build programmatically (e.g. 'http://localhost:8081/lists').
TypeErrors occur with these string constructions.
# FORM 1
lists_live_server_url = '%s%s' % (self.live_server_url, '/lists')
# FORM 2
lists_live_server_url = '{0}{1}'.format(self.live_server_url, '/lists')
self.browser.get(lists_live_server_url)
There is no python error with this form (nothing appended to string), albeit my test fails (as I would expect since it isn't accessing /lists).
self.browser.get(self.live_server_url)
Here is the python error that I'm getting.
/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4 /Applications/PyCharm.app/Contents/helpers/pycharm/django_test_manage.py test functional_tests.lists_tests.LiveNewVisitorTest.test_can_start_a_list_and_retrieve_it_later /Users/myusername/PycharmProjects/mysite_proj
Testing started at 11:55 AM ...
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/usr/local/lib/python3.4/site-packages/django/test/testcases.py", line 1104, in __call__
return super(FSFilesHandler, self).__call__(environ, start_response)
File "/usr/local/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
response = self.get_response(request)
File "/usr/local/lib/python3.4/site-packages/django/test/testcases.py", line 1087, in get_response
return self.serve(request)
File "/usr/local/lib/python3.4/site-packages/django/test/testcases.py", line 1099, in serve
return serve(request, final_rel_path, document_root=self.get_base_dir())
File "/usr/local/lib/python3.4/site-packages/django/views/static.py", line 54, in serve
fullpath = os.path.join(document_root, newpath)
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/posixpath.py", line 82, in join
path += b
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'
Am I unknowingly attempting to modify the live_server_url, which is leading to these TypeErrors? How could I programmatically build a string of live_server_url + "/lists"?
Here is the test that I am attempting...
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from django.test import LiveServerTestCase
class LiveNewVisitorTest(LiveServerTestCase):
def setUp(self):
self.browser = webdriver.Chrome()
self.browser.implicitly_wait(3)
def tearDown(self):
self.browser.close()
def test_can_start_a_list_and_retrieve_it_later(self):
#self.browser.get('http://localhost:8000/lists')
#self.browser.get('http://www.google.com')
#lists_live_server_url = '%s%s' % (self.live_server_url, '/lists')
#lists_live_server_url = '{0}{1}'.format(self.live_server_url, '/lists')
lists_live_server_url = self.live_server_url
self.browser.get(lists_live_server_url)
self.assertIn('To-Do', self.browser.title)
header_text = self.browser.find_element_by_tag_name('h1').text
self.assertIn('To-Do', header_text)
See this discussion on Reddit featuring the same error Traceback.
Basically, this is not a problem with anything within the Selenium tests but rather with your project's static file configuration.
From your question, I believe the key line within the Traceback is:
File "/usr/local/lib/python3.4/site-packages/django/views/static.py", line 54, in serve
fullpath = os.path.join(document_root, newpath)
This line indicates that an unsuccessful os.path.join is being attempted within django.views.static.
Set STATIC_ROOT in your project's settings.pyfile and you should be good.
Use StaticLiveServerTestCase instead may help

Flask SQLAlchemy Transaction error when using server_default in model in models_committed signal handler

My model (before) contains client-side defaults:
created_ts = db.Column(db.DateTime(timezone=True), default=dt.datetime.now)
My model (after) now contains server-side defaults:
created_ts = db.Column(db.DateTime(timezone=True), server_default=text('NOW()'))
However, I now start seeing the error:
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
In my models_committed hook:
#models_committed.connect_via(app)
def handle(sender, changes):
for model, operation in changes:
model.to_dict() # error here
I stole to_dict from flask_sandboy:
def to_dict(self):
"""Return dict representation of class by iterating over database
columns."""
value = {}
for column in self.__table__.columns:
attribute = getattr(self, column.name) # error here
if isinstance(attribute, datetime.datetime):
attribute = str(attribute)
value[column.name] = attribute
return value
So, getattr(self, column.name) seems to trigger the server-side default somehow (presumably, since that's the change I introduced).
From this line in my own code, I provide the rest of the stack trace:
File "/code/models/session.py", line 20, in to_dict
attribute = getattr(self, column.name)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 589, in get
value = callable_(state, passive)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 433, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 613, in load_scalar_attributes
only_load_props=attribute_names)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 235, in load_on_ident
return q.one()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2398, in one
ret = list(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2441, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2454, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2445, in _connection_from_session
**kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 880, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 885, in _connection_for_bind
engine, execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 305, in _connection_for_bind
self._assert_active()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 196, in _assert_active
"This session is in 'committed' state; no further "
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
How do I get around this problem?
pip freeze:
aniso8601==0.92
blinker==1.3
boto==2.36.0
Flask==0.10.1
Flask-Cors==1.10.3
Flask-HTTPAuth==2.4.0
Flask-SQLAlchemy==2.0
gunicorn==19.3.0
itsdangerous==0.24
psycopg2==2.6
pytz==2014.10
six==1.9.0
SQLAlchemy==0.9.9
Werkzeug==0.10.1
TL;DR solution #4 below works
The created_ts field is not known in that request thread, nor in the subsequent signal handler, since the value is chosen server-side by PostgresQL.
getattr(self, column.name) then tries to hydratepopulate that value by going back to the DB server, but unfortunately is now outside of a transaction.
A few options here:
Start a new session in the signal handler so as to retrieve the value from the DB (this is an untested suggestion)
Use the built-in _dict to get the model's state, but it won't have the created_ts field, because it is not known at this point:
Code
modeldict = dict(model.__dict__)
modeldict.pop('_sa_instance_state', None)
Stick with default=dt.datetime.now, but non-Python apps which write to the database need to provide created_ts.
So, the trade-off is whether the consumer of the model (as alluded to with the pika comment) needs created_ts or not.
(tested to work) Thanks to a combination of advice from agronholm on the #sqlalchemy IRC room and kbussel on this ticket, I tried to serialize the data while the session is still open, and when I'm sure the data is committed, I re-use the serialized data and by-pass going back to the DB:
from sqlalchemy.event import listens_for
from flask.ext.sqlalchemy import SignallingSession
#listens_for(SignallingSession, 'after_flush')
def after_flush_handler(session, tx):
try:
d = session._model_changes
except AttributeError:
return
if d:
changes = []
for model, operation in list(d.values()):
changes.append((model.to_dict(), operation))
session.info['my_changes'] = changes
d.clear()
#listens_for(SignallingSession, 'after_commit')
def after_commit_handler(session):
if 'my_changes' in session.info:
changes = session.info['my_changes']
for model, operation in changes:
# use model here, with all data populated

Unable to delete heroku table through django shell

I'm trying to delete a table hosted in heroku. Because it is in the shared database I can't connect to postgres directly to perform sentence. I execute delete through django shell. Issue is that table name is in uppercase and, for unknow reason, django try to delete it in lower cases:
>>> sql = '''drop table campings_horesEntrega cascade; <---see uppercases
... drop table campings_horesRecollida; '''
>>>
>>> from django.db import connections, transaction
>>> cursor = connections['default'].cursor()
>>> cursor.execute(sql)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 52, in execute
return self.cursor.execute(query, args)
DatabaseError: table "campings_horesentrega" does not exist <--- see lowercases
someone has a workaround to my issue?
thanks a lot.
You need to wrap your table names in quotes if you want postgres to respect case.
DROP TABLE "campings_horesEntrega" CASCADE;

Django: Why don't foreign key lookups automatically use the pk?

I have
class Achievement(MyBaseModel):
pass
class Alias(MyBaseModel):
achievements = models.ManyToManyField('Achievement')
>>> ach = Achievement.objects.all()[1]
This works :
>>> Alias.objects.all().filter(achievements__pk__contains=ach.pk).count()
77L
But this doesn't :
>>> Alias.objects.all().filter(achievements__contains=ach).count()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/var/home/ptarjan/django/mysite/django/db/models/query.py", line 489, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/var/home/ptarjan/django/mysite/django/db/models/query.py", line 507, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/var/home/ptarjan/django/mysite/django/db/models/sql/query.py", line 1258, in add_q
can_reuse=used_aliases)
File "/var/home/ptarjan/django/mysite/django/db/models/sql/query.py", line 1201, in add_filter
self.where.add((alias, col, field, lookup_type, value), connector)
File "/var/home/ptarjan/django/mysite/django/db/models/sql/where.py", line 48, in add
params = field.get_db_prep_lookup(lookup_type, value)
File "/var/home/ptarjan/django/mysite/django/db/models/fields/related.py", line 156, in get_db_prep_lookup
raise TypeError, "Related Field has invalid lookup: %s" % lookup_type
TypeError: Related Field has invalid lookup: contains
Why? (Django 1.0.2)
Looking at the query log, it is doing something that I didn't expect! That query yielded :
>>> connection.queries[-1]
{'time': '0.027', 'sql': u'SELECT COUNT(*) FROM `yourock_alias` INNER JOIN `yourock_achiever` ON (`yourock_alias`.`id` = `yourock_achiever`.`alias_id`) WHERE `yourock_achiever`.`achievement_id` LIKE BINARY %j0xvw9% '}
But doing this
>>> Alias.objects.all().filter(achievements=ach).count()
77L
Gives this query
>>> connection.queries[-1]
{'time': '0.023', 'sql': u'SELECT COUNT(*) FROM `yourock_alias` INNER JOIN `yourock_achiever` ON (`yourock_alias`.`id` = `yourock_achiever`.`alias_id`) WHERE `yourock_achiever`.`achievement_id` = j0xvw9 '}
which is what I wanted, but the = seems to me to mean that it IS the one object. The query that django is doing actually is returning if the object is anywhere in the achievement list.
Is this correctly setup and is just very counter-intuitive or am I doing something wrong?
I can't answer why the design decision was made to implement it this way, but most likely its to follow the Python philosophy that things should be specified explicitly and not implied.
The reason it doesn't work is because __contains expects a field to reference. The bit you passed it was a reference to a whole object.
In the second case, you are comparing objects. And, from Django documentation, in order to compare an object you have to use the == operator.
On top of that: why don't you use ach.alias_set.objects.count(), as explained in the query section of the manual?