PostgreSQL is able to have .distint('field name') database queries, however Sqlite isn't, so I created a try/except block which should run a more simple query if the user is using sqlite3.
try:
qs = qs.filter(tag__istartswith=self.q).order_by('tag').distinct('tag')
except NotImplementedError:
qs = qs.filter(tag__istartswith=self.q)
So if the user is using Sqlite I would expect the simple query in the except block to get executed, however the exception is thrown and the simple query never gets executed:
raise NotImplementedError('DISTINCT ON fields is not supported by this database backend')
NotImplementedError: DISTINCT ON fields is not supported by this database backend
Do you have any idea why this isn't working as expected?
Thanks
Because querysets are lazy and the error is not raised while constructing the query, but when you actually evaluate it.
You can try forcing your query to be evaluated to catch the exception:
try:
qs = qs.filter(tag__istartswith=self.q).order_by('tag').distinct('tag')
dummy_boolean_var = qs.exists()
except NotImplementedError:
qs = qs.filter(tag__istartswith=self.q)
EDIT: Apparently my untested version did not work as the assignment in the except clause tries to modify the original query. This is the working version as tested by OP:
try:
qa = qs
qs = qs.filter(tag__istartswith=self.q).order_by('tag').distinct('tag')
dummy_boolean_var = qs.exists()
except NotImplementedError:
qs = qa
qs = qs.filter(tag__istartswith=self.q)
Related
I use django-import-export 2.8.0 with Oracle 12c.
Line-by-line import via import_data() works without problems, but when I turn on the use_bulk=True option, it stops importing and does not throw any errors.
Why does not it work?
resources.py
class ClientsResources(resources.ModelResource):
class Meta:
model = Clients
fields = ('id', 'name', 'surname', 'age', 'is_active')
batch_size = 1000
use_bulk = True
raise_errors = True
views.py
def import_data(request):
if request.method == 'POST':
file_format = request.POST['file-format']
new_employees = request.FILES['importData']
clients_resource = ClientsResources()
dataset = Dataset()
imported_data = dataset.load(new_employees.read().decode('utf-8'), format=file_format)
result = clients_resource.import_data(imported_data, dry_run=True, raise_errors=True)
if not result.has_errors():
clients_resource.import_data(imported_data, dry_run=False)
return HttpResponseRedirect(request.META.get('HTTP_REFERER'))
data.csv
id,name,surname,age,is_active
18,XSXQAMA,BEHKZFI,89,Y
19,DYKNLVE,ZVYDVCX,20,Y
20,GPYXUQE,BCSRUSA,73,Y
21,EFHOGJJ,MXTWVST,93,Y
22,OGRCEEQ,KJZVQEG,52,Y
--UPD--
I used django-debug-toolbar and saw a very strange behavior with import-queries.
With Admin Panel doesnt work. I see all importing rows, but next it writes "Import finished, with 5 new and 0 updated clients.", and see this strange queries
Then I use import by my form and here simultaneous situation:
use_bulk by django-import-export (more)
And for comparing my handle create_bulk()
--UPD2--
I've tried to trail import logic and look what I found:
import_export/resources.py
def bulk_create(self, using_transactions, dry_run, raise_errors, batch_size=None):
"""
Creates objects by calling ``bulk_create``.
"""
print(self.create_instances)
try:
if len(self.create_instances) > 0:
if not using_transactions and dry_run:
pass
else:
self._meta.model.objects.bulk_create(self.create_instances, batch_size=batch_size)
except Exception as e:
logger.exception(e)
if raise_errors:
raise e
finally:
self.create_instances.clear()
This print() showed empty list in value.
This issue appears to be due to a bug in the 2.x version of django-import-export. It is fixed in v3.
The bug is present when running in bulk mode (use_bulk=True)
The logic in save_instance() is finding that 'new' instances have pk values set, and are then incorrectly treating them as updates, not creates.
I cannot determine how this would happen. It's possible this is related to using Oracle (though I cannot see how).
I have a Django application that executes a full-text-search on a database. The view that executes this query is my search_view (I'm ommiting some parts for the sake of simplicity). It just retrieve the results of the search on my Post model and send to the template:
def search_view(request):
posts = m.Post.objects.all()
query = request.GET.get('q')
search_query = SearchQuery(query, config='english')
qs = Post.objects.annotate(
rank=SearchRank(F('vector_column'), search_query) + TrigramSimilarity('post_title', query)
).filter(rank__gte=0.15).order_by('-rank'), 15
)
context = {
results = qs
}
return render(request, 'core/search.html', context)
The application is working just fine. The problem is with a test I created. Here is my tests.py:
class SearchViewTests(TestCase):
def test_search_without_results(self):
"""
If the user's query did not retrieve anything
show him a message informing that
"""
response = self.client.get(reverse('core:search') + '?q=eksjeispowjskdjies')
self.assertEqual(response.status_code, 200)
self.assertContains(response.content, "We didn\'t find anything on our database. We\'re sorry")
This test raises an ProgrammingError exception:
django.db.utils.ProgrammingError: function similarity(character varying, unknown) does not exist
LINE 1: ...plainto_tsquery('english'::regconfig, 'eksjeispowjskdjies')) + SIMILARITY...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
I understand very well this exception, 'cause I got it sometimes. The SIMILARITY function in Postgres accepts two arguments, and both need to be of type TEXT. The exception is raising because the second argument (my query term) is of type UNKNOWN, therefore the function won't work and Django raises the exception. And I don't understand why, because the actual search is working! Even in the shell it works perfectly:
In [1]: from django.test import Client
In [2]: c = Client()
In [3]: response = c.get(reverse('core:search') + '?page=1&q=eksjeispowjskdjies')
In [4]: response
Out[4]: <HttpResponse status_code=200, "text/html; charset=utf-8">
Any ideas about why test doesn't work, but the actual execution of the app works and console test works too?
I had the same problem and this how I solved it in my case:
First of all, the problem was that when Django creates the test database that it is going to use for tests it does not actually run all of your migrations. It simply creates the tables based on your models.
This means that migrations that create some extension in your database, like pg_trgm do not run when creating the test database.
One way to overcome this is to use a fixture in your conftest.py file which will make create said extensions before any tests run.
So, in your conftest.py file add the following:
# the following fixture is used to add the pg_trgm extension to the test database
#pytest.fixture(scope="session", autouse=True)
def django_db_setup(django_db_setup, django_db_blocker):
"""Test session DB setup."""
with django_db_blocker.unblock():
with connection.cursor() as cursor:
cursor.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm;")
You can of course replace pg_trgm with any other extension you require.
PS: You must make sure the extension you are trying to use works for the test database you have chosen. In order to change the database used by Django you can change the value of
DATABASES = {'default': env.db('your_database_connection_uri')}
in your application's settings.py.
I would like to execute a transaction to delete some values, after that, count in db and if the result is < 1, rollback, i tried the following code:
#login_required
#csrf_exempt
#transaction.atomic
def update_user_groups(request):
if request.POST:
userId = request.POST['userId']
groups = request.POST.getlist('groups[]')
result = None
with transaction.atomic():
try:
GroupsUsers.objects.filter(user_id=int(userId)).delete()
for group in groups:
group_user = GroupsUsers()
group_user.user_id = userId
group_user.group_id = group
group_user.save()
count = UsersInAdmin.objects.all().count()
if count < 1:
transaction.commit()
else:
transaction.rollback()
except Exception, e:
result = e
return JsonResponse(result, safe=False)
Thanks,
It's not possible to manually commit or rollback the transaction inside an atomic block.
Instead, you can raise an exception inside the atomic block. If the block completes, the transaction will be committed. If you raise an exception, the transaction will be rolled back. Outside the atomic block, you can catch the exception and carry on your view.
try:
with transaction.atomic():
do_stuff()
if ok_to_commit():
pass
else:
raise ValueError()
except ValueError:
pass
I think you should give the Django documentation about db transactions another look.
The most relevant remark is the following:
Avoid catching exceptions inside atomic!
When exiting an atomic block, Django looks at whether it’s exited normally or with an exception to determine whether to commit or roll back. If you catch and handle exceptions inside an atomic block, you may hide from Django the fact that a problem has happened. This can result in unexpected behavior.
Adapting the example from the docs, it looks like you should nest your try and with blocks like this:
#transaction.atomic
def update_user_groups(request):
# set your variables
try:
with transaction.atomic():
# do your for loop
if count < 1:
# You should probably make a more descriptive, non-generic exception
# see http://stackoverflow.com/questions/1319615/proper-way-to-declare-custom-exceptions-in-modern-python for more info
raise Exception('Count is less than one')
except:
handle_exception()
One of the views that I call works with two database tables. It attempts to find an object in the first table. If the object isn't found, I get a Server Error (500). I'm not sure what the code would look like, but I want to insert some code into the view that will execute if the Server Error occurs so that I can tell it to try to find the object in the second table.
Current Code:
#csrf_exempt
#login_required
def addEvent(request):
event_id = request.POST['event_id']
user = request.POST['profile']
event = Event.objects.get(event_id = event_id)
if event.DoesNotExist:
event = customEvent.objects.get(event_id = event_id)
user = Profile.objects.get(id = user)
user.eventList.add(event)
return HttpResponse(status = 200)
Most likely you are getting 500 error because you are not finding a record in the first table. To fix that, you just have to catch the DoesNotExist Exception (Mentioned here):
try:
obj = FooModel.objects.get(...)
except FooModel.DoesNotExist:
try:
obj = OtherModel.objects.get(...)
except OtherModel.DoesNotExist:
raise Http404
or you can simplify this by using a shortcut:
try:
obj = FooModel.objects.get(...)
except FooModel.DoesNotExist:
obj = get_object_or_404(OtherModel, ...)
from django.core.exceptions import FieldError
#This is a method of a class
def _order_item_list(self, item_list, order_items_by, previous_order_by):
if order_items_by == previous_order_by:
order_items_by = '-' + order_items_by
try:
result = item_list.order_by(order_items_by)
except FieldError:
result = item_list
return result, order_items_by
Now when I order by valid fields following the generated link,everything works perfect. When I edit a link and add some dummy fieldnames for ordering, it should be catched by this exception and the original list should be returned. But it is not happening, instead I always get a FieldError from django.
FieldError at ...
Cannot resolve keyword u'fgsdffds' into field. Choices are: ...
The reason the exception is not caught is because the QuerySet has not been evaluated yet.
To validate an arbitrary (user-specified) value used for a model field or order_by value, simply check to see if that model has a field by that name.
For example, say you have a model called Ticket and an arbitrary GET parameter called field_name. Here's how you might handle creating a valid QuerySet in views.py:
from django.db.models import FieldDoesNotExist
from myapp.models import Ticket
def index(request):
default_field = 'id'
field_name = request.GET.get('field_name', default_field)
try:
Ticket._meta.get_field_by_name(field_name)
except FieldDoesNotExist:
field_name = default_field
tickets = Ticket.objects.all().order_by(field_name)
return ...
This means there's a typo, or the exception happens elsewhere. Insert a debug line:
import pdb; pdb.set_trace()
before the try-except and see how the code is executed. Try PUDB or IPDB debuggers instead of the standard one. Many questions disappear when you have a debugger and can see exactly what goes wrong.
I faced the same problem and surely it was because the exception is later. In order to raise exception in try-catch block I modified the code in following manner:
try:
result = item_list.order_by(order_items_by)
**result = list(result)**
except FieldError:
result = item_list
This worked for me.