Django 1.10.1
On a page I have prepared a lot of controls. Some of them are organized as dynamically changing formsets. So, I don't even know how many of them are present.
I need a chained filters with AND, OR and NOT logical operations.
For example, I need something like this:
Entry.objects.filter(headline__startswith='What').exclude(pub_date__gte=datetime.date.today()).filter(pub_date__gte=datetime(2005, 1, 30)).filter(Q(pub_date=date(2005, 5, 2)) | Q(pub_date=date(2005, 5, 6))
Once more the number of filters is changing.
I was planning to ac like that: loop through request.POST and depending on a dozen of conditions prepare a string. The same string:
"Entry.objects.filter(headline__startswith='What').exclude(pub_date__gte=datetime.date.today()).filter(pub_date__gte=datetime(2005, 1, 30)).filter(Q(pub_date=date(2005, 5, 2)) | Q(pub_date=date(2005, 5, 6))"
So, the string is correct. But I can't make it work with exec().
I asked here: why it is not working. And the answer was: it will not work, run the Python code directly.
I can construct something like that:
entries = Entry.objects.filter( **kwargs )
But this is just one filter. I can't imagine how to chain such filters.
Could you help me here&
You can chain a queryset on multiple lines - adding additional queries depending on view logic. These querysets are not executed as they are created, only when they are called.
I am not sure if it is the "best" or most pythonic way, but something like this works very well for me. This is using Django Rest Framework helper methods but it should be applicable without:
def get(self, request, format=None, *args, **kwargs):
name = request.GET.get('name', None)
location = request.GET.get('location', None)
id = request.GET.get('id', None)
results = Model.objects.all().order_by('-created') # this would be the default with no filters on it
results = results.filter(name__iexact=name) if name else results
results = results.filter(location__icontains=location) if location else results
results = results.filter(id__exact=id) if id else results
# process/serialize/send response
Not knowing which parameters you will get can be tricky, but an idea may be to loop over your kwargs and follow the same pattern in the loop - here's a stab (note I haven't tested it):
def get(self, request, format=None, *args, **kwargs):
results = Model.objects.all().order_by('-created') # this would be the default
for param, value in kwargs.items():
results = results.filter(***need to convert param to orm field***= value)
Related
New to django. I'm doing my best to implement CRUD using Django, mongodb, and mongoengine. I'm able to query the database and render my page with the correct information from the database. I'm also able to change some document fields using javascript and do an Ajax POST back to the original Django View class with the correct csrf token.
The data payload I'm sending back and forth is a list of each Document Model (VirtualPageModel) serialized to json (each element contains ObjectId string along with the other specific fields from the Model.)
This is where it starts getting murky. In order to update the original document in my View Class post function I do an additional query using the object id and loop through the dictionary items, setting the respective fields each time. I then call save and any new data is pushed to the Mongo collection correctly.
I'm not sure if what I'm doing to update existing documents is correct or in the spirit of django's abstracted database operations. The deeper I get the more I feel like I'm not using some fundamental facility earlier on (provided by either django or mongoengine) and because of this I'm having to make things up further downstream.
The way my code is now I would not be able to create a new document (although that's easy enough to fix). However what I'm really curious about is how I would know when to delete a document which existed in the initial query, but was removed by the user/javascript code? Am I overthinking things and the contents of my POST should contain a list of ObjectIds to delete (sounds like a security risk although this would be an internal tool.)
I was assuming that my View Class might maintain either the original document objects (or simply ObjectIds) it queried and I could do my comparisions off of that set, but I can't seem to get that information to persist (as a class variable in VolumeSplitterView) from its inception to when I received the POST at the end.
I would appreciate if anyone could take a look at my code. It really seems like the "ease of use" facilities of Django start to break when paired with Mongo and/or a sufficiently complex Model schema which needs to be directly available to javascript as opposed to simple Forms.
I was going to use this dev work to become django battle-hardened in order to tackle a future app which will be much more complicated and important. I can hack on this thing all day and make it functional, but what I'm really interested in is anyone's experience in using Django + MongoDB + MongoEngine to implement CRUD on a Database Schema which is not vary Form-centric (think more nested metadata).
Thanks.
model.py: uses mongoengine Field types.
class MongoEncoder(JSONEncoder):
def default(self, o):
if isinstance(o, VirtualPageModel):
data_dict = (o.to_mongo()).to_dict()
if isinstance(data_dict.get('_id'), ObjectId):
data_dict.update({'_id': str(data_dict.get('_id'))})
return data_dict
else:
return JSONEncoder.default(self, o)
class SubTypeModel(EmbeddedDocument):
filename = StringField(max_length=200, required=True)
page_num = IntField(required=True)
class VirtualPageModel(Document):
volume = StringField(max_length=200, required=True)
start_physical_page_num = IntField()
physical_pages = ListField(EmbeddedDocumentField(SubTypeModel),
default=list)
error_msg = ListField(StringField(),
default=list)
def save(self, *args, **kwargs):
print('In save: {}'.format(kwargs))
for k, v in kwargs.items():
if k == 'physical_pages':
self.physical_pages = []
for a_page in v:
tmp_pp = SubTypeModel()
for p_k, p_v in a_page.items():
setattr(tmp_pp, p_k, p_v)
self.physical_pages.append(tmp_pp)
else:
setattr(self, k, v)
return super(VirtualPageModel, self).save(*args, **kwargs)
views.py: My attempt at a view
class VolumeSplitterView(View):
#initial = {'key': 'value'}
template_name = 'click_model/index.html'
vol = None
start = 0
end = 20
def get(self, request, *args, **kwargs):
self.vol = self.kwargs.get('vol', None)
records = self.get_records()
records = records[self.start:self.end]
vp_json_list = []
img_filepaths = []
for vp in records:
vp_json = json.dumps(vp, cls=MongoEncoder)
vp_json_list.append(vp_json)
for pp in vp.physical_pages:
filepath = get_file_path(vp, pp.filename)
img_filepaths.append(filepath)
data_dict = {
'img_filepaths': img_filepaths,
'vp_json_list': vp_json_list
}
return render_to_response(self.template_name,
{'data_dict': data_dict},
RequestContext(request))
def get_records(self):
return VirtualPageModel.objects(volume=self.vol)
def post(self, request, *args, **kwargs):
if request.is_ajax:
vp_dict_list = json.loads(request.POST.get('data', []))
for vp_dict in vp_dict_list:
o_id = vp_dict.pop('_id')
original_doc = VirtualPageModel.objects.get(id=o_id)
try:
original_doc.save(**vp_dict)
except Exception:
print(traceback.format_exc())
I've written application in Django and tested it using SQLite. But now in production I want to use PostgreSQL and there's problem, because calling:
empty_list = []
Foo.objects.exclude(pk__in=empty_list).delete()
raises ProgrammingError (bad query - postgres doesn't accept in queries empty IN () )
I don't want to rewrite all code and wrap all excludes and filters in method that checks for empty list. I also don't want to write thousand of ifs or modify Django code. Is there any elegant solution to solve this?
OK, I found this: http://www.tryolabs.com/Blog/2013/07/05/run-time-method-patching-python/
It's simple, just write sth like this:
from django.db.models import QuerySet
def new_exclude(self, *args, **kwargs):
kw = dict()
for key, value in kwargs.items():
if is_not_empty_list(value) and is_not_empty_queryset(value): #other checks?
kw[key] = value
if len(kw):
return old_exclude(self, *args, **kw)
else:
return self
old_exclude = QuerySet.exclude
QuerySet.exclude = new_exclude
But personally I don't like this solution, so I won't accept it and I'm waiting for better one.
I've got a line in python:
streams = Stream.objects.filter(info__isnull = False)\
.order_by('-score', 'info__title')
I feed that to a RequestContext
c = RequestContext(request, {
'online_streams': streams.filter(online = True),
'offline_streams': streams.filter(online = False)
})
It sorts a table of ~50 rows by their score and title. This is good, but there is one row I would like to always be top. Simply fetching it beforehand and filtering it out of the line above, then give it to the RequestContext separately won't work, since I'd have to modify the template. Also I might change which row should be on top later, or perhaps make it multiple rows instead.
Is there a filter argument I can insert that says something in the lines of "Order all rows by score then title, except row ID 8, which should always be top"? Or could I perhaps change the order of the QuerySet manually after the filtering?
Thanks for any suggestions
You can override Model manager method:
class MyManager(models.Manager):
def filter(self, *args, **kwargs):
return super(MyManager, self).filter(*args, **kwargs).exclude(id=8).order_by(...)
then in model
class Stream(models.Model):
...
objects = MyManager()
EDIT
to make sure it is included you can query:
from django.db.models import Q
return super(MyManager, self).filter(Q(id=8) | Q(*args, **kwargs)).order_by(...)
I couldn't find a way to use query sets to do this, so I ended up converting the query set to a list and sorting it using the magic compare function. It ended up looking something like this:
The magic sorter, checking for names that should be stuck "on top":
# Comparison magic
def __cmp__(self, other):
# On top
if self.name in stream_ontop:
return 1
if other.name in stream_ontop:
return -1
# Score
if self.score > other.score:
return 1
if self.score < other.score:
return -1
# Title alphabetical
if self.info.title > other.info.title:
return -1
if self.info.title < other.info.title:
return 1
return 0
Then I fetched and sorted them like this from my model class
# Custom sorting
#staticmethod
def getAllSorted(online):
streams = list(Stream.objects.filter(online=online, info__isnull=False))
streams.sort(key=None, reverse=True)
return streams
I am trying to follow the code listed on https://github.com/alex/django-ajax-validation/blob/master/ajax_validation/views.py
I have been able to understand a small chunk of it. I have added comments stating my understanding of what is happening.
I would really appreciate some assistance on questions I listed in comments next to the lines I couldn't quite follow.
def validate(request, *args, **kwargs):
# I thing it is some sort of initializations but I cannot really understand what's happening
form_class = kwargs.pop('form_class')
defaults = {
'data': request.POST
}
extra_args_func = kwargs.pop('callback', lambda request, *args, **kwargs: {})
kwargs = extra_args_func(request, *args, **kwargs)
defaults.update(kwargs)
form = form_class(**defaults)
if form.is_valid(): #straightforward, if there is no error then the form is valid
data = {
'valid': True,
}
else:
# if we're dealing with a FormSet then walk over .forms to populate errors and formfields
if isinstance(form, BaseFormSet): #I cannot really understand what is BaseFromSet
errors = {}
formfields = {}
for f in form.forms: # I am guessing that this is for when there are multiple form submitted for validation
for field in f.fields.keys(): # I think he is looping over all fields and checking for error. what does add_prefix () return? and what is formfields[]?
formfields[f.add_prefix(field)] = f[field]
for field, error in f.errors.iteritems():
errors[f.add_prefix(field)] = error
if form.non_form_errors():
errors['__all__'] = form.non_form_errors() # what is the '__all__'?
else:
errors = form.errors
formfields = dict([(fieldname, form[fieldname]) for fieldname in form.fields.keys()])
# if fields have been specified then restrict the error list
if request.POST.getlist('fields'): # I am having a hard time understanding what this if statement does.
fields = request.POST.getlist('fields') + ['__all__']
errors = dict([(key, val) for key, val in errors.iteritems() if key in fields])
final_errors = {} # here the author of this code totally lost me.
for key, val in errors.iteritems():
if '__all__' in key:
final_errors[key] = val
elif not isinstance(formfields[key].field, forms.FileField):
html_id = formfields[key].field.widget.attrs.get('id') or formfields[key].auto_id
html_id = formfields[key].field.widget.id_for_label(html_id)
final_errors[html_id] = val
data = {
'valid': False or not final_errors,
'errors': final_errors,
}
json_serializer = LazyEncoder() # Why does the result have to be returned in json?
return HttpResponse(json_serializer.encode(data), mimetype='application/json')
validate = require_POST(validate) # a decorator that requires a post to submit
LazyEncoder
class LazyEncoder(JSONEncoder):
def default(self, obj):
if isinstance(obj, Promise):
return force_unicode(obj)
return obj
form_class = kwargs.pop('form_class')
This is simply pulling the keyword argument, form_class, that was passed in via the URL conf.
(r'^SOME/URL/$', 'ajax_validation.views.validate',
{'form_class': ContactForm}, # this keyword argument.
'contact_form_validate')
BaseFormSet is simply the formset class doing the work behind the scenes. When you don't know, search the source! grep -ri "baseformset" . It's an invaluable tool.
Take a look at at django.forms.formsets to see how formset_factory produces new "formset" classes based on the BaseFormSet, hence the factory part!
I am guessing that this is for when there are multiple form submitted for validation
Yes, that's exactly what a formset is for (dealing with multiple forms)
I think he is looping over all fields and checking for error. what does add_prefix () return? and what is formfields[]?
Yes, that would be looping through the field names.
add_prefix() is for prefixing form field names with a specific form. Because a formset repeats form elements multiple times, each field needs a unique prefix, such as 0-field1, 1-field1, etc.
formfields is just an empty dictionary defined a few lines above.
what is the 'all'?
__all__ is defined at the top of django.forms.forms
NON_FIELD_ERRORS = '__all__'
It's just what non field specific errors (such as constraints across 2 fields) are stored under in the errors dictionary as opposed to errors[fieldname].
I am having a hard time understanding what this if statement does.
The author has left a note:
# if fields have been specified then restrict the error list
if request.POST.getlist('fields'):
It's checking if you specified any specific fields to validate in your URLConf, this is not django but ajax_validation.
You can see that he's overwriting his errors dictionary based on only the fields specified, thus passing on the validation only for those fields.
errors = dict([(key, val) for key, val in errors.iteritems() if key in fields])
here the author of this code totally lost me.
The author has mapped a custom errors and fields dictionary to specific field names with prefixes, (as opposed to the usual FormSet with each form having its own errors dictionary, unaware of the formset itself) which he presumably uses in the AJAX response to validate all fields.
Normally, you can iterate over a formset and go through the errors on a form by form basis, but not so if you need to validate all of them through ajax.
The line pulling html_id should be straight forward most of the time, but it's there because form widgets CAN add interesting things to the end of the ID's based on whether or not the widget is a radio select for example.
From source comments :
# RadioSelect is represented by multiple <input type="radio"> fields,
# each of which has a distinct ID. The IDs are made distinct by a "_X"
# suffix, where X is the zero-based index of the radio field. Thus,
# the label for a RadioSelect should reference the first one ('_0').
Why does the result have to be returned in json?
Because it's an ajax request and javascript easily eats json.
2- could you go through these lines of code...
extra_args_func = kwargs.pop('callback', lambda request, *args, **kwargs: {})
Either return a keyword argument named 'callback' (which if passed in, is supposed to be a function that accepts request and return a dictionary), and if it wasn't, return a lambda function that only returns an empty dictionary.
I'm not sure what the specific use is for the extra context. You could use it to run arbitrary snippets of code without modifying or subclassing ajax_validation...
It might help you to run this code, and put a debugger breakpoint in somewhere so you can step through and examine the variables and methods. You can do this by simply putting this line where you want to break:
import pdb; pdb.set_trace()
and you will be dumped into the debugger in the console.
I am writing an application in Django, which uses [year]/[month]/[title-text] in the url to identitfy news items. To manage the items I have defined a number of urls, each starting with the above prefix.
urlpatterns = patterns('msite.views',
(r'^(?P<year>[\d]{4})/(?P<month>[\d]{1,2})/(?P<slug>[\w]+)/edit/$', 'edit'),
(r'^(?P<year>[\d]{4})/(?P<month>[\d]{1,2})/(?P<slug>[\w]+)/$', 'show'),
(r'^(?P<year>[\d]{4})/(?P<month>[\d]{1,2})/(?P<slug>[\w]+)/save$', 'save'),
)
I was wondering, if there is a mechanism in Django, which allows me to preprocess a given request to the views edit, show and save. It could parse the parameters e.g. year=2010, month=11, slug='this-is-a-title' and extract a model object out of them.
The benefit would be, that I could define my views as
def show(news_item):
'''does some stuff with the news item, doesn't have to care
about how to extract the item from request data'''
...
instead of
def show(year, month, slug):
'''extract the model instance manually inside this method'''
...
What is the Django way of solving this?
Or in a more generic way, is there some mechanism to implement request filters / preprocessors such as in JavaEE and Ruby on Rails?
You need date based generic views and create/update/delete generic views maybe?
One way of doing this is to write a custom decorator. I tested this in one of my projects and it worked.
First, a custom decorator. This one will have to accept other arguments beside the function, so we declare another decorator to make it so.
decorator_with_arguments = lambda decorator: lambda * args, **kwargs: lambda func: decorator(func, *args, **kwargs)
Now the actual decorator:
#decorator_with_arguments
def parse_args_and_create_instance(function, klass, attr_names):
def _function(request, *args, **kwargs):
model_attributes_and_values = dict()
for name in attr_names:
value = kwargs.get(name, None)
if value: model_attributes_and_values[name] = value
model_instance = klass.objects.get(**model_attributes_and_values)
return function(model_instance)
return _function
This decorator expects two additional arguments besides the function it is decorating. These are respectively the model class for which the instance is to be prepared and injected and the names of the attributes to be used to prepare the instance. In this case the decorator uses the attributes to get the instance from the database.
And now, a "generic" view making use of a show function.
def show(model_instance):
return HttpResponse(model_instance.some_attribute)
show_order = parse_args_and_create_instance(Order, ['order_id'])(show)
And another:
show_customer = parse_args_and_create_instance(Customer, ['id'])(show)
In order for this to work the URL configuration parameters must contain the same key words as the attributes. Of course you can customize this by tweaking the decorator.
# urls.py
...
url(r'^order/(?P<order_id>\d+)/$', 'show_order', {}, name = 'show_order'),
url(r'^customer/(?P<id>\d+)/$', 'show_customer', {}, name = 'show_customer'),
...
Update
As #rebus correctly pointed out you also need to investigate Django's generic views.
Django is python after all, so you can easily do this:
def get_item(*args, **kwargs):
year = kwargs['year']
month = kwargs['month']
slug = kwargs['slug']
# return item based on year, month, slug...
def show(request, *args, **kwargs):
item = get_item(request, *args, **kwargs)
# rest of your logic using item
# return HttpResponse...