Sphinx documentation on Django templatetags - django

I am using Sphinx with autodoc for documenting my Django project.
Design guys want to have documentation page(s) about all the template tags that are defined in the project. Of course, I can make such a page by enumerating all the template processing functions by hand, but I think it is not DRY, isn't it? In fact, template tag processing functions are all marked with #register.inclusion_tag decorator. So it seems to be possible and natural for some routine to collect them all and put into documentation.
The same about filter functions.
I've googled it, searched Django documentation but in vein. I can hardly believe that such a natural functionality hasn't been implemented by someone.

I didn't stop at this point and implemented a Sphinx autodoc extension.
Snippet 2. Sphinx autodoc extension
"""
Extension of Sphinx autodoc for Django template tag libraries.
Usage:
.. autotaglib:: some.module.templatetags.mod
(options)
Most of the `module` autodoc directive flags are supported by `autotaglib`.
Andrew "Hatter" Ponomarev, 2010
"""
from sphinx.ext.autodoc import ModuleDocumenter, members_option, members_set_option, bool_option, identity
from sphinx.util.inspect import safe_getattr
from django.template import get_library, InvalidTemplateLibrary
class TaglibDocumenter(ModuleDocumenter):
"""
Specialized Documenter subclass for Django taglibs.
"""
objtype = 'taglib'
directivetype = 'module'
content_indent = u''
option_spec = {
'members': members_option, 'undoc-members': bool_option,
'noindex': bool_option,
'synopsis': identity,
'platform': identity, 'deprecated': bool_option,
'member-order': identity, 'exclude-members': members_set_option,
}
#classmethod
def can_document_member(cls, member, membername, isattr, parent):
# don't document submodules automatically
return False
def import_object(self):
"""
Import the taglibrary.
Returns True if successful, False if an error occurred.
"""
# do an ordinary module import
if not super(ModuleDocumenter, self).import_object():
return False
try:
# ask Django if specified module is a template tags library
# and - if it is so - get and save Library instance
self.taglib = get_library(self.object.__name__)
return True
except InvalidTemplateLibrary, e:
self.taglib = None
self.directive.warn(unicode(e))
return False
def get_object_members(self, want_all):
"""
Decide what members of current object must be autodocumented.
Return `(members_check_module, members)` where `members` is a
list of `(membername, member)` pairs of the members of *self.object*.
If *want_all* is True, return all members. Else, only return those
members given by *self.options.members* (which may also be none).
"""
if want_all:
return True, self.taglib.tags.items()
else:
memberlist = self.options.members or []
ret = []
for mname in memberlist:
if mname in taglib.tags:
ret.append((mname, self.taglib.tags[mname]))
else:
self.directive.warn(
'missing templatetag mentioned in :members: '
'module %s, templatetag %s' % (
safe_getattr(self.object, '__name__', '???'), mname))
return False, ret
def setup(app):
app.add_autodocumenter(TaglibDocumenter)
This extension defines Sphinx directive autotaglib which behaves much like automodule, but enumerates only tag implementing functions.
Example:
.. autotaglib:: lib.templatetags.bfmarkup
:members:
:undoc-members:
:noindex:

For the record, Django has an automatic documentation system (add django.contrib.admindocs to your INSTALLED_APPS).
This will give you extra views in the admin (usually at /admin/docs/) that represent your models, views (based on the URL), template tags and template filters.
More documentation for this can be found in the admindocs section.
You can take a look at that code to include it in your documentation or at the extensions for the Django documentation.

I solved the problem and would like to share my snippets - in case they will be useful to someone.
Snippet 1. Simple documenter
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'
from django.template import get_library
def show_help(libname):
lib = get_library(libname)
print lib, ':'
for tag in lib.tags:
print tag
print lib.tags[tag].__doc__
if __name__ == '__main__':
show_help('lib.templatetags.bfmarkup')
Before running this script you should setup PYTHONPATH environment variable.

Related

Override method of a django builtin Class

Abstract of Question:
I am trying to extend (not replace) some method of any of django's builtin Class in my custom module which would allow to further extend the extended(overridden) method.
Lets say I have two custom modules mod1 and mod2. Want to override same method of some django's class say get_apps_list of AdminSite, In mod1 I want to add a line to say hello, in mod2 it should say hi.
Desired output:
There should be nothing if none of the modules installed,
it should say hi if mod1 installed and hello if mod2
installed.
And hi and hello if both installed
Question with real example:
Just for example, I need to modify the implementation of AdminSite.get_app_list like following
From
#app['models'].sort(key=lambda x: x['name'])
To
app['models'] = sort_with_name_length(app['models']) //my own method
Expected/desired approach: I supposed that it should be achievable by just writing following code in any of my custom module's models.py or sites.py file
class MyAdminSite(AdminSite):
def get_app_list(self, request):
app_list = super().get_app_list(request)
for app in app_list:
#app['models'].sort(key=lambda x: x['name'])
app['models'] = sort_with_name_length(app['models']) #an example change I need
return app_list
But what above code does is nothing, its never executed, until I use Monkey patching guided by this answer.
What I could achieve
from django.contrib.admin import AdminSite
class MyAdminSite(AdminSite):
def get_app_list(self, request):
# res = super(MyAdminSite, self).get_app_list(request) //gives following error
# super(type, obj): obj must be an instance or subtype of type
# So i have to rewrite complete method again in my module like following
app_dict = self._build_app_dict(request)
app_list = sorted(app_dict.values(), key=lambda x: x['name'].lower())
for app in app_list:
#app['models'].sort(key=lambda x: x['name'])
app['models'] = sort_with_name_length(app['models'])
return app_list
AdminSite.get_app_list = MyAdminSite.get_app_list
Problem being faced: Above does what I need in a totally undesired way. This solution has two problems
It will not allow me Multilevel Inheritance (I would not be able to have child and grand child)
Its actually even not an overriding, its just a replacement of implementation as it gives error using super
Just for elaboration, following is an example of similar behavior overriding in odoo, what I want to achieve with django
The exact expected/desired behavior is offered by odoo.
You can see get_auth_signup_qcontext method of auth_oauth's main controller
https://github.com/odoo/odoo/blob/14.0/addons/auth_oauth/controllers/main.py
https://github.com/odoo/odoo/blob/14.0/addons/auth_signup/controllers/main.py
What is does is if auth_oauth module is installed anywhere we call get_auth_signup_qcontext it would first go to child(auth_oauth)'s get_auth_signup_qcontext method which will call super in it. But if auth_oauth is not installed anywhere we call get_auth_signup_qcontext will directly hit auth_signup's method
Method 1. Use your own admin site's urls
One can simply use their own admin site's urls (reference):
from django.contrib.admin import AdminSite
class MyAdminSite(AdminSite):
def get_app_list(self, request):
app_list = super().get_app_list(request)
for app in app_list:
#app['models'].sort(key=lambda x: x['name'])
app['models'] = sort_with_name_length(app['models']) #an example change I need
return app_list
admin_site = MyAdminSite(name='myadmin')
admin_site.register(MyModel) # registering models
Then in your urls:
from django.urls import path
from myapp.admin import admin_site
urlpatterns = [
path('myadmin/', admin_site.urls),
]
Method 2. Overriding the default admin site
You can override the admin site:
In apps.py:
from django.contrib.admin.apps import AdminConfig
class MyAdminConfig(AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
In settings.py:
INSTALLED_APPS = [
...
'myproject.apps.MyAdminConfig', # replaces 'django.contrib.admin'
...
]
Quite an awkward and hacky but completely independent/loosely coupled solution to really override/extend (not replace) a builtin class of django's method is
Caution read the complete answer before using code :)
Add following code to models.py of any of your custom (installed) app
from django.contrib.admin import AdminSite
original_get_app_list = AdminSite.get_app_list
class AdminSiteExtension1(AdminSite):
def get_app_list(self, request):
// do something here to manipulate earlier than calling parent
// Following is the solution line
app_list = original_get_app_list(self, request)
for app in app_list:
app['name'] = app['name'] + '_my_app'
return app_list
AdminSite.get_app_list = AdminSiteExtension1.get_app_list
AdminSite is inherited just to make the use of self
Its really loosely coupled as u need to do nothing anywhere else. Even you can extend it further in any other module and you will not need AdminSiteExtension1 in that module because the updates to get_app_list of original AdminSite reside in original method as long as the app having AdminSiteExtension1 is installed.
Disclosure: I am not really a well learned and visionary programmer so I cannot imagine a situation where this solution can cause any problem, so if someone guides, it would be welcome. otherwise the beneficiary has to take care him/herself.

Avoiding circular imports in Django Models (Config class)

I've created a Configuration model in django so that the site admin can change some settings on the fly, however some of the models are reliant on these configurations. I'm using Django 2.0.2 and Python 3.6.4.
I created a config.py file in the same directory as models.py.
Let me paracode (paraphase the code? Real Enum has many more options):
# models.py
from .config import *
class Configuration(models.Model):
starting_money = models.IntegerField(default=1000)
class Person(models.Model):
funds = models.IntegarField(default=getConfig(ConfigData.STARTING_MONEY))
# config.py
from .models import Configuration
class ConfigData(Enum):
STARTING_MONEY = 1
def getConfig(data):
if not isinstance(data, ConfigData):
raise TypeError(f"{data} is not a valid configuration type")
try:
config = Configuration.objects.get_or_create()
except Configuration.MultipleObjectsReturned:
# Cleans database in case multiple configurations exist.
Configuration.objects.exclude(Configuration.objects.first()).delete()
return getConfig(data)
if data is ConfigData.MAXIMUM_STAKE:
return config.max_stake
How can I do this without an import error? I've tried absolute imports
You can postpone loading the models.py by loading it in the getConfig(data) function, as a result we no longer need models.py at the time we load config.py:
# config.py (no import in the head)
class ConfigData(Enum):
STARTING_MONEY = 1
def getConfig(data):
from .models import Configuration
if not isinstance(data, ConfigData):
raise TypeError(f"{data} is not a valid configuration type")
try:
config = Configuration.objects.get_or_create()
except Configuration.MultipleObjectsReturned:
# Cleans database in case multiple configurations exist.
Configuration.objects.exclude(Configuration.objects.first()).delete()
return getConfig(data)
if data is ConfigData.MAXIMUM_STAKE:
return config.max_stake
We thus do not load models.py in the config.py. We only check if it is loaded (and load it if not) when we actually execute the getConfig function, which is later in the process.
Willem Van Onsem's solution is a good one. I have a different approach which I have used for circular model dependencies using django's Applications registry. I post it here as an alternate solution, in part because I'd like feedback from more experienced python coders as to whether or not there are problems with this approach.
In a utility module, define the following method:
from django.apps import apps as django_apps
def model_by_name(app_name, model_name):
return django_apps.get_app_config(app_name).get_model(model_name)
Then in your getConfig, omit the import and replace the line
config = Configuration.objects.get_or_create()
with the following:
config_class = model_by_name(APP_NAME, 'Configuration')
config = config_class.objects.get_or_create()

Partial matching search in Wagtail with Postgres

I've got a wagtail site powered by Postgres and would like to implement a fuzzy search on all documents. However, according to wagtail docs "SearchField(partial_match=True) is not handled." Does anyone know of a way I can implement my own partial matching search?
I'm leaving this question intentionally open-ended because I'm open to pretty much any solution that works well and is fairly scalable.
We’re currently rebuilding the Wagtail search API in order to make autocomplete usable roughly the same way across backends.
For now, you can use directly the IndexEntry model that stores search data. Unfortunately, django.contrib.postgres.search does not contain a way to do an autocomplete query, so we have to do it ourselves for now. Here is how to do that:
from django.contrib.postgres.search import SearchQuery
from wagtail.contrib.postgres_search.models import IndexEntry
class SearchAutocomplete(SearchQuery):
def as_sql(self, compiler, connection):
return "to_tsquery(''%s':*')", [self.value]
query = SearchAutocomplete('postg')
print(IndexEntry.objects.filter(body_search=query).rank(query))
# All results containing words starting with “postg”
# should be displayed, sorted by relevance.
It doesn't seem to be documented yet, but the gist of autocomplete filtering with Postgres, using a request object, is something like
from django.conf import settings
from wagtail.search.backends import get_search_backend
from wagtail.search.backends.base import FilterFieldError, OrderByFieldError
def filter_queryset(queryset, request):
search_query = request.GET.get("search", "").strip()
search_enabled = getattr(settings, 'WAGTAILAPI_SEARCH_ENABLED', True)
if 'search' in request.GET and search_query:
if not search_enabled:
raise BadRequestError("search is disabled")
search_operator = request.GET.get('search_operator', None)
order_by_relevance = 'order' not in request.GET
sb = get_search_backend()
try:
queryset = sb.autocomplete(search_query, queryset, operator=search_operator, order_by_relevance=order_by_relevance)
except FilterFieldError as e:
raise BadRequestError("cannot filter by '{}' while searching (field is not indexed)".format(e.field_name))
except OrderByFieldError as e:
raise BadRequestError("cannot order by '{}' while searching (field is not indexed)".format(e.field_name))
The line to note is the call to sb.autocomplete.
If you want to use custom fields with autocomplete, you'll also need to add them into search_fields as an AutocompleteField in addition to a SearchField -- for example
search_fields = Page.search_fields + [
index.SearchField("field_to_index", partial_match=True)
index.AutocompleteField("field_to_index", partial_match=True),
...
This solution is working for Wagtail 2.3. If you using an older version, it is unlikely to work, and if you are using a future version, hopefully the details will be incorporated into the official documents, which currently state that autocomplete with Postgres is NOT possible. Thankfully, that has turned out to not be true, due to the work of Bertrand Bordage in the time since he wrote the other answer.

How to limit file types on file uploads for ModelForms with FileFields?

My goal is to limit a FileField on a Django ModelForm to PDFs and Word Documents. The answers I have googled all deal with creating a separate file handler, but I am not sure how to do so in the context of a ModelForm. Is there a setting in settings.py I may use to limit upload file types?
Create a validation method like:
def validate_file_extension(value):
if not value.name.endswith('.pdf'):
raise ValidationError(u'Error message')
and include it on the FileField validators like this:
actual_file = models.FileField(upload_to='uploaded_files', validators=[validate_file_extension])
Also, instead of manually setting which extensions your model allows, you should create a list on your setting.py and iterate over it.
Edit
To filter for multiple files:
def validate_file_extension(value):
import os
ext = os.path.splitext(value.name)[1]
valid_extensions = ['.pdf','.doc','.docx']
if not ext in valid_extensions:
raise ValidationError(u'File not supported!')
Validating with the extension of a file name is not a consistent way. For example I can rename a picture.jpg into a picture.pdf and the validation won't raise an error.
A better approach is to check the content_type of a file.
Validation Method
def validate_file_extension(value):
if value.file.content_type != 'application/pdf':
raise ValidationError(u'Error message')
Usage
actual_file = models.FileField(upload_to='uploaded_files', validators=[validate_file_extension])
An easier way of doing it is as below in your Form
file = forms.FileField(widget=forms.FileInput(attrs={'accept':'application/pdf'}))
Django since 1.11 has a FileExtensionValidator for this purpose:
class SomeDocument(Model):
document = models.FileFiled(validators=[
FileExtensionValidator(allowed_extensions=['pdf', 'doc'])])
As #savp mentioned, you will also want to customize the widget so that users can't select inappropriate files in the first place:
class SomeDocumentForm(ModelForm):
class Meta:
model = SomeDocument
widgets = {'document': FileInput(attrs={'accept': 'application/pdf,application/msword'})}
fields = '__all__'
You may need to fiddle with accept to figure out exactly what MIME types are needed for your purposes.
As others have mentioned, none of this will prevent someone from renaming badstuff.exe to innocent.pdf and uploading it through your form—you will still need to handle the uploaded file safely. Something like the python-magic library can help you determine the actual file type once you have the contents.
For a more generic use, I wrote a small class ExtensionValidator that extends Django's built-in RegexValidator. It accepts single or multiple extensions, as well as an optional custom error message.
class ExtensionValidator(RegexValidator):
def __init__(self, extensions, message=None):
if not hasattr(extensions, '__iter__'):
extensions = [extensions]
regex = '\.(%s)$' % '|'.join(extensions)
if message is None:
message = 'File type not supported. Accepted types are: %s.' % ', '.join(extensions)
super(ExtensionValidator, self).__init__(regex, message)
def __call__(self, value):
super(ExtensionValidator, self).__call__(value.name)
Now you can define a validator inline with the field, e.g.:
my_file = models.FileField('My file', validators=[ExtensionValidator(['pdf', 'doc', 'docx'])])
I use something along these lines (note, "pip install filemagic" is required for this...):
import magic
def validate_mime_type(value):
supported_types=['application/pdf',]
with magic.Magic(flags=magic.MAGIC_MIME_TYPE) as m:
mime_type=m.id_buffer(value.file.read(1024))
value.file.seek(0)
if mime_type not in supported_types:
raise ValidationError(u'Unsupported file type.')
You could probably also incorporate the previous examples into this - for example also check the extension/uploaded type (which might be faster as a primary check than magic.) This still isn't foolproof - but it's better, since it relies more on data in the file, rather than browser provided headers.
Note: This is a validator function that you'd want to add to the list of validators for the FileField model.
I find that the best way to check the type of a file is by checking its content type. I will also add that one the best place to do type checking is in form validation. I would have a form and a validation as follows:
class UploadFileForm(forms.Form):
file = forms.FileField()
def clean_file(self):
data = self.cleaned_data['file']
# check if the content type is what we expect
content_type = data.content_type
if content_type == 'application/pdf':
return data
else:
raise ValidationError(_('Invalid content type'))
The following documentation links can be helpful:
https://docs.djangoproject.com/en/3.1/ref/files/uploads/ and https://docs.djangoproject.com/en/3.1/ref/forms/validation/
I handle this by using a clean_[your_field] method on a ModelForm. You could set a list of acceptable file extensions in settings.py to check against in your clean method, but there's nothing built-into settings.py to limit upload types.
Django-Filebrowser, for example, takes the approach of creating a list of acceptable file extensions in settings.py.
Hope that helps you out.

Django with Pluggable MongoDB Storage troubles

I'm trying to use django, and mongoengine to provide the storage backend only with GridFS. I still have a MySQL database.
I'm running into a strange (to me) error when I'm deleting from the django admin and am wondering if I am doing something incorrectly.
my code looks like this:
# settings.py
from mongoengine import connect
connect("mongo_storage")
# models.py
from mongoengine.django.storage import GridFSStorage
class MyFile(models.Model):
name = models.CharField(max_length=50)
content = models.FileField(upload_to="appsfiles", storage=GridFSStorage())
creation_time = models.DateTimeField(auto_now_add=True)
last_update_time = models.DateTimeField(auto_now=True)
I am able to upload files just fine, but when I delete them, something seems to break and the mongo database seems to get in an unworkable state until I manually delete all FileDocument.objects. When this happens I can't upload files or delete them from the django interface.
From the stack trace I have:
/home/projects/vector/src/mongoengine/django/storage.py in _get_doc_with_name
doc = [d for d in docs if getattr(d, self.field).name == name] ...
▼ Local vars
Variable Value
_[1]
[]
d
docs
Error in formatting: cannot set options after executing query
name
u'testfile.pdf'
self
/home/projects/vector/src/mongoengine/fields.py in __getattr__
raise AttributeError
Am I using this feature incorrectly?
UPDATE:
thanks to #zeekay's answer I was able to get a working gridfs storage plugin to work. I ended up not using mongoengine at all. I put my adapted solution on github. There is a clear sample project showing how to use it. I also uploaded the project to pypi.
Another Update:
I'd highly recommend the django-storages project. It has lots of storage backed options and is used by many more people than my original proposed solution.
I think you are better off not using MongoEngine for this, I haven't had much luck with it either. Here is a drop-in replacement for mongoengine.django.storage.GridFSStorage, which works with the admin.
from django.core.files.storage import Storage
from django.conf import settings
from pymongo import Connection
from gridfs import GridFS
class GridFSStorage(Storage):
def __init__(self, host='localhost', port=27017, collection='fs'):
for s in ('host', 'port', 'collection'):
name = 'GRIDFS_' + s.upper()
if hasattr(settings, name):
setattr(self, s, getattr(settings, name))
for s, v in zip(('host', 'port', 'collection'), (host, port, collection)):
if v:
setattr(self, s, v)
self.db = Connection(host=self.host, port=self.port)[self.collection]
self.fs = GridFS(self.db)
def _save(self, name, content):
self.fs.put(content, filename=name)
return name
def _open(self, name, *args, **kwars):
return self.fs.get_last_version(filename=name)
def delete(self, name):
oid = fs.get_last_version(filename=name)._id
self.fs.delete(oid)
def exists(self, name):
return self.fs.exists({'filename': name})
def size(self, name):
return self.fs.get_last_version(filename=name).length
GRIDFS_HOST, GRIDFS_PORT and GRIDFS_COLLECTION can be defined in your settings or passed as host, port, collection keyword arguments to GridFSStorage in your model's FileField.
I referred to Django's custom storage documenation, and loosely followed this answer to a similar question.