Using the Google CTemplate library, I have built a TemplateDictionary of params. Such a dictionary is a map of string keys to a variety of value types.
Typically, one passes CTemplate a template file wherein placeholders for each key in the dictionary are found and substituted.
In one case, though, I wish to emit the entire dictionary in JSON form, and the template language syntax doesn't appear to provide reflection such that I can write placeholders to loop over an unknown number of unknown keys in any arbitrary dictionary.
Did I miss some functionality?
If so, how can I add it?
Will I have to patch the CTemplate code? Much of what I seem to need for the job appears to be marked private i.e. for internal use only...
I've ended up hacking the CTemplate source in template_dictionary.h and template_dictionary.cc, cloning class class TemplateDictionary::DictionaryPrinter to produce a new class class TemplateDictionary::DictionaryJsonPrinter, adapting its member functions to emit JSON syntax.
Related
When I have to transfer complex values (e.g. a list of dicts) through a Django template to the front end, I typically use json_script to try and prevent XSS vectors.
Recently, I started using lit-element, which has a neat way of pulling attribute values from your custom elements and providing them as properties to your component. You can say:
<my-element items="{{ serialized array of items }}"></my-element>
Then lit-element will take whatever string value is passed to the items attribute and call JSON.parse() on it, so I need a way of serializing my value to JSON.
Since that is relatively trivial in itself, my initial idea was to write a custom template filter and try to match how json_script escapes values. But then I read the source for that function and it explicitly states:
Escape all the HTML/XML special characters with their unicode escapes,
so value is safe to be output anywhere except for inside a tag attribute.
This sounds like attribute values are potentially a more serious XSS vector. So I guess my question is - how to serialize data (in Django/Python) to JSON so it's safe for use in tag attribute values?
I think the reason it says can be used anywhere apart from attributes is that this function doesn't escape speechmarks, this means you would be able to close an attribute and start a new one easily. Django provides the escapejs template tag that escapes speech marks as well.
This function is in the same source that you linked in your question. Following the approach used there should be safe.
I have created custom Django model-field subclasses based on CharField but which use to_python() to ensure that the model objects returned have more complex objects (some are lists, some are dicts with a specific format, etc.) -- I'm using MySQL so some of the PostGreSql field types are not available.
All is working great, but Pylint believes that all values in these fields will be strings and thus I get a lot of "unsupported-membership-test" and "unsubscriptable-object" warnings on code that uses these models. I can disable these individually, but I would prefer to let Pylint know that these models return certain object types. Type hints are not helping, e.g.:
class MealPrefs(models.Model):
user = ...foreign key...
prefs: dict[str, list[str]] = \
custom_fields.DictOfListsExtendsCharField(
default={'breakfast': ['cereal', 'toast'],
'lunch': ['sandwich']},
)
I know that certain built-in Django fields return correct types for Pylint (CharField, IntegerField) and certain other extensions have figured out ways of specifying their type so Pylint is happy (MultiSelectField) but digging into their code, I can't figure out where the "magic" specifying the type returned would be.
(note: this question is not related to the INPUT:type of Django form fields)
Thanks!
I had a look at this out of curiosity, and I think most of the "magic" actually comes for pytest-django.
In the Django source code, e.g. for CharField, there is nothing that could really give a type hinter the notion that this is a string. And since the class inherits only from Field, which is also the parent of other non-string fields, the knowledge needs to be encoded elsewhere.
On the other hand, digging through the source code for pylint-django, though, I found where this most likely happens:
in pylint_django.transforms.fields, several fields are hardcoded in a similar fashion:
_STR_FIELDS = ('CharField', 'SlugField', 'URLField', 'TextField', 'EmailField',
'CommaSeparatedIntegerField', 'FilePathField', 'GenericIPAddressField',
'IPAddressField', 'RegexField', 'SlugField')
Further below, a suspiciously named function apply_type_shim, adds information to the class based on the type of field it is (either 'str', 'int', 'dict', 'list', etc.)
This additional information is passed to inference_tip, which according to the astroid docs, is used to add inference info (emphasis mine):
astroid can be used as more than an AST library, it also offers some
basic support of inference, it can infer what names might mean in a
given context, it can be used to solve attributes in a highly complex
class hierarchy, etc. We call this mechanism generally inference
throughout the project.
astroid is the underlying library used by Pylint to represent Python code, so I'm pretty sure that's how the information gets passed to Pylint. If you follow what happens when you import the plugin, you'll find this interesting bit in pylint_django/.plugin, where it actually imports the transforms, effectively adding the inference tip to the AST node.
I think if you want to achieve the same with your own classes, you could either:
Directly derive from another Django model class that already has the associated type you're looking for.
Create, and register an equivalent pylint plugin, that would also use Astroid to add information to the class so that Pylint know what to do with it.
I thought initially that you use a plugin pylint-django, but maybe you explicitly use prospector that automatically installs pylint-django if it finds Django.
The checker pylint neither its plugin doesn't check the code by use information from Python type annotations (PEP 484). It can parse a code with annotations without understanding them and e.g. not to warn about "unused-import" if a name is used in annotations only. The message unsupported-membership-test is reported in a line with expression something in object_A simply if the class A() doesn't have a method __contains__. Similarly the message unsubscriptable-object is related to method __getitem__.
You can patch pylint-django for your custom fields this way:
Add a function:
def my_apply_type_shim(cls, _context=None): # noqa
if cls.name == 'MyListField':
base_nodes = scoped_nodes.builtin_lookup('list')
elif cls.name == 'MyDictField':
base_nodes = scoped_nodes.builtin_lookup('dict')
else:
return apply_type_shim(cls, _context)
base_nodes = [n for n in base_nodes[1] if not isinstance(n, nodes.ImportFrom)]
return iter([cls] + base_nodes)
into pylint_django/transforms/fields.py
and also replace apply_type_shim by my_apply_type_shim in the same file at this line:
def add_transforms(manager):
manager.register_transform(nodes.ClassDef, inference_tip(my_apply_type_shim), is_model_or_form_field)
This adds base classes list or dict respectively, with their magic methods explained above, to your custom field classes if they are used in a Model or FormView.
Notes:
I thought also about a plugin stub solution that does the same, but the alternative with "prospector" seems so complicated for SO that I prefer to simply patch the source after installation.
Classes Model or FormView are the only classes created by metaclasses, used in Django. It is a great idea to emulate a metaclass by a plugin code and to control the analysis simple attributes. If I remember, MyPy, referenced in some comment here, has also a plugin mypy-django for Django, but only for FormView, because writing annotations for django.db is more complicated than to work with attributes. - I was trying to work on it for one week.
I'm trying to setup a LinkColumn and I've seen in the examples that a the args parameter has usually the form args=[A('pk')]. I'm wondering what is the meaning of the A().
From the documentation of django-tables, A is the Accessor Class.
A string describing a path from one object to another via attribute/index accesses. For convenience, the class has an alias A to allow for more concise code.
Relations are separated by a . character.
So basically you are using the primary key in this example to access the objects.
From django-tables2 source code
class Accessor(str):
'''
A string describing a path from one object to another via attribute/index
accesses. For convenience, the class has an alias `.A` to allow for more concise code.
Relations are separated by a ``.`` character.
'''
I'm using Sitecore. I want to get the full name/description of a DMS rule in programcode by Sitecore ID, for example: "Where the DayOfWeek has a value that is equal to Tuesday".
Who knows how to do this?
Thanks a lot.
Jordy
I don't know of a simple way, but the class responsible for rendering the rule text is Sitecore.Shell.Applications.Rules.RulesRenderer in Sitecore.Client.dll.
Its constructor accepts the XML from a rules field and you call the Render method, passing in a prepared HtmlTexteWriter. It also has a bunch of fairly self-explanatory private methods like RenderRule, RenderCondition etc.
I'm sure if you decompile that class you can pick out the bits you need.
I've found myself unsatisfied with Django's ability to render JSON data. If I use built in serializes then database foreign key relationships are not included in the data (only the keys). Also, it seems to be impossible to include custom data in the json feed that isn't part of the model being serialized.
As a test I implemented a template that rendered some JSON for the resultset of a particular model. I was able to include/exclude whatever parts of the model I wanted and was able to include custom data as well.
The test seemed to work well and wasn't slower than the recommended serialization methods.
Are there any pitfalls to this using this method of serialization?
While it's hard to say definitively whether this method has any pitfalls, it's the method we use in production as you control everything that is serialized, even if the underlying model is changed. We've been running a high traffic application in for almost two years using this method.
Hope this helps.
One problem might be escaping metacharacters like ". Django's template system automatically escapes dangerous characters, but it's set up to do that for HTML. You should look up exactly what the template escaping does, and compare that to what's dangerous in JSON. Otherwise, you could cause XSS problems.
You could think about constructing a data structure of dicts and lists, and then running a JSON serializer on that, rather than directly on your database model.
I don't understand why you see the choice as being either 'use Django serializers' or 'write JSON in templates'. The middle way, which to my mind is much more robust and fits your use case well, is to build up your data as Python lists/dictionaries and then simply use simplejson.dumps() to convert it to a JSON string.
We use this method to get custom JSON format consumed by datatables.net
It was the easiest method we find to accomplish this task and it looks very fine with no problems so far.
You can find details here: http://datatables.net/development/server-side/django
So far, generating JSON from templates, we've run into the need to escape newlines. Looking at doing simplejson.dumps() next.