Django Object is Not Serializable CommandError using Dumpdata with Natural Keys - django

Question

I am trying to use 'natural keys' for serialization (docs) in a "manage.py dumpdata" command:
python manage.py dumpdata --natural-primary --natural-foreign --indent 4 --format json --verbosity 1 > tests\test_fixtures\test_db2.json
and I am getting the following error when I use --natural-foreign on other apps that use the Project or Task model (which they all must by design):
CommandError: Unable to serialize database: Object of type Project is not JSON serializable
Exception ignored in: <generator object cursor_iter at 0x000001EF62481B48>
Traceback (most recent call last):
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\db\models\sql\compiler.py", line 1586, in cursor_iter
cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.
If I just dumpdata from this, the 'projects' app, it works, but other apps are built with entities related to Project or Task and there the --natural-foreign option fails.
The problem occurs when a model (say Question) calls for a natural_key from Task, which includes a for a natural_key from Project.
If I use the Pycharm Python Console to access querysets of Projects or Tasks ('q' here), this works:
serializers.serialize('json', q, indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)
But if 'w' is a list of Question objects from another app that have a Task foreign key I get this error:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\__init__.py", line 128, in serialize
s.serialize(queryset, **options)
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\base.py", line 115, in serialize
self.end_object(obj)
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\json.py", line 53, in end_object
json.dump(self.get_dump_object(obj), self.stream, **self.json_kwargs)
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\__init__.py", line 179, in dump
for chunk in iterable:
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 405, in _iterencode_dict
yield from chunks
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 405, in _iterencode_dict
yield from chunks
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 325, in _iterencode_list
yield from chunks
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 438, in _iterencode
o = _default(o)
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\json.py", line 104, in default
return super().default(o)
File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Project is not JSON serializable
The models are:
# projects.models.py
class BaseModelWithHistory(models.Model):
history = HistoricalRecords(inherit=True)
natural_key_fields = ('id',) # default
class Meta:
abstract = True
def natural_key(self):
fieldlist = [getattr(self, fieldname) for fieldname in self.natural_key_fields]
return tuple(fieldlist)
# natural_key.dependencies = ['projects.Project', 'projects.Task'] # serialize these first.
class Project(BaseModelWithHistory):
"""
'Projects' group Tasks.
"""
project_name = models.CharField(max_length=200, default="development_project")
project_short_description = models.CharField(
max_length=500,
default="This is the default text.")
target_group = models.ManyToManyField(Group, blank=True)
objects = ProjectDiscreteManager()
natural_key_fields = ('project_name',)
class Task(BaseModelWithHistory):
number = models.PositiveIntegerField(default=0)
name = models.CharField(max_length=200, default='new task')
project = models.ForeignKey(Project, on_delete=models.CASCADE)
target_group = models.ManyToManyField(Group, blank=True)
app_label = models.CharField(max_length=80, choices=app_choices(), null=True, blank=True)
objects = discrete_manager_factory('project')
natural_key_fields = ('project', 'name', 'number')
Things that I wouldn't expect to cause this problem but I could certainly be wrong:
The ProjectDiscreteManager and others created by discrete_manager_factory() behave exactly as the default manager (models.Manager()), unless its called with a request from an identified user, in which case it adds a filter to see if that user is in the Group.
All models define the natural_keys tuple because the parent class defines it as ('id',); most models overwrite this with more representative fields.
With the natural_key.dependencies list set for all models to prioritize Project and then Task, I get a 'can't resolve dependencies' error for every other model. I think this ticket relates, but am not sure how to track down whether this fix is already in the Django 3.0.6 I'm using and I should just straighten up & fly right, or if my Han Solo 'this should wooork' will one day soon be rewarded. [Update: I worked out it is coming in Django 3.1.1, but I'm not sure that it is going to fix the "can't resolve dependencies" error I've created for myself.]