Django: Data corrupted after loading? (possible programmer error) - django

I may be loading data the wrong way.
excerpt of data.json:
{
"pk": "1",
"model": "myapp.Course",
"fields":
{
"name": "Introduction to Web Design",
"requiredFor": [9],
"offeringSchool": 1,
"pre_reqs": [],
"offeredIn": [1, 5, 9]
}
},
I run python manage.py loaddata -v2 data:
Installed 36 object(s) from 1
fixture(s)
Then, I go to check the above object using the Django shell:
>>> info = Course.objects.filter(id=1)
>>> info.get().pre_reqs.all()
[<Course: Intermediate Web Programming>] # WRONG! There should be no pre-reqs
>>> from django.core import serializers
>>> serializers.serialize("json", info)
'[{"pk": 1, "model": "Apollo.course", "fields": {"pre_reqs": [11], "offeredIn": [1, 5, 9], "offeringSchool": 1, "name": "Introduction to Web Design", "requiredFor": [9]}}]'
The serialized output of the model is not the same as the input that was given to loaddata. The output has a non-empty pre_req list, whereas the input's pre_reqs field is empty. What am I doing wrong?

I think there is already content in your many-to-many table pre_reqs (with FK=1) (before you load your JSON data).
It seems the loader will not delete already existing tuples in many-to-many tables.
Have a look at the django.core.serializer.base.DeserializedObject class.
The DeserializedObject.save method only adds new relations.

Related

Django id vs pk in fixtures/loaddata

A strange (to the uninitiated lol) issue with models using a custom CharField for a primary key/id:
id = models.CharField(max_length=10, primary_key=True)
Feeling I know what I'm doing, I've created the following (json) fixtures file:
[
{
"model": "products.product",
"id": "am11",
"fields": {
"title": "Test Product A",
}
},
{
"model": "products.product",
"id": "am22",
"fields": {
"title": "Test Product B",
}
}
]
, and proceeded with loading it:
✗ python manage.py loaddata fixtures/products.json
Installed 2 object(s) from 1 fixture(s)
Well, it kinda lied. A check on the admin page or in a shell shows that there's only one Product in the database - the last one in the fixture's list. Curiously enough, attempts to delete this Product via the admin page silently fail, only via the shell can it actually be deleted. Further investigation (in the shell) revealed an interesting problem - that (single) Product created has pk/id set to an empty string(?!..but explains why admin fails to delete it). If I manually create another, either on admin page or in the shell, the new product appears without any issues, both id and pk set to the string given. But loaddata with fixture fails on this. Originally discovered this problem when a basic test failed - given the same fixture, in failed the assertion on the number of products in the queryset, claiming there's just one.
Now, I was able to "fix" the problem by renaming 'id' to 'pk' in the fixture file. I say 'fix', because I don't understand what bit me here. Any clue will be appreciated. Thanks!

Marshaling items in lists using flask restful

I am writing a Flask-RESTFUL resource. It returns a Model object called DocumentSet with the following structure:
DocumentSet:
id: int
documents: list of Document
Document is another Model object with the following structure:
Document:
id: int
...other fields...
I want to write a #marshal_with decorator that returns the DocumentSet id along with a list of the Document ids like so:
{
id: 5,
document_ids: [1, 2, 3]
}
I've been banging my head against the output marshaling syntax to no avail. Some of the things I've tried:
{'id': fields.Integer, 'document_ids':fields.List(fields.Integer, attribute='documents.id')}
{'id': fields.Integer, 'document_ids':fields.List(fields.Nested({'id':fields.Integer}), attribute='documents')}
What's the magic incantation?
The magic incantation is
{'id': fields.Integer, 'name': fields.String, 'document-ids':{'id': fields.Integer}}
It was right there in the "Complex Structures" paragraph in the documentation.

initial_data for django running with MongoDB

After much hardship, I have managed to convert my django project, that previously ran with sqlite, to run with MongoDB.
This is great, beside from the fact that my old version had a massive initial_data.json file, which now fails to work with the new db when running django's syncdb command.
EDIT:
this is an example of the initial_data.json file :
[{"pk":1,
"model": "vcb.dishtype",
"fields": {
"photo": "images/dishes/breakfast.jpg",
"name": "Breakfast"
}
},
{"pk":2,
"model": "vcb.dishtype",
"fields": {
"photo": "images/dishes/bread_and_pastry.jpg",
"name": "Bread and pastry"
}
}]
and after running the syncdb I get:
DeserializationError: Problem installing fixture
'C:\Users..\initial_data.json' : u'pk'
It seems to be a problem with the MongoDB objectId and how I defined the initial_data file.
I tried to remove all the pks fields from the file, but still the same error.
EDIT
I tried putting just two fixtures, if I don't set the pk, I get the same error as above. If I do set it, I get :
"DatabaseErroe: Problem installing fixture 'C:..\initial_data.json':
could not load vcb.dishtype(pk=1): AutoField (default primary key)
values must be strings representing an ObjectId on MongoDB (got u'1'
instead)".
which is a similar problem I had with the django Site, that was solved with the help of this thread: Django MongoDB Engine error when running tellsiteid
This raises my suspicion that there's a different way to set the fixtures in this infrastructure. Maybe syncdb isn't the way to go, and there should be a sort of dump maybe?
I've searched google, and nothing seems to tackle this problem. I'm obviously asking the wrong questions.
what should I do, to create fixtures in my altered project?
thanks a lot,
Nitzan
From your error message, I assume you are using Django MongoDB Engine?
Your pk values must be valid ObjectIds, try using values like:
'000000000000000000000001'
'000000000000000000000002'
etc
You can get ObjectIds or check that you have correct values:
>>> from bson.objectid import ObjectId
>>> ObjectId()
> ObjectId('52af59bac38f8420734d064d')
>>> ObjectId('000000000000000000000001')
> ObjectId('000000000000000000000001')
>>> ObjectId('bad')
*error*

Django deep serialization - follow reverse foreign key constraints

I have two models with a foreign key relationship:
class Company(models.Model):
field 1
field 2
class Employee(models.Model):
company = Model.ForeignKey('Company')
field 3
field 4
I would like to JSON serialize a company instance, and include all employees that have foreign-key relationships to it. IE, I'd like to create JSON that looks something like the following, in that it includes all fields for a company and all fields for all related employees.
[
{
"pk": 2,
"model": "app.company",
"fields": {
"field1": "value",
"field2": "value",
"employee": [
{
"pk": 19,
"model": "app.employee",
"fields": {
"field3": "value",
"field4": "value",
}
},
{
"pk": 25,
"model": "app.employee",
"fields": {
"field3": "value",
"field4": "value",
}
}
]
}
}
]
The Django serializer doesn't serialize relationships. Other questions here have asked how to deep serialize, but in the opposite direction -- IE, serialize an employee along with its related company. Answers to these questions have pointed out that the wadofstuff django-full-serializer plugin allows you to do this kind of deep serialization. The problem is that the wadofstuff plugin only follows these relationships unidirectionally -- it won't follow a reverse foreign key constraint. So, I'm trying to roll my own here. Any suggestions on how to accomplish this?
So, here's a super budget way of doing this which works for my purposes, but I feel like there has to be a better way (including it here in case others are looking for how to do this). It works for me only because I'm sending just one Company object at a time, and as such the fact that it doesn't explicitly preserve the relationship hierarchy isn't a big deal.
Given Company instance of "company":
companyJSON = serializers.serialize('json', [company, ])
employeeJSON = serializers.serialize('json', company.employee_set.all())
fullJSON = companyJSON[:-1] + ", " + employeeJSON[1:]

Django - Serializing model with additional data

I am trying to serialize some model data along with some extra information like so:
data = {
'model_data': serializers.serialize('json', SomeModel._default_manager.all(), fields=('name','last_updated')),
'urls': {
'updateURL':'http://www.bbc.co.uk',
},
}
json = simplejson.dumps(data)
It seams my 'model_data' object is being serialized twice as it seems to be returned as a string and not a valid json object:
Object
model_data: "[{"pk": 1, "model": "models.SomeModel", "fields": {"last_updated": null, "name": "Name test"}}]"
urls: Object
What am I doing wrong here?
How about having the value of the model_data field be handled again by another JSON processor? I think it will be the same, since the JSON processor always expects a string that is always in the correct format.