Create json file for many to many fields in django - django

I need to create json to insert data using inside both models, for tags I have created the json but I don't get how to create json for Question model to insert data directly from it.
model.py
class Tag(models.Model):
id = models.AutoField(primary_key=True)
name = models.CharField(max_length=255)
class Question(models.Model):
name = models.CharField(max_length=255)
Tag_name = models.ManyToManyField(Tag)
Tag json look like:
[
{ "name": "a" },
{ "name": "b" }
]
Since Tag_name is many to many field it create 2 tables in sqlite but I want to add data using one json only.How to make a json so that data in both table get inserted ?

Let suppose you have read json file and now you want to store that data to db
json_data = get_json_data() # write code to read json here
tag_list = []
for tag in json_data:
tag_list.append(Tag(**tag))
added_tags = Tag.objects.bulk_create(tag_list)
added_tags = [t.id for t in added_tags]
question_object = # write code to get specific question object in which you want to add this
question_object.Tag_name.add(*added_tags)

You could opt for the builtin management command, ./manage.py loaddata. I think it's easier.
You only need a json fixture for the tag model... the fixture, tags.json, could look like:
[
{
"model":"yourapp.tag",
"pk":1,
"fields":{"name": "a" }
},
{
"model":"yourapp.tag",
"pk":2,
"fields":{"name": "b" }
}
]
Store your tags.json inside a folder called fixtures inside your app. Finally, run ./manage.py loaddata tags.json. It will prepopulate the Tag model. Hope it helps...

Related

Django, Create GIN index for child element in JSON Array field

I have a model that uses PostgreSQL and has field like this:
class MyModel(models.Model):
json_field = models.JSONField(default=list)
This field contains data like this:
[
{"name": "AAAAA", "product": "11111"},
{"name": "BBBBB", "product": "22222"},
]
Now I want to index by json_field -> product field, because it is being used as identification. Then i want to create GinIndex like this:
class Meta:
indexes = [
GinIndex(name='product_json_idx', fields=['json_field->product'], opclasses=['jsonb_path_ops'])
]
When I try to create migration, I get error like this:
'indexes' refers to the nonexistent field 'json_field->product'.
How to create GinIndex that will be used for child attribute in Json Array?
Please don't use a JSONField [Django-doc] for well-structured data: if the structure is clear, like here where we have a list of objects where each object has a name and a product, it makes more sense to work with extra models, like:
class MyModel(models.Model):
# …
pass
class Product(models.Model):
# …
pass
class Entry(models.Model):
my_model = models.ForeignKey(MyModel, on_delete=models.CASCADE)
name = models.CharField(max_length=255)
product = models.ForeignKey(Product, on_delete=models.CASCADE)
This will automatically add indexes on the ForeignKeys, but will also make querying simpeler and usually more efficient.
While databases like PostgreSQL indeed have put effort into making JSON columns easier to query, aggregate, etc. usually it is still beter to perform database normalization [wiki], especially since it has more means for referential integrity, and a lot of aggregates are simpeler on linear data.
If for example later a product is removed, it will require a lot of work to inspect the JSON blobs to remove that product. This is however a scenario that both Django and PostgreSQL databases cover with ON DELETE triggers and which will likely be more effective and safe when using the Django toolchain for this.

Generating a Django Model that can have multiple values in one field

I am trying to generate a Django model that can handle multiple values in a single field. As such, when the first field is queried through a view, a user should select a value for the second field through a select box.
To give a background of the problem, my seeding fixture looks like this...
[
{
"model":"myapp.location",
"pk":1,
"fields":{
"county": "countyname",
"places":{
"name": "placename",
"name": "placename",
"name": "placename",
"name": "placename",
"name": "placename"
}
}
}
]
In the above scenario, location is the intended name of my model. Now, through a form, I want a user to be presented with 'countynames'. Upon selecting a countyname, the user should be presented with 'placenames' in the selected county for them to choose.
I have tried the following format for the model...
class Location(models.Model):
county = models.CharField(max_length=100)
places = models.CharField(max_length=100, choices=places.name)
def __str__(self):
return self.countyname
Now, I know that the error that is thrown, ('places' is not defined), is warranted. I was asking whether there is a way to define it (places), as it is in the fixture, or if anyone has a better implementation for such a model... any alternative way is welcome and appreciated as I can't think of anything at this point.
So, after fiddling with two models and foreign keys as suggested in the comments above, I decided to amend the model, which also led to changing the fixture. I read about ArrayFields in Postgres + Django here. I amended the field 'places' to be an ArrayField as shown:
from django.contrib.postgres.fields import ArrayField
class Location(models.Model):
county = models.CharField(max_length=100)
places = ArrayField(models.CharField(max_length=100), blank=True)
def __str__(self):
return self.county
Next, it was just a matter of changing the JSON fixture to:
[
{
"model":"myapp.location",
"pk":1,
"fields":{
"county": "countyname",
"places":["placename","placename","placename","placename"]
}
}
]
After running python manage.py loaddata fixturename.json ,
it worked and the DB was seeded!

Customising the ID used in Marshmallow URLFor

I am using a Sqlite database, and flask, marshmallow, sqlalchemy to serve as a web api for a front end project.
I'm using a UUID stored as a blob in the database, and trying to stylise the data when its returned to the calling code.
I can convert the Id when its returned in the model using marshmallow's Function field, but using the HyperLinks, the Id is still being output as a byte array string:
class Item(Base):
__tablename__ = 'Items'
Id = Column(LargeBinary, primary_key=True)
def __repr__(self):
return '<Item {}>'.format(self.Name)
class ItemSchema(ma.Schema):
Id = fields.Function(lambda obj: str(UUID(bytes=obj.Id)))
_links = ma.Hyperlinks(
{"url": ma.URLFor("item_detail", id="<Id>"), "collection": ma.URLFor("items")}
)
class Meta:
fields = ("_links", "Id")
Is there a way to format the < Id > that is output in the links?
i.e
{"url": ma.URLFor("item_detail", id="<str(UUID(bytes=Id))>"), "collection": ma.URLFor("items")}
Here's how it is currently output:
{"_links": {"url": "/api/items/b%27%5Cx86%5Cxacx__%5Cxf9%5Cxc2J%5Cx80a6%5Cxa7%5Cx95%5Cx10%5Cx91%5Cxff%27", "collection": "/api/items"}, "Id": "86ac785f-5ff9-c24a-8061-36a7951091ff"}
I want it to look like:
{"_links": {"url": "/api/items/86ac785f-5ff9-c24a-8061-36a7951091ff", "collection": "/api/items"}, "Id": "86ac785f-5ff9-c24a-8061-36a7951091ff"}
I want the link to use the UUID format, not the byte array.
I found out the way around this. It was with the model set up and not marshmallow.
As the database was/is storing the pk as a blob, I was able to use SQLAlchemy_utils UUID column, setting the binary to true, and it all worked:
from sqlalchemy_utils import UUIDType
from uuid import UUID
....
Id = Column(UUIDType(binary=True), primary_key=True)

Marshmallow serialize nested with parent field

Sorry if this has been asked before, I could not actually find a solution or similar question (maybe using the wrong words).
I'm updating an existing Flask API that receives data from a client we don't control (can't change the JSON data format), using marshmallow and peewee.
The data format comes this way:
{
"site_id": "0102931",
"update_date": "2018/02/11-09:33:23",
"updated_by": "chan1",
"crc": "a82131cf232ff120aaf00001293f",
"data": [{"num": 1,
"id": "09213/12312/1",
"chain": "chain2",
"operator": "0000122",
"op_name": "Fred",
"oid": "12092109300293"
},
{"num": 2,
"id": "09213/12312/2",
"chain": "chain1",
"operator": "0000021",
"op_name": "Melissa",
"oid": "8883390393"
}]
}
We are not interested about anything in the main block, but the site_id, which must be copied into each of the objects in the list when deserializing to create the models and store the data.
This is the model in peeewee:
class production_item(db.Model):
site_id = TextField(null=False)
id_prod = TextField(null=False)
num = SmallIntegerField(null=False)
chain = TextField(null=False)
operator = TextField(null=False)
operator_name = TextField(null=True)
order_id = TextField(null=False)
And this is the marshamallow schema:
class prodItemSchema(Schema):
num=String(required=True)
id=String(required=True)
chain=String(required=True)
operator=String(required=True)
op_name=String(required=False, allow_none=True)
oid=String(required=False, allow_none=True)
I can't find a way to pass the site-id from the main structure with load() method and pre-load / post-load decorators for the prodItemSchema, so the model can't be created. Also, I'd like for marshmallow to validate the whole structure for me, not doing in two parts between the resource and the schema, as they are doing in the code right now.
But can't find a way in the documentation to make something like this, is that possible?
In marshmallow it's possible to pass values from a parent scheme to its children before serialization by using the pre_dump decorator on the parent scheme to set the context. Once the context is set, a function field can be used to obtain the value from the parent.
class Parent(Schema):
id = fields.String(required=True)
data = fields.Nested('Child', many=True)
#pre_dump
def set_context(self, parent, **kwargs):
self.context['site_id'] = parent['id']
return data
class Child(Schema):
site_id = fields.Function(inherit_from_parent)
def inherit_from_parent(child, context):
child['site_id'] = context['site_id']
return child

Translating between DRF's serializer and model design

I am using Django Rest Framework to support the use of Annotator.JS(http://annotatorjs.org/) in the frontend of my web application. The problem is the model I use to store any annotations that a user makes is different to the JSON that Annotator.JS sends from the front end in an AJAX request.
The structure of Annotator.JS' JSON is:
{
"id": "39fc339cf058bd22176771b3e3187329", # unique id (added by backend)
"annotator_schema_version": "v1.0", # schema version: default v1.0
"created": "2011-05-24T18:52:08.036814", # created datetime in iso8601 format (added by backend)
"updated": "2011-05-26T12:17:05.012544", # updated datetime in iso8601 format (added by backend)
"text": "A note I wrote", # content of annotation
"quote": "the text that was annotated", # the annotated text (added by frontend)
"uri": "http://example.com", # URI of annotated document (added by frontend)
"ranges": [ # list of ranges covered by annotation (usually only one entry)
{
"start": "/p[69]/span/span", # (relative) XPath to start element
"end": "/p[70]/span/span", # (relative) XPath to end element
"startOffset": 0, # character offset within start element
"endOffset": 120 # character offset within end element
}
],
"user": "alice", # user id of annotation owner (can also be an object with an 'id' property)
"consumer": "annotateit", # consumer key of backend
"tags": [ "review", "error" ], # list of tags (from Tags plugin)
}
The structure of my Annotation model is:
class Annotation(models.Model):
datapoint = models.ForeignKey('datapoint.Datapoint', related_name='%(class)s_parent_datapoint_relation')
owner = models.ForeignKey('users.User', related_name='%(class)s_creator_relation')
# Key fields from the Annotator JSON Format: http://docs.annotatorjs.org/en/latest/annotation-format.html
annotator_schema_version = models.CharField(max_length=8, blank=True)
text = models.TextField(blank=True)
quote = models.TextField()
uri = models.URLField(blank=True)
range_start = models.CharField(max_length=50, blank=True)
range_end = models.CharField(max_length=50, blank=True)
range_startOffset = models.BigIntegerField()
range_endOffset = models.BigIntegerField()
tags = TaggableManager(blank=True)
How can I create a serializer that can translate from the model structure to the JSON?
P.S. Annotator.JS allows the user to send extra information with the JSON structure noted above so the fact that Datapoint isn't included in the JSON structure isn't an issue. This can be passed along without any problem. Owner would hopefully equal User in the JSON.
Thanks for any help, it is greatly appreciated.
Just using a default ModelSerializer should get you serialization for all the simple fields for free (version, text, quote, uri), just by specifying which fields you want serialized. The other fields look straightforward as well:
To compose the ranges object, you can use a SerializerMethodField which lets you define a custom serializer method (return an array of dictionaries containing values from the range_* attributes on your model). Note that if you also need to be able to deserialize JSON into your model you'll need to define custom fields.
To go from owner -> user_id SlugRelatedField (assuming you can access the user id you want from the User object)
You can also use SlugRelatedField to go from TaggableManager -> ["tag", "values"] (again, assuming you can access the value you want through the model object managed by TaggableManager.
In general, everything you want is described in pretty good detail in the documentation.