Store list or collection in Apache Avro schema - list

Question

I'm currently creating Avro schema to store twitter data streams.
My data source in JSON:
{
'id': '123456789',
'text': 'bla bla bla...',
'entities': {
'hashtags': [{'text':'hashtag1'},{'text':'hashtag2'}]
}
}
in Cassandra, I can define collection (sets or lists) to store hashtags data.
But I have no idea how to define this structure in Apache Avro.
Here's my best try:
{"namespace": "ln.twitter",
"type": "record",
"name": "main",
"fields": [
{"name": "id","type": "string"},
{"name": "text","type": "string"},
{"name": "hashtags","type": "string"} // is there any better format for this ?
]
}
Need your advice please.
Thanks,
Yusata.

Answer 1

The entities field needed explicit records (or maps) inside. Here's a schema that should work:
{
"type": "record",
"name": "Main",
"fields": [
{
"name": "id",
"type": "string"
},
{
"name": "text",
"type": "string"
},
{
"name": "entities",
"type": {
"type": "record",
"name": "Entities",
"fields": [
{
"name": "hashtags",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "Hashtag",
"fields": [
{
"name": "text",
"type": "string"
}
]
}
}
}
]
}
}
]
}
In case it's helpful, you can use this tool to generate an (anonymous) Avro schema from any valid JSON record. You'll then just need to add names to the record types.
You can try it on your example after switching its ' to ":
{
"id": "123456789",
"text": "bla bla bla...",
"entities": {"hashtags": [{"text": "hashtag1"}, {"text": "hashtag2"}]}
}