I'm using JSONField in some of my Django models and would like to migrate this data from Oracle to Postgres.
So far I haven't had any luck keeping this JSON data intact when using Django's dumpdata and loaddata commands, the data is transformed into string representations of the JSON. I've yet to find a good solution to this... Ideas?
I ended up solving this problem by overriding Django's included JSON serializer, specifically the handle_field method, in a custom serializer file called custom_json_serializer.py. By doing this I can ensure that specific JSONFields stay as is, without being converted to string.
On the chance anyone else runs into this issue, these are the steps I took. I had to add this custom serializer to the settings.py file:
SERIALIZATION_MODULES = {
'custom_json': 'myapp.utils.custom_json_serializer',
}
and then call it when serializing the data from Django:
python manage.py dumpdata mymodel --format=custom_json --indent=2 --traceback > mymodel_data.json
The custom serializer looks like:
from django.core.serializers.json import Serializer as JSONSerializer
from django.utils.encoding import is_protected_type
# JSONFields that are normally incorrectly serialized as strings
json_fields = ['problem_field1', 'problem_field2']
class Serializer(JSONSerializer):
"""
A fix on JSONSerializer in order to prevent stringifying JSONField data.
"""
def handle_field(self, obj, field):
value = field._get_val_from_obj(obj)
# Protected types (i.e., primitives like None, numbers, dates,
# and Decimals) are passed through as is. All other values are
# converted to string first.
if is_protected_type(value) or field.name in json_fields:
self._current[field.name] = value
else:
self._current[field.name] = field.value_to_string(obj)
The really strange part is that before this fix some JSONFields were serializing just fine, while others were not. That is why I took the approach of specifying the fields to be handled. Now all data is serializing correctly.
I haven't used the JSONField before, but what I do is:
import json
data_structure = json.loads(myData)
Maybe that will work for what you need as well. There's likely a better way to deal with this.
EDIT: If you end up using the package json - only then is the following solution applicable.
If you are using Python 2.6 and above you can use:
import json
otherwise, you can use the simplejson that is bundled with django.utils (for Python < 2.6).
from django.utils import simplejson as json
That way you can continue to use the same package name, and take your code to Google App Engine as it supports Python 2.5.2 at the moment.
Related
I want to save the python dictionary inside the Django model as JSON and I want to convert that JSON back into a python dictionary when that data is fetched.
I know I can do it inside view but I want to implement it in the model so it can return dictionary object when queried.
is there any signal or any post_fetch method that I can use to achieve it, I couldn't find anything googling it...
You might want to look into simply using the Python JSON package, unless you are using Postgres as your database - in which case JSONField is the way to go. The json package is explained here and you could use it in a model like so if I understand what you are saying:
import json
class MyModel(models.Model):
json_field = models.TextField() # or you can use JSONField if using Postgres
#property
def get_json_field_as_dictionary(self):
return json.loads(self.json_field)
def set_json_field(self, mydict):
self.json_field = json.dumps(mydict)
#classmethod
def get_json_from_dictionary(cls, mydict):
return json.dumps(mydict)
When you are saving to the database, you can use json.dumps(myDictionary) or convert the dictionary to JSON by calling MyModelObject.set_json_field(myDictionary) to convert a Python dictionary to JSON and then store it in the json_field of the model. To retrieve the JSON data as a dictionary, you simply call MyModel.objects.last().get_json_field_as_dictionary (or whatever you prefer to call it, json_dictionary perhaps so it would be MyModel.objects.last().json_dictionary) and it will return the value of that property as if it were an element in the model without having to do the conversion each time.
Or, if you are using Postgres as your backend, this is a lot easier with JSONField:
class MyModel(models.Model):
json_field = models.JSONField(null=True)
And to save:
myObject = myModel.objects.create(json_field=myDictionary)
If you clarify I can update my answer to explain better.
You may want to use JSONField. It allows storing data encoded as JSON and retrieve them as the corresponding Python data type, including dictionary.
JSONField only works on PostgreSQL with Django < 3.1, but works on any database with Django >= 3.1.
I've developed a complex data analysis model using Python (for the sake of simplicity, let's call it analysis.py) which contains multiple long algorithms to transform a set of raw data and spits out output in the form of dataframe as well.
I've been looking into some tutorials recently on Django framework and have tried building a simple web app that resembles a forum/blog post web app.
With the dataframe output from the analysis.py, I would like to display the data in the form of multiple charts on the web app.
Can anyone point me to a direction to move from here? I've looked for multiple resources online but I think I am unable to find the correct keywords that match with what I actually want to build. I just want a shell code kind of a thing to integrate into my current django backend in order to be able to call the dataframe from the analysis.py
Thank you very much in advance.
It amazes me somehow that after posting a question here, I managed to come up with better keywords in finding the solution that is closely matched to my intended product.
Again, I would like to apologize if my question was too vague or did not contain any script that I've done so far, since I was merely looking for direction to move forward. And fortunately I did after giving it much thought and integrating different solutions from multiple sources on the net.
I'm not sure if my solution is the best solution, but it works for me and if it at least helps other people who were on the same boat as I did before, I'd be glad!
So my solution entails importing a dataframe from the analysis.py and then pass the data that I want as API Endpoints (using Django Rest Framework) in order to be displayed in my dashboard.
The following are some details (with some scripts) of my solution:
views.py
# Importing the external python script (analysis.py).
#This is where the data from csv is imported and is transformed.
#The script is written in a function form and returns the dataframe.
from . import analysis
from .analysis import data_transform
#Django Rest Framework
from rest_framework.views import APIView
from rest_framework.response import Response
from django.http import HttpResponse, JsonResponse
from django.shortcuts import render, redirect
class ChartData(APIView):
authentication_classes = []
permission_classes =[]
def get(self, request, format=None):
# Assign dataframe output to a new dataframe
df = data_transform()
# Assigning variables from datafram
data1 = df['VOL'].values.tolist()
data2 = df['PRICE'].values.tolist()
data = {
"data1":data1,
"data2":data2,
}
#Return endpoints in Json format
return Response(data)
Analysis.py looks more like this
import pandas as pd
import numpy as np
def data_transform()
#where all the transformations happened
return dataframe
And then I passed the endpoints ('data') to a html in order to visualize them using chart.js. You can find some good resources in the Tube. Look up keywords chart.js and django.
Again, I'm not sure this is the best method. I am open to a better solution, if you guys have any!
What is/are the best practices to use get_model() and when should it be imported ?
Ref: https://docs.djangoproject.com/en/1.8/ref/applications/
You usually use get_model() when you need to dynamically get a model class.
A practical example: when writing a RunPython operation for a migration, you get the app registry as one of the args, and you use apps.get_model('TheModel') to import historical models.
Another example: you have an app which has dynamically built serializers and you set their Meta.model to the class you just got with get_model() .
Yet another example is importing models in AppConfig.ready() with self.get_model().
An important thing to remember, if you are using AppConfig.get_model() or apps.get_models(), that they can be used only once the application registry is fully populated.
The other option (from .models import TheModel) is just the default way to import models anywhere in your code.
These are just examples though, there are many other possible scenarios.
I Prefer, use .models import, cause is a simple way to get the Model Object.
But if you works with metaclasses, maybe the get_model, would be the best option.
def get_model(self, app_label, model_name=None):
"""
Returns the model matching the given app_label and model_name.
As a shortcut, this function also accepts a single argument in the
form <app_label>.<model_name>.
model_name is case-insensitive.
Raises LookupError if no application exists with this label, or no
model exists with this name in the application. Raises ValueError if
called with a single argument that doesn't contain exactly one dot.
"""
self.check_models_ready()
if model_name is None:
app_label, model_name = app_label.split('.')
return self.get_app_config(app_label).get_model(model_name.lower())
Maybe this SO POST, can help too.
Example:
class MyModel(models.Model):
field1=models.CharField(..)
field2=models.DateTimeField()
def today(self):
return self.field2
When I look at this in the admin site, field2 is formatted differently than the today field.
How can I tell the admin site to treat today like it's treating field2? I.e., tell Django admin that 'today' is a models.DateTimeField?
Here is what it's showing:
Field2 today
April 5, 2011, 9:10 a.m. 2011-04-11 08:47:27
To obtain DateTime object call datetime.datetime.now() instead of datetime.datetime.today()
EDIT:
Or use models.DateField() for field2 instead of models.DateTimeField() :-)
EDIT2:
Here is the solution:
def today(self):
from django.utils import formats
return formats.localize(self.field2)
That's some really really weird behaviour. At a total guess, it may have something to do with django settings; specifically the DATETIME_FORMAT (and related) settings. The framework probably does introspection on fields, and if they are of DateTime type, are rendered according to the aforementioned settings.
Introspection on methods wouldn't make sense in the majority of cases, so I could understand this behaviour if it is the case.
Try modifying the settings accordingly (provide different datetime formats), and see if the fields change and the method remains the same.
Edit:
Looking at django.contrib.databrowse.datastructures, there is a section of code that does something like:
if isinstance(self.field, models.DateTimeField):
objs = capfirst(formats.date_format(self.raw_value, 'DATETIME_FORMAT'))
I'd imagine a similar thing happening within the admin app, though I can't find an exact reference at the moment.
To achieve what you want, you'll need to format your datetime appropriately:
def today(self):
from django.conf import settings
return self.field2.strftime(settings.DATETIME_FORMAT)
Or, using #cata's comment:
def today(self):
from django.utils.formats import localize
return localize(self.field2)
If you choose to supply a "list_display" item through your own function, and you're not happy with the default output, you'll need to format it yourself. In this case, if you want to have identical formatting to what the DateTime database field ends up with:
from django.utils import formats
def today(self):
return formats.localize(self.field2)
Background:
templates/admin/change_list.html
uses the template tag
django.contrib.admin.templatetags.admin_list.result_list
which in turn will call
django.contrib.admin.templatetags.admin_list.items_for_result()
to render the individual column values for each row.
You'll see that both your values start off being a DateTime instance, either through database lookup or calling your function, see
django.contrib.admin.util.lookup_field()
but the return value "f" will only be a field if there was a database field. You provided a function, so lookup_field() will only provide the value, and "f" will be None.
So in items_for_result(), your value will run through the "if f is None" block and miss out on
result_repr = display_for_field(value, f)
In other words,
django.contrib.admin.util.display_for_field()
will only be called on the database value to format according to the field type, so this is the treatment your function value is missing out on:
elif isinstance(field, models.DateField) or isinstance(field, models.TimeField):
return formats.localize(value)
and you'll need to do that last line yourself, as shown above.
EDIT: Regarding your question
How can I tell the admin site to treat
today like it's treating field2? I.e.,
tell Django admin that 'today' is a
models.DateTimeField?
It's not a models.DateTimeField, it's a function value. If it were a models.DateTimeField, it would be describing your model. Look at all the stuff that entails: http://docs.djangoproject.com/en/dev/ref/models/fields/
In your example, you really could just use field2. Apparently you want to do things to its value, calculate it etc. - so what's today.db_column then?
That said, it would be nice if function values that are DateTime instances were run through format.localize() by default, as that's what the documentation on localization seems to be promising.
By the way, I would rather define a formatted value in the ModelAdmin than in the model itself. I usually call it something like "formatted_today" (to keep the datetime value of the original today()), it's just that if the Admin is the only place that needs the formatted value, imho that's where it should be defined.
All previous answers provide solutions, that will handle timezone info incorrectly in new Django versions.
field2 and today will generally show different time if settings.USE_TZ==True.
I found this question today and have spent some time to figure out the correct way:
from django.db import models
from django.contrib import admin
from django.utils.timezone import template_localtime
from django.utils.formats import localize
class MyModel(models.Model):
# ...
field2=models.DateTimeField()
class MyModelAdmin(admin.ModelAdmin):
# ...
def today(self, obj):
return localize(template_localtime(obj.field2))
admin.site.register(MyModel, MyModelAdmin)
I've a large number of models (120+) and I would like to let users of my application export all of the data from them in XML format.
I looked at django-piston, but I would like to do this with minimum code. Basically I'd like to have something like this:
GET /export/applabel/ModelName/
Would stream all instances of ModelName in applabel together with it's tree of related objects .
I'd like to do this without writing code for each model.
What would be the best way to do this?
The standard django dumpdata command is not flexible enough to export single models. You can use the makefixture command to do that http://github.com/ericholscher/django-test-utils/blob/master/test_utils/management/commands/makefixture.py
If I'd have to do this, as a basic starting point I'd start from something like:
from django.core.management import call_command
def export_view(request, app_label, model_slug):
# You can do some stuff here with the specified model if needed
# ct = ContentType.objects.get(app_label=app_label, model=model_slug)
# model_class = ct.model_class()
# I don't know if this is a correct specification of params
# command line example: python manage.py makefixture --format=xml --indent=4 YourModel[3] auth.User[:5]
# You'll have to test it out and tweak it
call_command("makefixture", "file.xml", '%s.%s[:]' % (app_label, model_slug), format='xml')