I'm getting a frustrating intermittent error in Django, when attempting to run the objects.all() for a Queryset that is prefetching. There is an issue where occasionally the model._meta seems to be missing fields, between instantiation of a queryset and running an iteration through it. It's almost as if the queryset's prefetch doesn't actually get run in time for the iteration of the list of objects.
In this example, data.service_log is simply a queryset with some prefetched items called servicelog. When I run the queryset in the shell, I can look at all the fields in the self.names_to_path(lookup_splitted, self.get_meta()) method on the queryset. They are all there, specifically the "servicelog."
Notice this Traceback for this error it says that "servicelog" is not an available field, yet it lists it in the list of fields to choose from. This seems to be a Django bug, but I can't be sure because I can't explain or isolate the behavior. I can't possibly be the only person getting this error. It seems to be in django/db/models/sql/query.py in the names_to_paths() method. Here's the code that is failing to resolve:
query.py names_to_paths():
field = None
try:
field = opts.get_field(name)
except FieldDoesNotExist:
if name in self.annotation_select:
field = self.annotation_select[name].output_field
if field is not None:
# Fields that contain one-to-many relations with a generic
# model (like a GenericForeignKey) cannot generate reverse
# relations and therefore cannot be used for reverse querying.
if field.is_relation and not field.related_model:
raise FieldError(
"Field %r does not generate an automatic reverse "
"relation and therefore cannot be used for reverse "
"querying. If it is a GenericForeignKey, consider "
"adding a GenericRelation." % name
)
try:
model = field.model._meta.concrete_model
except AttributeError:
model = None
else:
# We didn't find the current field, so move position back
# one step.
pos -= 1
if pos == -1 or fail_on_missing:
field_names = list(get_field_names_from_opts(opts))
available = sorted(field_names + list(self.annotation_select))
raise FieldError("Cannot resolve keyword %r into field. "
"Choices are: %s" % (name, ", .join(available)))
break
field does not get set in the first try, then the condition statementif field is not Nonefails so we enter theelseblock. There theposgets reduced by one, but since this field is 'servicelog' it is already at 0. However, when I try this in the shell, it always finds thefieldwithopts.get_field('servicelog')`. Only when run from WSGI and Apache2 does this failure occur. Again, it isn't all the time, which makes it extremely difficult to test. I'm perplexed by this, and am not sure where to look for clues. PLEASE if anybody has ANY ideas of what to explore, I'd be so appreciative.
Traceback (most recent call last):
File "/var/www/fast/services/views/edit.py", line 12897, in service_log
for service in data.service_log:
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1076, in _fetch_all
self._prefetch_related_objects()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 656, in _prefetch_related_objects
prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1457, in prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1556, in prefetch_one_level
prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level)))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/related_descriptors.py", line 802, in get_prefetch_queryset
queryset = queryset._next_is_sticky().filter(**query)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 790, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 808, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1243, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1269, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1149, in build_filter
lookups, parts, reffed_expression = self.solve_lookup_type(arg)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1035, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1330, in names_to_path
"Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword u'servicelog' into field. Choices are: additional_county_worker_notes, adoption_disrupted, adoption_first_name, adoption_last_name, adoption_middle_name, adoption_placement_date, adoption_placement_reason, adoption_placement_reason_id, adoption_placement_type, adoption_risk_level, adoption_risk_level_id, adoption_termination_date, adoption_termination_destination, adoption_termination_reason, adoption_termination_reason_id, adoptive_placement, agency, agency_id, all_items_outstanding, all_items_past_due, appeal_process_date, attached_file, attends_college, attorney_email_address, attorney_email_address_id, attorney_fax, attorney_fax_id, attorney_investigator_email_address, attorney_investigator_email_address_id, attorney_investigator_fax, attorney_investigator_fax_id, attorney_investigator_name, attorney_investigator_phone, attorney_investigator_phone_id, attorney_name, attorney_phone, attorney_phone_id, blood_related_to_applicants, blood_relationship, casa_email_address, casa_email_address_id, casa_fax, casa_fax_id, casa_name, casa_phone, casa_phone_id, certification_items_outstanding, certification_items_past_due, classification, classification_id, client, client_id, county_adoption_worker, county_adoption_worker_cell, county_adoption_worker_cell_id, county_adoption_worker_email_address, county_adoption_worker_email_address_id, county_adoption_worker_fax, county_adoption_worker_fax_id, county_adoption_worker_office, county_adoption_worker_office_id, county_adoption_worker_phone, county_adoption_worker_phone_id, county_adoption_worker_title, county_case_number, county_worker, county_worker_cell, county_worker_cell_id, county_worker_email_address, county_worker_email_address_id, county_worker_fax, county_worker_fax_id, county_worker_office, county_worker_office_id, county_worker_phone, county_worker_phone_id, county_worker_title, court, court_case_name, court_case_number, court_department, court_id, created, current_grade, date_identified_adoptive, date_placed_with_home, deleted, discharge_summary, eligibility_worker, eligibility_worker_email_address, eligibility_worker_email_address_id, eligibility_worker_phone, eligibility_worker_phone_id, emergency_placement, employed_80_hours, enable_discharge_summary, expected_type, expected_type_id, extended_family_contact_allowed, final_payment_amount, finalization_date, foreign_placement, hearing_36626_date, homestudy, id, incident_placement_1, incident_placement_2, incident_placement_3, incident_placement_4, individualized_plan_review, inhousemove, interpretive_summary, item_due, items_approved, items_pending, items_rejected, items_update_requested, la_county_id, medi_cal, medi_cal_eligibility_phone, medi_cal_eligibility_phone_id, medi_cal_eligibility_worker, medi_cal_id, modified, monthly_monitored_visit_hours, mother_child, move_in_type, move_out_type, new_protective_custody_petition, non_minor_dependent, non_truant, notes, number_of_files_required, other_school_contact, other_school_contact_first_name, other_school_contact_last_name, parent_payment_override_annually, parent_payment_override_daily, parent_payment_override_monthly, parental_contact_allowed, parental_group, parental_group_id, payment_amount, percent_certified, percent_items_complete, person_number, placement, placement_date, placement_id, placement_payment_override_annually, placement_payment_override_daily, placement_payment_override_monthly, placement_reason, placement_reason_details, placement_reason_id, placement_self, placer_shelter_bed, prior_placement, progress_summary, projected_adoption_36626_date, projected_adoption_finalization_date, projected_adoption_placement_date, recordreview, requires_educational_support, requires_mental_health_services, respite, school, school_different, school_id, school_liaison_email, school_liaison_first_name, school_liaison_last_name, school_liaison_phone, school_liaison_phone_extension, school_notes, serial_number, servicecontact_onbehalf, servicedeliverylog, servicelog, social_worker_at_termination, social_worker_at_termination_id, special_health_care_needs, state_case_number, teachers, termination_date, termination_destination, termination_reason, termination_reason_details, termination_reason_id, therapist, therapy_code, therapy_supervision_requirements, treatment_abilities, treatment_needs, treatment_preferences, treatment_strengths, treatmentneed, update_requested, update_requested_by, update_requested_by_id, update_requested_date, update_requested_note, updateable, use_number_required, uses_psychotropic_medication, visit_frequency_override, visit_frequency_override_id, visitation_restrictions, who_can_pickup_at_home, who_can_pickup_at_school, who_can_visit
UPDATE - adding models/offending view code
models.py
class ParentalGroup(models.Model):
many fields...
class Placement(models.Model):
parental_group = models.ForeignKey(ParentalGroup, null=True, blank=True)
many more fields...
class ServiceLog(models.Model):
parental_group = models.ForeignKey(ParentalGroup, null=True, blank=True)
placement = models.ManyToManyField(Placement, blank=True)
many more fields...
views.py:
data.service_log = ServiceLog.objects.filter(
parental_group=data.pg,
).prefetch_related(
Prefetch(
'placement',
queryset=Placement.objects.all(),
to_attr='placements'
),
)
for service in data.service_log:
some code to generate data to pass to template...
return render_to_response(...)
I had a similar problem using Django 1.8.6 on Gunicorn/Django runserver. I haven't managed to reproduce the error in shell/notebook environment either.
I solved the randomly occurring FieldError by adding related_name to the ManyToManyField. I used a through model in the ManyToManyField though.
In your case:
class ServiceLog(models.Model):
parental_group = models.ForeignKey(ParentalGroup, null=True, blank=True)
placement = models.ManyToManyField(Placement, blank=True, related_name='servicelog')
Jan
I am trying to create a REST API with Neo4j and Django in the backend.
The problem is that even when I have Django models using Neo4Django , I can't use frameworks like Tastypie or Piston that normally serialize models into JSON (or XML).
Sorry if my question is confusing or not clear, I am newbie to webservices.
Thanks for you help
EDIT: So I started with Tastypie and followed the tutorial on this page http://django-tastypie.readthedocs.org/en/latest/tutorial.html. I am looking for displaying the Neo4j JSON response in the browser, but when I try to access to http://127.0.0.1:8000/api/node/?format=json I get this error instead:
{"error_message": "'NoneType' object is not callable", "traceback": "Traceback (most recent call last):\n\n File \"/usr/local/lib/python2.6/dist-packages/tastypie/resources.py\", line 217, in wrapper\n response = callback(request, *args, **kwargs)\n\n File \"/usr/local/lib/python2.6/dist-packages/tastypie/resources.py\", line 459, in dispatch_list\n return self.dispatch('list', request, **kwargs)\n\n File \"/usr/local/lib/python2.6/dist-packages/tastypie/resources.py\", line 491, in dispatch\n response = method(request, **kwargs)\n\n File \"/usr/local/lib/python2.6/dist-packages/tastypie/resources.py\", line 1298, in get_list\n base_bundle = self.build_bundle(request=request)\n\n File \"/usr/local/lib/python2.6/dist-packages/tastypie/resources.py\", line 718, in build_bundle\n obj = self._meta.object_class()\n\nTypeError: 'NoneType' object is not callable\n"}
Here is my code :
api.py file:
class NodeResource (ModelResource): #it doesn't work with Resource neither
class meta:
queryset= Node.objects.all()
resource_name = 'node'
urls.py file:
node_resource= NodeResource()
urlpatterns = patterns('',
url(r'^api/', include(node_resource.urls)),
models.py file :
class Node(models.NodeModel):
p1 = models.StringProperty()
p2 = models.StringProperty()
I would advise steering away from passing Neo4j REST API responses directly through your application. Not only would you not be in control of the structure of these data formats as they evolve and deprecate (which they do) but you would be exposing unnecessary internals of your database layer.
Besides Neo4Django, you have a couple of other options you might want to consider. Neomodel is another model layer designed for Django and intended to act like the built-in ORM; you also have the option of the raw OGM layer provided by py2neo which may help but isn't Django-specific.
It's worth remembering that Django and its plug-ins have been designed around a traditional RDBMS, not a graph database, so none of these solutions will be perfect. Whatever you choose, you're likely to have to carry out a fair amount of transformation work to create your application's API.
Django-Tastypie allows to create REST APIs with NoSQL databases as well as mentioned in http://django-tastypie.readthedocs.org/en/latest/non_orm_data_sources.html.
The principle is to use tastypie.resources.Resource and not tastypie.resources.ModelResource which is SPECIFIC to RDBMS, then main functions must be redefined in order to provide a JSON with the desired parameters.
So I took the example given in the link, modified it and used Neo4j REST Client for Python to get an instance of the db and perform requests, and it worked like a charm.
Thanks for all your responses :)
Thanks to recent contributions, Neo4django now supports Tastypie out of the box! I'd love to know what you think if you try it out.
EDIT:
I've just run through the tastypie tutorial, and posted a gist with the resulting example. I noticed nested resources are a little funny, but otherwise it works great. I'm pretty sure the gents who contributed the patches enabling this support also know how to take care of nested resources- I'll ask them to speak up.
EDIT:
As long as relationships are specified in the ModelResource, they work great. If anyone would like to see examples, let me know.
Well my answer was a bit vague so I'm gonna post how a solved the problem with some code:
Assume that I want to create an airport resource with some attributes. I will structure this in 3 different files (for readability reasons).
First : airport.py
This file will contain all the resource attributes and a constructor too :
from models import *
class Airport(object):
def __init__ (self, iata, icao, name, asciiName, geonamesId, wikipedia, id, latitude, longitude):
self.icao = icao
self.iata = iata
self.name = name
self.geonamesId = geonamesId
self.wikipedia = wikipedia
self.id = id
self.latitude = latitude
self.longitude = longitude
self.asciiName = asciiName
This file will be used in order to create resources.
Then the second file : AirportResource.py:
This file will contain the resource attributes and some basic methods depending on which request we want our resource to handle.
class AirportResource(Resource):
iata = fields.CharField(attribute='iata')
icao = fields.CharField(attribute='icao')
name = fields.CharField(attribute='name')
asciiName = fields.CharField(attribute='asciiName')
latitude = fields.FloatField(attribute='latitude')
longitude = fields.FloatField(attribute='longitude')
wikipedia= fields.CharField(attribute='wikipedia')
geonamesId= fields.IntegerField(attribute='geonamesId')
class Meta:
resource_name = 'airport'
object_class = Airport
allowed_methods=['get', 'put']
collection_name = 'airports'
detail_uri_name = 'id'
def detail_uri_kwargs(self, bundle_or_obj):
kwargs = {}
if isinstance(bundle_or_obj, Bundle):
kwargs['id'] = bundle_or_obj.obj.id
else:
kwargs['id'] = bundle_or_obj.id
return kwargs
As mentioned in the docs, if we want to create an API that handle CREATE, GET, PUT, POST and DELETE requests, we must override/implement the following methods :
def obj_get_list(self, bundle, **kwargs) : to GET a list of objects
def obj_get(self, bundle, **kwargs) : to GET an individual object
def obj_create(self, bundle, **kwargs) to create an object (CREATE method)
def obj_update(self, bundle, **kwargs) to update an object (PUT method)
def obj_delete(self, bundle, **kwargs) to delete an object (DELETE method)
(see http://django-tastypie.readthedocs.org/en/latest/non_orm_data_sources.html)
Normally, in ModelResource all those methods are defined and implemented, so they can be used directly without any difficulty. But in this case, they should be customized according to what we want to do.
Let's see an example of implementing obj_get_list and obj_get :
For obj_get_list:
In ModelResource, the data is FIRSTLY fetched from the database, then it could be FILTERED according to the filter declared in META class ( see http://django-tastypie.readthedocs.org/en/latest/interacting.html). But I didn't wish to implement such behavior (get everything then filter), so I made a query to Neo4j given the query string parameters:
def obj_get_list(self,bundle, **kwargs):
data=[]
params= []
for key in bundle.request.GET.iterkeys():
params.append(key)
if "search" in params :
query= bundle.request.GET['search']
try:
results = manager.searchAirport(query)
data = createAirportResources(results)
except Exception as e:
raise NotFound(e)
else:
raise BadRequest("Non valid URL")
return data
and for obj_get:
def obj_get(self, bundle, **kwargs):
id= kwargs['id']
try :
airportNode = manager.getAirportNode(id)
airport = createAirportResources([airportNode])
return airport[0]
except Exception as e :
raise NotFound(e)
and finally a generic function that takes as parameter a list of nodes and returns a list of Airport objects:
def createAirportResources(nodes):
data= []
for node in nodes:
iata = node.properties['iata']
icao = node.properties['icao']
name = node.properties['name']
asciiName = node.properties['asciiName']
geonamesId = node.properties['geonamesId']
wikipedia = node.properties['wikipedia']
id = node.id
latitude = node.properties['latitude']
longitude = node.properties['longitude']
airport = Airport(iata, icao, name, asciiName, geonamesId, wikipedia, id, latitude, longitude)
data.append(airport)
return data
Now the third manager.py : which is in charge of making queries to the database and returning results :
First of all, I get an instance of the database using neo4j rest client framework :
from neo4jrestclient.client import *
gdb= GraphDatabase("http://localhost:7474/db/data/")
then the function which gets an airport node :
def getAirportNode(id):
if(getNodeType(id) == type):
n= gdb.nodes.get(id)
return n
else:
raise Exception("This airport doesn't exist in the database")
and the one to perform search (I am using a server plugin, see Neo4j docs for more details):
def searchAirport(query):
airports= gdb.extensions.Search.search(query=query.strip(), searchType='airports', max=6)
if len(airports) == 0:
raise Exception('No airports match your query')
else:
return results
Hope this will help :)
UPDATE: I ended up creating a new forms.Form instead of using a ModelForm
In a ModelForm, I replaced a field by excluding it and adding a new one
with the same name, as shown below in AddRestaurantForm. When saving
the form with the code shown below, I get an error in form.save_m2m()
("Truncated incorrect DOUBLE value"), which seems to be due to the
function to attempt to save the tag field, while it is excluded.
Is the save_m2m() function supposed to save excluded fields?
Is there anything wrong in my code?
Thanks
Jul
(...)
new_restaurant = form.save(commit=False)
new_restaurant.city = city
new_restaurant.save()
tags = form.cleaned_data['tag']
if(tags!=''): tags=tags.split(',')
for t in tags:
tag, created = Tag.objects.get_or_create(name = t.strip())
tag.save()
new_restaurant.tag.add(tag)
new_restaurant.save()
form.save_m2m()
models.py
class Tag(models.Model):
name = models.CharField(max_length=100, unique=True)
class Restaurant(models.Model):
name = models.CharField(max_length=50)
city=models.ForeignKey(City)
category=models.ManyToManyField(Category)
tag=models.ManyToManyField(Tag, blank=True, null=True)
forms.py
class AddRestaurantForm(ModelForm):
name = forms.CharField(widget=forms.TextInput(attrs=classtext))
city = forms.CharField(widget=forms.TextInput(attrs=classtext), max_length=100)
tag = forms.CharField(widget=forms.TextInput(attrs=classtext), required=False)
class Meta:
model = Restaurant
exclude = ('city','tag')
Traceback:
File "/var/lib/python-support/python2.5/django/core/handlers/base.py"
in get_response
92. response = callback(request, *callback_args,
**callback_kwargs) File "/home/jul/atable/../atable/resto/views.py" in addRestaurant
498. form.save_m2m() File "/var/lib/python-support/python2.5/django/forms/models.py" in
save_m2m
75. f.save_form_data(instance, cleaned_data[f.name]) File "/var/lib/python-support/python2.5/django/db/models/fields/
related.py" in save_form_data
967. setattr(instance, self.attname, data) File "/var/lib/python-support/python2.5/django/db/models/fields/
related.py" in set
627. manager.add(*value) File "/var/lib/python-support/python2.5/django/db/models/fields/
related.py" in add
430. self._add_items(self.source_col_name, self.target_col_name, *objs) File
"/var/lib/python-support/python2.5/django/db/models/fields/
related.py" in _add_items
497. [self._pk_val] + list(new_ids)) File "/var/lib/python-support/python2.5/django/db/backends/util.py" in
execute
19. return self.cursor.execute(sql, params) File "/var/lib/python-support/python2.5/django/db/backends/mysql/
base.py" in execute
84. return self.cursor.execute(query, args) File "/var/lib/python-support/python2.5/MySQLdb/cursors.py" in execute
168. if not self._defer_warnings: self._warning_check() File "/var/lib/python-support/python2.5/MySQLdb/cursors.py" in
_warning_check
82. warn(w[-1], self.Warning, 3) File "/usr/lib/python2.5/warnings.py" in warn
62. globals) File "/usr/lib/python2.5/warnings.py" in warn_explicit
102. raise message
Exception Type: Warning at /restaurant/add/ Exception Value:
Truncated incorrect DOUBLE value: 'a'
I see you also posted this same question to Django-users. I'll copy the answer I've posted there:
Firstly, it is no use just giving the name of the error. Errors come
with tracebacks, for good reason: they allow us to see exactly where
the error is occurring, and the context.
Anyway, in your case, there doesn't seem to be any need to use
save_m2m. The documentation states:
"Calling save_m2m() is only required if you use save(commit=False)"
In your case, you've already saved the form to get the new_restaurant
instance, and you're adding tags to that instance with no problem. The
last two calls, to new_restaurant.save() and form.save_m2m(), are unnecessary.
You don't need either of the last two "save" calls. Your tags relation will be implicitly saved by virtue of the add(). I'd just drop those from the code.