Django 1.9.5: FieldError for reverse lookup fields occasionally - django

Question

I'm getting a frustrating intermittent error in Django, when attempting to run the objects.all() for a Queryset that is prefetching. There is an issue where occasionally the model._meta seems to be missing fields, between instantiation of a queryset and running an iteration through it. It's almost as if the queryset's prefetch doesn't actually get run in time for the iteration of the list of objects.
In this example, data.service_log is simply a queryset with some prefetched items called servicelog. When I run the queryset in the shell, I can look at all the fields in the self.names_to_path(lookup_splitted, self.get_meta()) method on the queryset. They are all there, specifically the "servicelog."
Notice this Traceback for this error it says that "servicelog" is not an available field, yet it lists it in the list of fields to choose from. This seems to be a Django bug, but I can't be sure because I can't explain or isolate the behavior. I can't possibly be the only person getting this error. It seems to be in django/db/models/sql/query.py in the names_to_paths() method. Here's the code that is failing to resolve:
query.py names_to_paths():
field = None
try:
field = opts.get_field(name)
except FieldDoesNotExist:
if name in self.annotation_select:
field = self.annotation_select[name].output_field
if field is not None:
# Fields that contain one-to-many relations with a generic
# model (like a GenericForeignKey) cannot generate reverse
# relations and therefore cannot be used for reverse querying.
if field.is_relation and not field.related_model:
raise FieldError(
"Field %r does not generate an automatic reverse "
"relation and therefore cannot be used for reverse "
"querying. If it is a GenericForeignKey, consider "
"adding a GenericRelation." % name
)
try:
model = field.model._meta.concrete_model
except AttributeError:
model = None
else:
# We didn't find the current field, so move position back
# one step.
pos -= 1
if pos == -1 or fail_on_missing:
field_names = list(get_field_names_from_opts(opts))
available = sorted(field_names + list(self.annotation_select))
raise FieldError("Cannot resolve keyword %r into field. "
"Choices are: %s" % (name, ", .join(available)))
break
field does not get set in the first try, then the condition statementif field is not Nonefails so we enter theelseblock. There theposgets reduced by one, but since this field is 'servicelog' it is already at 0. However, when I try this in the shell, it always finds thefieldwithopts.get_field('servicelog')`. Only when run from WSGI and Apache2 does this failure occur. Again, it isn't all the time, which makes it extremely difficult to test. I'm perplexed by this, and am not sure where to look for clues. PLEASE if anybody has ANY ideas of what to explore, I'd be so appreciative.
Traceback (most recent call last):
File "/var/www/fast/services/views/edit.py", line 12897, in service_log
for service in data.service_log:
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1076, in _fetch_all
self._prefetch_related_objects()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 656, in _prefetch_related_objects
prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1457, in prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1556, in prefetch_one_level
prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level)))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/related_descriptors.py", line 802, in get_prefetch_queryset
queryset = queryset._next_is_sticky().filter(**query)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 790, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 808, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1243, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1269, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1149, in build_filter
lookups, parts, reffed_expression = self.solve_lookup_type(arg)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1035, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1330, in names_to_path
"Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword u'servicelog' into field. Choices are: additional_county_worker_notes, adoption_disrupted, adoption_first_name, adoption_last_name, adoption_middle_name, adoption_placement_date, adoption_placement_reason, adoption_placement_reason_id, adoption_placement_type, adoption_risk_level, adoption_risk_level_id, adoption_termination_date, adoption_termination_destination, adoption_termination_reason, adoption_termination_reason_id, adoptive_placement, agency, agency_id, all_items_outstanding, all_items_past_due, appeal_process_date, attached_file, attends_college, attorney_email_address, attorney_email_address_id, attorney_fax, attorney_fax_id, attorney_investigator_email_address, attorney_investigator_email_address_id, attorney_investigator_fax, attorney_investigator_fax_id, attorney_investigator_name, attorney_investigator_phone, attorney_investigator_phone_id, attorney_name, attorney_phone, attorney_phone_id, blood_related_to_applicants, blood_relationship, casa_email_address, casa_email_address_id, casa_fax, casa_fax_id, casa_name, casa_phone, casa_phone_id, certification_items_outstanding, certification_items_past_due, classification, classification_id, client, client_id, county_adoption_worker, county_adoption_worker_cell, county_adoption_worker_cell_id, county_adoption_worker_email_address, county_adoption_worker_email_address_id, county_adoption_worker_fax, county_adoption_worker_fax_id, county_adoption_worker_office, county_adoption_worker_office_id, county_adoption_worker_phone, county_adoption_worker_phone_id, county_adoption_worker_title, county_case_number, county_worker, county_worker_cell, county_worker_cell_id, county_worker_email_address, county_worker_email_address_id, county_worker_fax, county_worker_fax_id, county_worker_office, county_worker_office_id, county_worker_phone, county_worker_phone_id, county_worker_title, court, court_case_name, court_case_number, court_department, court_id, created, current_grade, date_identified_adoptive, date_placed_with_home, deleted, discharge_summary, eligibility_worker, eligibility_worker_email_address, eligibility_worker_email_address_id, eligibility_worker_phone, eligibility_worker_phone_id, emergency_placement, employed_80_hours, enable_discharge_summary, expected_type, expected_type_id, extended_family_contact_allowed, final_payment_amount, finalization_date, foreign_placement, hearing_36626_date, homestudy, id, incident_placement_1, incident_placement_2, incident_placement_3, incident_placement_4, individualized_plan_review, inhousemove, interpretive_summary, item_due, items_approved, items_pending, items_rejected, items_update_requested, la_county_id, medi_cal, medi_cal_eligibility_phone, medi_cal_eligibility_phone_id, medi_cal_eligibility_worker, medi_cal_id, modified, monthly_monitored_visit_hours, mother_child, move_in_type, move_out_type, new_protective_custody_petition, non_minor_dependent, non_truant, notes, number_of_files_required, other_school_contact, other_school_contact_first_name, other_school_contact_last_name, parent_payment_override_annually, parent_payment_override_daily, parent_payment_override_monthly, parental_contact_allowed, parental_group, parental_group_id, payment_amount, percent_certified, percent_items_complete, person_number, placement, placement_date, placement_id, placement_payment_override_annually, placement_payment_override_daily, placement_payment_override_monthly, placement_reason, placement_reason_details, placement_reason_id, placement_self, placer_shelter_bed, prior_placement, progress_summary, projected_adoption_36626_date, projected_adoption_finalization_date, projected_adoption_placement_date, recordreview, requires_educational_support, requires_mental_health_services, respite, school, school_different, school_id, school_liaison_email, school_liaison_first_name, school_liaison_last_name, school_liaison_phone, school_liaison_phone_extension, school_notes, serial_number, servicecontact_onbehalf, servicedeliverylog, servicelog, social_worker_at_termination, social_worker_at_termination_id, special_health_care_needs, state_case_number, teachers, termination_date, termination_destination, termination_reason, termination_reason_details, termination_reason_id, therapist, therapy_code, therapy_supervision_requirements, treatment_abilities, treatment_needs, treatment_preferences, treatment_strengths, treatmentneed, update_requested, update_requested_by, update_requested_by_id, update_requested_date, update_requested_note, updateable, use_number_required, uses_psychotropic_medication, visit_frequency_override, visit_frequency_override_id, visitation_restrictions, who_can_pickup_at_home, who_can_pickup_at_school, who_can_visit
UPDATE - adding models/offending view code
models.py
class ParentalGroup(models.Model):
many fields...
class Placement(models.Model):
parental_group = models.ForeignKey(ParentalGroup, null=True, blank=True)
many more fields...
class ServiceLog(models.Model):
parental_group = models.ForeignKey(ParentalGroup, null=True, blank=True)
placement = models.ManyToManyField(Placement, blank=True)
many more fields...
views.py:
data.service_log = ServiceLog.objects.filter(
parental_group=data.pg,
).prefetch_related(
Prefetch(
'placement',
queryset=Placement.objects.all(),
to_attr='placements'
),
)
for service in data.service_log:
some code to generate data to pass to template...
return render_to_response(...)

Answer 1

I had a similar problem using Django 1.8.6 on Gunicorn/Django runserver. I haven't managed to reproduce the error in shell/notebook environment either.
I solved the randomly occurring FieldError by adding related_name to the ManyToManyField. I used a through model in the ManyToManyField though.
In your case:
class ServiceLog(models.Model):
parental_group = models.ForeignKey(ParentalGroup, null=True, blank=True)
placement = models.ManyToManyField(Placement, blank=True, related_name='servicelog')
Jan