Django session loses complex object in production, but not in dev - django

So I am successfully storing a complex object (non-model) in my session in development. I've tried every session engine and cache type and they are all working in development (Pycharm). However, when I move the code to production, while no error are thrown, the session losses the object.
Here is the method I use to set the session object:
def instantiate_command_object(request):
try:
ssc = request.session['specimen_search_criteria']
logger.debug('found ssc session variable')
except KeyError:
logger.debug('failed to find ssc session variable')
ssc = SpecimenSearchCommand()
return ssc
Then in a method that runs asynchronously via an ajax call I start making changes to the object in the session:
def ajax_add_collection_to_search(request):
ssc = instantiate_command_object(request)
collection_id = request.GET.get('collection')
collection = Collection.objects.get(pk=collection_id)
if collection and collection not in ssc.collections:
ssc.collections.append(collection)
# save change to session
request.session['specimen_search_criteria'] = ssc
# refresh search results
ssc.search()
return render(request, '_search.html')
All this works as far as it goes. However, if I then refresh the browser, the session is lost. Here is a snippet from the template:
{% with criteria=request.session.specimen_search_criteria %}
<div class="search-criteria" id="search-criteria">
<div class="row">
Sesssion:
{{ request.session }}<br/>
Search:
{{ request.session.specimen_search_criteria }}<br/>
Created:
{{ request.session.specimen_search_criteria.key }}<br/>
Collections:
{{ request.session.specimen_search_criteria.collections }}<br/>
Again, in development I can refresh all day and the same object will be returned. In production, it will either create a new object or occasionally will return a previously created copy.
A few relevant items:
The production server is running Apache httpd with mod_wsgi.
I've tried memcached, databasecache, etc. the behavior remains the same. Always works in development, never in production.
I've tried it with
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'
and without. I can see the session info in the database and when I unpickle it it just seems to be pointing to a location in memory for the complex object.
I'm guessing this might have something to do with running in a multi-user environment, but again, I'm not using locmem and I've tried all of the caching approaches to no effect.
To be clear, the session itself seems to be fine, I can store a string or other simple item in it and it will stick. It's the complex object within the session that seems to be getting lost.
Edit: I might also point out that if I refresh the browser immediately following the return of the search criteria it will actually return successfully. Anything more than about a second and it will disappear.
Edit (adding code of SpecimenSearchCommand):
class SpecimenSearchCommand:
def __init__(self):
pass
created = datetime.datetime.now()
key = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(6))
jurisdictions = []
taxa = []
strata = []
collections = []
chrons = []
has_images = False
query = None # The active SQL query, not the actual result records
page_size = 50
current_page = 1
sort_order = 'number'
results = [] # Page of results from paginator
def is_empty(self):
if len(self.jurisdictions) == 0 and len(self.taxa) == 0 and len(self.strata) == 0 and \
len(self.collections) == 0 and len(self.chrons) == 0 and self.has_images is False:
return True
else:
return False
def get_results(self):
paginator = Paginator(self.query, self.page_size)
try:
self.results = paginator.page(self.current_page)
except PageNotAnInteger:
self.results = paginator.page(1)
except TypeError:
return []
except EmptyPage:
self.results = paginator.page(paginator.num_pages)
return self.results
def get_results_json(self):
points = []
for s in self.results:
if s.locality.latitude and s.locality.longitude:
points.append({"type": "Feature",
"geometry": {"type": "Point",
"coordinates": [s.locality.longitude, s.locality.latitude]},
"properties": {"specimen_id": s.id,
"sci_name": s.taxon.scientific_name(),
"cat_num": s.specimen_number(),
"jurisdiction": s.locality.jurisdiction.full_name()}
})
return json.dumps({"type": "FeatureCollection", "features": points})
def search(self):
if self.is_empty():
self.query = None
return
query = Specimen.objects.filter().distinct().order_by(self.sort_order)
if len(self.taxa) > 0:
query = query.filter(taxon__in=get_hierarchical_search_elements(self.taxa))
if len(self.jurisdictions) > 0:
query = query.filter(locality__jurisdiction__in=get_hierarchical_search_elements(self.jurisdictions))
if len(self.strata) > 0:
query = query.filter(stratum__in=get_hierarchical_search_elements(self.strata))
if len(self.chrons) > 0:
query = query.filter(chron__in=get_hierarchical_search_elements(self.chrons))
if len(self.collections) > 0:
query = query.filter(collection__in=get_hierarchical_search_elements(self.collections))
if self.has_images:
query = query.filter(images__isnull=False)
self.query = query
return
def get_hierarchical_search_elements(elements):
search_elements = []
for element in elements:
search_elements = set().union(search_elements, element.get_descendants(True))
return search_elements

OK, so as Daniel pointed out, the attributes of the SSC class were class-level instead of instance level. The correct version looks like this now:
self.created = datetime.datetime.now()
self.key = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(6))
self.jurisdictions = []
self.taxa = []
self.strata = []
self.collections = []
self.chrons = []
self.has_images = False
self.query = None # The active SQL query, not the actual result records
self.page_size = 50
self.current_page = 1
self.sort_order = 'number'
self.results = [] # Page of results from paginator

Related

Why does the render_template keep on showing the old value of the flask form?

I've been searching for an answer for hours. I apologise if I missed something.
I'm using the same form multiple times in order to add rows to my database.
Every time I check an excel file to pre-fill some of the wtforms StringFields with known information that the user may want to change.
The thing is: I change the form.whatever.data and when printing it, it shows the new value. But when I render the template it keeps showing the old value.
I tried to do form.hours_estimate.data = "" before assigning it a new value just in case but it didn't work.
I will attach here the route I'm talking about. The important bit is after # Get form ready for next service. If there's more info needed please let me know.
Thank you very much.
#coordinator_bp.route("/coordinator/generate-order/<string:pev>", methods=['GET', 'POST'])
#login_required
def generate_order_services(pev):
if not (current_user.is_coordinator or current_user.is_manager):
return redirect(url_for('public.home'))
# Get the excel URL
f = open("./app/database/datafile", 'r')
filepath = f.read()
f.close()
error = None
if GenerateServicesForm().submit1.data and GenerateServicesForm().validate():
# First screen submit (validate the data -> first Service introduction)
form = FillServiceForm()
next_service_row = get_next_service_row(filepath)
if next_service_row is None:
excel_info = excel_get_pev(filepath)
error = "Excel error. Service code not found. If you get this error please report the exact way you did it."
return render_template('coordinator/get_pev_form.html', form=GetPevForm(), error=error, info=excel_info)
service_info = get_service_info(filepath, next_service_row)
service_code = service_info[0]
start_date = service_info[1]
time_estimate = service_info[2]
objects = AssemblyType.get_all()
assembly_types = []
for assembly_type in objects:
assembly_types.append(assembly_type.type)
form.service_code.data = service_code
form.start_date.data = start_date
form.hours_estimate.data = time_estimate
return render_template('coordinator/fill_service_form.html', form=form, error=error, assembly_types=assembly_types)
if FillServiceForm().submit2.data:
if not FillServiceForm().validate():
objects = AssemblyType.get_all()
assembly_types = []
for assembly_type in objects:
assembly_types.append(assembly_type.type)
return render_template('coordinator/fill_service_form.html', form=FillServiceForm(), error=error,
assembly_types=assembly_types)
# Service screen submits
# Here we save the data of the last submit and ready the next one or end the generation process
# Ready the form
form = FillServiceForm()
next_service_row = get_next_service_row(filepath)
if next_service_row is None:
excel_info = excel_get_pev(filepath)
error = "Excel error. Service code not found. If you get this error please report the exact way you did it."
return render_template('coordinator/get_pev_form.html', form=GetPevForm(), error=error, info=excel_info)
service_info = get_service_info(filepath, next_service_row)
service_code = service_info[0]
form.service_code.data = service_code
# create the service (this deletes the service code from the excel)
service = create_service(form, filepath)
if isinstance(service,str):
return render_template('coordinator/fill_service_form.html', form=form, error=service)
# Get next service
next_service_row = get_next_service_row(filepath)
if next_service_row is None:
# This means there is no more services pending
return "ALL DONE"
else:
# Get form ready for next service
service_info = get_service_info(filepath, next_service_row)
service_code = service_info[0]
start_date = service_info[1]
time_estimate = service_info[2]
print("time_estimate")
print(time_estimate) # I get the new value.
objects = AssemblyType.get_all()
assembly_types = []
for assembly_type in objects:
assembly_types.append(assembly_type.type)
form.service_code.data = service_code
form.start_date.data = start_date
form.hours_estimate.data = time_estimate
print(form.hours_estimate.data) # Here I get the new value. Everything should be fine.
# In the html, the old value keeps on popping.
return render_template('coordinator/fill_service_form.html', form=form, error=error,
assembly_types=assembly_types)
number_of_services = excel_get_services(filepath=filepath, selected_pev=pev)
# Get the number of the first excel row of the selected pev
first_row = excel_get_row(filepath, pev)
if first_row is None:
excel_info = excel_get_pev(filepath)
error = "Excel error. PEV not found. If you get this error please report the exact way you did it."
return render_template('coordinator/get_pev_form.html', form=GetPevForm(), error=error, info=excel_info)
service_code = []
start_date = []
time_estimate_code = []
quantity = []
# Open the excel
wb = load_workbook(filepath)
# grab the active worksheet
ws = wb.active
for idx in range(number_of_services):
# Append the data to the lists
service_code.append(ws.cell(row=first_row+idx, column=12).value)
start_date.append(str(ws.cell(row=first_row + idx, column=5).value)[:10])
time_estimate_code.append(ws.cell(row=first_row+idx, column=7).value)
quantity.append(ws.cell(row=first_row + idx, column=9).value)
wb.close()
return render_template('coordinator/generate_services_form.html',
form=GenerateServicesForm(),
pev=pev,
service_code=service_code,
start_date=start_date,
time_estimate_code=time_estimate_code,
quantity=quantity)
Well I found a workarround: I send the data outside the form like this:
return render_template('coordinator/fill_service_form.html', form=form, error=error,
assembly_types=assembly_types,
service_code=service_code,
start_date=start_date,
time_estimate=time_estimate)
And replace the jinja form for this:
<input class="form-control" placeholder="2021-04-23" name="start_date" type="text" value="{{start_date}}">
I'm still using the form (name= the form field name) and at the same time I input the value externally.
I hope this helps somebody.

SqlAlchemy Query.All() Unexpectedly Returning OperationalError

I'm implementing search functionality using Elasticsearch in a "Reddit clone" web application that I'm developing. I want to support searching for threads, users, and subreddits, but when I enter a search query and search for one of the 3 above mentioned categories that does not hold any matches, I'm getting an unexpected "OperationalError" instead of an empty set of results.
As shown in the code I included, I attempted to use the sqlalchemy.orm.query.Query.all() function which returned the following error:
OperationalError: (sqlite3.OperationalError) near "END": syntax error
[SQL: SELECT user.id AS user_id, user.username AS user_username, user.email AS user_email, user.password_hash AS user_password_hash, user.last_sign_in AS user_last_sign_in
FROM user
WHERE 1 != 1 ORDER BY CASE user.id END]
(Background on this error at: http://sqlalche.me/e/e3q8)
I researched other StackOverflow posts and found that the first() function internally processes the database result and returns None if no results are found, but when I switched to that function, I faced this error:
OperationalError: (sqlite3.OperationalError) near "END": syntax error
[SQL: SELECT user.id AS user_id, user.username AS user_username, user.email AS user_email, user.password_hash AS user_password_hash, user.last_sign_in AS user_last_sign_in
FROM user
WHERE 1 != 1 ORDER BY CASE user.id END
LIMIT ? OFFSET ?]
[parameters: (1, 0)]
(Background on this error at: http://sqlalche.me/e/e3q8)
Checking the documentation for SqlAlchemy, I don't see any mention of this error in either function, and reading the meaning of OperationalError, I'm concerned that my database setup is possibly incorrect.
app/routes.py: This is the route that handles search requests made to the following URL: http://localhost:5000/search?q=&index=
#app.route('/search', methods=['GET'])
def search():
print 'Hit the /search route!'
if not g.search_form.validate():
return redirect(url_for('index'))
page = request.args.get('page', 1, type=int)
target_index = request.args.get('index', 'thread')
if target_index == 'thread':
results, total = Thread.search(g.search_form.q.data, page, app.config['POSTS_PER_PAGE'])
print 'Called Thread.search(), total results = {}'.format(total['value'])
elif target_index == 'user':
results, total = User.search(g.search_form.q.data, page, app.config['POSTS_PER_PAGE'])
print 'Called User.search(), total results = {}'.format(total['value'])
elif target_index == 'subreddit':
results, total = Subreddit.search(g.search_form.q.data, page, app.config['POSTS_PER_PAGE'])
print 'Called Subreddit.search(), total results = {}'.format(total['value'])
else:
return render_template('404.html')
try:
results = results.all()
except OperationalError:
results = [None]
total = total['value']
next_url = url_for('search', index=target_index, q=g.search_form.q.data, page=page + 1) if total > page * app.config['POSTS_PER_PAGE'] else None
prev_url = url_for('search', index=target_index, q=g.search_form.q.data, page=page - 1) if page > 1 else None
results_list = zip(results, [None] * len(results)) # Temporarily to match expected input for template
return render_template('search.html', title=_('Search'), results_list=results_list, next_url=next_url, prev_url=prev_url, query=g.search_form.q.data, index=target_index)
app/models.py:
class SearchableMixin(object):
#classmethod
def search(cls, expression, page, per_page):
ids, total = query_index(cls.__tablename__, expression, page, per_page)
if total == 0:
return cls.query.filter_by(id=0), 0
when = []
for i in range(len(ids)):
when.append((ids[i], i))
return cls.query.filter(cls.id.in_(ids)).order_by(
db.case(when, value=cls.id)), total
#classmethod
def before_commit(cls, session):
session._changes = {
'add': list(session.new),
'update': list(session.dirty),
'delete': list(session.deleted)
}
#classmethod
def after_commit(cls, session):
for obj in session._changes['add']:
if isinstance(obj, SearchableMixin):
add_to_index(obj.__tablename__, obj)
for obj in session._changes['update']:
if isinstance(obj, SearchableMixin):
add_to_index(obj.__tablename__, obj)
for obj in session._changes['delete']:
if isinstance(obj, SearchableMixin):
remove_from_index(obj.__tablename__, obj)
session._changes = None
#classmethod
def reindex(cls):
for obj in cls.query:
add_to_index(cls.__tablename__, obj)
db.event.listen(db.session, 'before_commit', SearchableMixin.before_commit)
db.event.listen(db.session, 'after_commit', SearchableMixin.after_commit)
# Below is one model that implements SearchableMixin to allow searching # for users. Thread and Subreddit models follow the same logic.
class User(db.Model, UserMixin, SearchableMixin):
__searchable__ = ['username']
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(64), index=True, unique=True)
# <Remaining User model fields here...>
app/search.py: (Holds the underlying search functions to query Elasticsearch indices)
def add_to_index(index, model):
if not app.elasticsearch:
return
payload = {}
for field in model.__searchable__:
payload[field] = getattr(model, field)
app.elasticsearch.index(index=index, doc_type=index, id=model.id,
body=payload)
def remove_from_index(index, model):
if not app.elasticsearch:
return
app.elasticsearch.delete(index=index, doc_type=index, id=model.id)
def query_index(index, query, page, per_page):
if not app.elasticsearch:
return [], 0
search = app.elasticsearch.search(
index=index,
body={'query': {'multi_match': {'query': query, 'fields': ['*']}},
'from': (page - 1) * per_page, 'size': per_page})
ids = [int(hit['_id']) for hit in search['hits']['hits']]
return ids, search['hits']['total']
As my included app/routes.py shows, I made a workaround by catching the OperationalError and treating it as an indicator that no results were found, but since the all() documentation makes no mention of it, I did not expect there to be this exception being raised.
I simplified the generated query a bit by hiding all fields you retrieve behind the asterisk.
SELECT user.*
FROM user
WHERE 1 != 1
ORDER BY CASE user.id END
First of all, this query will not return any values as long as 1 != 1 is a where clause, since that is false by definition. Is it possible ids is empty? That might also very much explain the ill-formatted CASE statement, which is the source of the error. Normally, case(dict(a=1, b=2), value=User.name) should result in CASE WHEN name = 'a' THEN 1 WHEN name = 'b' THEN 2 END, which would properly execute.

Postgres vs SQL lite page not found discrapancy

I have my dev environment with django - SQLlite and my prod is with django
- Postgres
I have view that works perfectly on SQL lite
login_required
def receipt_pdf(request,pk):
try:
Receipt_payment_id_dict = Receipt_payment.objects.filter(is_active = True ).aggregate(Max('id'))
if Receipt_payment_id_dict:
Receipt_payment_id = Receipt_payment_id_dict['id__max']
Receipt_payment_dict = get_object_or_404(Receipt_payment, pk=Receipt_payment_id)
else:
Receipt_payment_dict = "no data"
except ValueError:
raise Http404("At least one active value should be in -Receipt_payment-, contact your administrator")
try:
general_id_dict = General_configurations.objects.filter(is_active = True ).aggregate(Max('id'))
if general_id_dict:
general_id = general_id_dict['id__max']
general_dict = get_object_or_404(General_configurations, pk=general_id)
else:
general_dict = "no data"
except ValueError:
raise Http404("At least one active value should be in -General_configurations-, contact your administrator")
payment = get_object_or_404(LeasePayment, pk=pk)
# leasetenant = lease.tenant_set.all().order_by('-id')
# Create the HttpResponse object with the appropriate PDF headers.
response = HttpResponse(content_type='application/pdf')
response['Content-Disposition'] = 'attachment; filename="Receipt.pdf"'
buffer = BytesIO()
# Create the PDF object, using the BytesIO object as its "file."
p = canvas.Canvas(buffer)
But When I execute same code with Postgres I get
No Receipt_payment matches the given query.
What could be the problem?

Django pagination while objects are being added

I've got a website that shows photos that are always being added and people are seeing duplicates between pages on the home page (last added photos)
I'm not entirely sure how to approach this problem but this is basically whats happening:
Home page displays latest 20 photos [0:20]
User scrolls (meanwhile photos are being added to the db
User loads next page (through ajax)
Page displays photos [20:40]
User sees duplicate photos because the photos added to the top of the list pushed them down into the next page
What is the best way to solve this problem? I think I need to somehow cache the queryset on the users session maybe? I don't know much about caches really so a step-by-step explanation would be invaluable
here is the function that gets a new page of images:
def get_images_paginated(query, origins, page_num):
args = None
queryset = Image.objects.all().exclude(hidden=True).exclude(tags__isnull=True)
per_page = 20
page_num = int(page_num)
if origins:
origins = [Q(origin=origin) for origin in origins]
args = reduce(operator.or_, origins)
queryset = queryset.filter(args)
if query:
images = watson.filter(queryset, query)
else:
images = watson.filter(queryset, query).order_by('-id')
amount = images.count()
images = images.prefetch_related('tags')[(per_page*page_num)-per_page:per_page*page_num]
return images, amount
the view that uses the function:
def get_images_ajax(request):
if not request.is_ajax():
return render(request, 'home.html')
query = request.POST.get('query')
origins = request.POST.getlist('origin')
page_num = request.POST.get('page')
images, amount = get_images_paginated(query, origins, page_num)
pages = int(math.ceil(amount / 20))
if int(page_num) >= pages:
last_page = True;
else:
last_page = False;
context = {
'images':images,
'last_page':last_page,
}
return render(request, '_images.html', context)
One approach you could take is to send the oldest ID that the client currently has (i.e., the ID of the last item in the list currently) in the AJAX request, and then make sure you only query older IDs.
So get_images_paginated is modified as follows:
def get_images_paginated(query, origins, page_num, last_id=None):
args = None
queryset = Image.objects.all().exclude(hidden=True).exclude(tags__isnull=True)
if last_id is not None:
queryset = queryset.filter(id__lt=last_id)
...
You would need to send the last ID in your AJAX request, and pass this from your view function to get_images_paginated:
def get_images_ajax(request):
if not request.is_ajax():
return render(request, 'home.html')
query = request.POST.get('query')
origins = request.POST.getlist('origin')
page_num = request.POST.get('page')
# Get last ID. Note you probably need to do some type casting here.
last_id = request.POST.get('last_id', None)
images, amount = get_images_paginated(query, origins, page_num, last_id)
...
As #doniyor says you should use Django's built in pagination in conjunction with this logic.

Django formset unit test

I can't run a unit test with formset.
I try to do a test:
class NewClientTestCase(TestCase):
def setUp(self):
self.c = Client()
def test_0_create_individual_with_same_adress(self):
post_data = {
'ctype': User.CONTACT_INDIVIDUAL,
'username': 'dupond.f',
'email': 'new#gmail.com',
'password': 'pwd',
'password2': 'pwd',
'civility': User.CIVILITY_MISTER,
'first_name': 'François',
'last_name': 'DUPOND',
'phone': '+33 1 34 12 52 30',
'gsm': '+33 6 34 12 52 30',
'fax': '+33 1 34 12 52 30',
'form-0-address1': '33 avenue Gambetta',
'form-0-address2': 'apt 50',
'form-0-zip_code': '75020',
'form-0-city': 'Paris',
'form-0-country': 'FRA',
'same_for_billing': True,
}
response = self.c.post(reverse('client:full_account'), post_data, follow=True)
self.assertRedirects(response, '%s?created=1' % reverse('client:dashboard'))
and I have this error:
ValidationError: [u'ManagementForm data is missing or has been
tampered with']
My view :
def full_account(request, url_redirect=''):
from forms import NewUserFullForm, AddressForm, BaseArticleFormSet
fields_required = []
fields_notrequired = []
AddressFormSet = formset_factory(AddressForm, extra=2, formset=BaseArticleFormSet)
if request.method == 'POST':
form = NewUserFullForm(request.POST)
objforms = AddressFormSet(request.POST)
if objforms.is_valid() and form.is_valid():
user = form.save()
address = objforms.forms[0].save()
if url_redirect=='':
url_redirect = '%s?created=1' % reverse('client:dashboard')
logon(request, form.instance)
return HttpResponseRedirect(url_redirect)
else:
form = NewUserFullForm()
objforms = AddressFormSet()
return direct_to_template(request, 'clients/full_account.html', {
'form':form,
'formset': objforms,
'tld_fr':False,
})
and my form file :
class BaseArticleFormSet(BaseFormSet):
def clean(self):
msg_err = _('Ce champ est obligatoire.')
non_errors = True
if 'same_for_billing' in self.data and self.data['same_for_billing'] == 'on':
same_for_billing = True
else:
same_for_billing = False
for i in [0, 1]:
form = self.forms[i]
for field in form.fields:
name_field = 'form-%d-%s' % (i, field )
value_field = self.data[name_field].strip()
if i == 0 and self.forms[0].fields[field].required and value_field =='':
form.errors[field] = msg_err
non_errors = False
elif i == 1 and not same_for_billing and self.forms[1].fields[field].required and value_field =='':
form.errors[field] = msg_err
non_errors = False
return non_errors
class AddressForm(forms.ModelForm):
class Meta:
model = Address
address1 = forms.CharField()
address2 = forms.CharField(required=False)
zip_code = forms.CharField()
city = forms.CharField()
country = forms.ChoiceField(choices=CountryField.COUNTRIES, initial='FRA')
In particular, I've found that the ManagmentForm validator is looking for the following items to be POSTed:
form_data = {
'form-TOTAL_FORMS': 1,
'form-INITIAL_FORMS': 0
}
Every Django formset comes with a management form that needs to be included in the post. The official docs explain it pretty well. To use it within your unit test, you either need to write it out yourself. (The link I provided shows an example), or call formset.management_form which outputs the data.
It is in fact easy to reproduce whatever is in the formset by inspecting the context of the response.
Consider the code below (with self.client being a regular test client):
url = "some_url"
response = self.client.get(url)
self.assertEqual(response.status_code, 200)
# data will receive all the forms field names
# key will be the field name (as "formx-fieldname"), value will be the string representation.
data = {}
# global information, some additional fields may go there
data['csrf_token'] = response.context['csrf_token']
# management form information, needed because of the formset
management_form = response.context['form'].management_form
for i in 'TOTAL_FORMS', 'INITIAL_FORMS', 'MIN_NUM_FORMS', 'MAX_NUM_FORMS':
data['%s-%s' % (management_form.prefix, i)] = management_form[i].value()
for i in range(response.context['form'].total_form_count()):
# get form index 'i'
current_form = response.context['form'].forms[i]
# retrieve all the fields
for field_name in current_form.fields:
value = current_form[field_name].value()
data['%s-%s' % (current_form.prefix, field_name)] = value if value is not None else ''
# flush out to stdout
print '#' * 30
for i in sorted(data.keys()):
print i, '\t:', data[i]
# post the request without any change
response = self.client.post(url, data)
Important note
If you modify data prior to calling the self.client.post, you are likely mutating the DB. As a consequence, subsequent call to self.client.get might not yield to the same data, in particular for the management form and the order of the forms in the formset (because they can be ordered differently, depending on the underlying queryset). This means that
if you modify data[form-3-somefield] and call self.client.get, this same field might appear in say data[form-8-somefield],
if you modify data prior to a self.client.post, you cannot call self.client.post again with the same data: you have to call a self.client.get and reconstruct data again.
Django formset unit test
You can add following test helper methods to your test class [Python 3 code]
def build_formset_form_data(self, form_number, **data):
form = {}
for key, value in data.items():
form_key = f"form-{form_number}-{key}"
form[form_key] = value
return form
def build_formset_data(self, forms, **common_data):
formset_dict = {
"form-TOTAL_FORMS": f"{len(forms)}",
"form-MAX_NUM_FORMS": "1000",
"form-INITIAL_FORMS": "1"
}
formset_dict.update(common_data)
for i, form_data in enumerate(forms):
form_dict = self.build_formset_form_data(form_number=i, **form_data)
formset_dict.update(form_dict)
return formset_dict
And use them in test
def test_django_formset_post(self):
forms = [{"key1": "value1", "key2": "value2"}, {"key100": "value100"}]
payload = self.build_formset_data(forms=forms, global_param=100)
print(payload)
# self.client.post(url=url, data=payload)
You will get correct payload which makes Django ManagementForm happy
{
"form-INITIAL_FORMS": "1",
"form-TOTAL_FORMS": "2",
"form-MAX_NUM_FORMS": "1000",
"global_param": 100,
"form-0-key1": "value1",
"form-0-key2": "value2",
"form-1-key100": "value100",
}
Profit
There are several very useful answers here, e.g. pymen's and Raffi's, that show how to construct properly formatted payload for a formset post using the test client.
However, all of them still require at least some hand-coding of prefixes, dealing with existing objects, etc., which is not ideal.
As an alternative, we could create the payload for a post() using the response obtained from a get() request:
def create_formset_post_data(response, new_form_data=None):
if new_form_data is None:
new_form_data = []
csrf_token = response.context['csrf_token']
formset = response.context['formset']
prefix_template = formset.empty_form.prefix # default is 'form-__prefix__'
# extract initial formset data
management_form_data = formset.management_form.initial
form_data_list = formset.initial # this is a list of dict objects
# add new form data and update management form data
form_data_list.extend(new_form_data)
management_form_data['TOTAL_FORMS'] = len(form_data_list)
# initialize the post data dict...
post_data = dict(csrf_token=csrf_token)
# add properly prefixed management form fields
for key, value in management_form_data.items():
prefix = prefix_template.replace('__prefix__', '')
post_data[prefix + key] = value
# add properly prefixed data form fields
for index, form_data in enumerate(form_data_list):
for key, value in form_data.items():
prefix = prefix_template.replace('__prefix__', f'{index}-')
post_data[prefix + key] = value
return post_data
The output (post_data) will also include form fields for any existing objects.
Here's how you might use this in a Django TestCase:
def test_post_formset_data(self):
url_path = '/my/post/url/'
user = User.objects.create()
self.client.force_login(user)
# first GET the form content
response = self.client.get(url_path)
self.assertEqual(HTTPStatus.OK, response.status_code)
# specify form data for test
test_data = [
dict(first_name='someone', email='someone#email.com', ...),
...
]
# convert test_data to properly formatted dict
post_data = create_formset_post_data(response, new_form_data=test_data)
# now POST the data
response = self.client.post(url_path, data=post_data, follow=True)
# some assertions here
...
Some notes:
Instead of using the 'TOTAL_FORMS' string literal, we could import TOTAL_FORM_COUNT from django.forms.formsets, but that does not seem to be public (at least in Django 2.2).
Also note that the formset adds a 'DELETE' field to each form if can_delete is True. To test deletion of existing items, you can do something like this in your test:
...
post_data = create_formset_post_data(response)
post_data['form-0-DELETE'] = True
# then POST, etc.
...
From the source, we can see that there is no need include MIN_NUM_FORM_COUNT and MAX_NUM_FORM_COUNT in our test data:
MIN_NUM_FORM_COUNT and MAX_NUM_FORM_COUNT are output with the rest of the management form, but only for the convenience of client-side code. The POST value of them returned from the client is not checked.
This doesn't seem to be a formset at all. Formsets will always have some sort of prefix on every POSTed value, as well as the ManagementForm that Bartek mentions. It might have helped if you posted the code of the view you're trying to test, and the form/formset it uses.
My case may be an outlier, but some instances were actually missing a field set in the stock "contrib" admin form/template leading to the error
"ManagementForm data is missing or has been tampered with"
when saved.
The issue was with the unicode method (SomeModel: [Bad Unicode data]) which I found investigating the inlines that were missing.
The lesson learned is to not use the MS Character Map, I guess. My issue was with vulgar fractions (¼, ½, ¾), but I'd assume it could occur many different ways. For special characters, copying/pasting from the w3 utf-8 page fixed it.
postscript-utf-8