Parse large file and paginate / load its parts with scrolling - django

Question

I'm looking for suggestions and the most Django way of loading large variable content (say massive 10,000 lines list) part by part to user page to display only some lines before user asks for more.
This is a detailed scenario (I hope it makes sense to you. It is just a simple example to help dealing with large template variables and pagination):
User goes to website.com/searchfiles which is hosted on my Django backend and returned as a template
searchfiles.html template contains one form with Select drop-down menu to let choose a file that already exists on server (say there are 20 massive log files). Below the drop-down menu there is a text box that allows user to enter a regular expression string. So only two items in the form.
P.S. Each file is usually pretty big e.g. 20-30MB
When user selects the file and enters regular expression in the text box and clicks on "Submit", HTTP POST is made
Django backend receives POST, reads the filename + regexp string and executes function dosearch(FILE, pattern)
dosearch function does something like this:
dosearch(FILE, pattern):
result = []
fh = open(FILE, 'r')
for line in fh:
if re.match(pattern, line):
result.append(line)
return result
Now, result is a list that, depending on pattern, can be pretty large (e.g. 10-20MB). Processing of the file is completed now and I want to present user with "result" variable. After HTTP post, user is redirected to website.com/parsed.
As you can imagine, my goal in step 7 is to return this variable to the user after HTTP POST. But because "result" variable can be huge, I don't want to just dump, say, 10,000 lines of output directly to the page. What I want to achieve is the way that maybe first 200 lines are displayed, and as user is scrolling down, additional 200 lines are loaded once user reaches the bottom of the page.
To keep it simple, ignore the scroll part. User can be also presented with [NEXT] button to click and load additional 200 entries and so on.
What is the most Django way to achieve this? Do I need to save results variable to database and use Ajax?
Also assume that multiple users are going to use the very same page/website so I need to be able to distinguish between two users searching two different files at the same time.
When user navigates out, "result" variable that was generated should be destroyed from the memory.

Answer 1

I can think of two possibilities:
A. using a Model
class ResultLine(models.Model):
user = models.ForeignKey(User)
sequence_number = models.IntegerField()
line = models.CharField(max_length=1000)
created_at = models.DateTimeField(auto_now_add=True)
After parsing the file you would store each result line as a instance of this model, using sequence_number to specify the order of the lines.
In your result view you could use pagination or generic ListView to show the first lines, or use AJAX to fetch more lines.
You would need to add a delete button to clear the users data from this model, or run periodical jobs (maybe using crontab and a custom management command) to delete old result lines.
B. using session data
Another possibility would be to store the result in the users session.
request.session['result_list'] = dosearch(FILE, pattern)
Depending on the session engine there could be size restrictions; this post states that the database-backed sessions are only limited by the database engine (which means you could store many MB or even GB of data in the session).
Also, your server needs sufficient RAM to hold the whole result list of multiple users.
And later in your result view you just read from the session instead from a model.
Performance-wise there are differences: both approaches store the data in the database (with database-backed sessions), but option A allows you to do partial reads in your result view, while option B always reads the whole result list into memory on each request (because the whole session dict is stored in encoded format).