I've been trying to learn Django, but I'm still pretty much a web dev newbie, so please bear with me. Maybe something is just fundamentally wrong with this question...
For example, lets say some data exists in a JSON stream that is updated constantly. I'm trying to capture bits of that data and store it in my database, and it's displayed when I visit my Django built page. I guess there's two ways to do this:
In my views.py, it checks the data source, updates the database, and displays the information through a html file. This just seems like it's not the right way to do it. The source would be polled every time the page is viewed.
I would think the correct way to do it is have an application on the server that polls the data source every 1 minute or whatever and updates the database. The views.py only displays information from the database.
Am I even thinking about this correctly? I haven't found any information/examples on how to write the application that would sit on the server and constantly update the database.
Thanks!!
The second way is the right way to go about this, and the application that you would write to poll the json stream does not have to be written with django.
If you want to use the same models for the application, you can implement it as a custom management command, then run the command using cron at an interval. The command would poll the stream, update the database. Your view would then read the database and display the data.
If you want to do this in "realtime" (I use the word realtime here loosely), the server that is hosting the json stream should allow for "push" or a socket connection that will remain open.
Related
I am currently developing a computer based test web app with Django, and I am trying to figure out the best way to persist the choices users make during the course of the test.
I want to persist the choices because they might leave the page due to a crash or something else and I want them to be able to resume from exactly where they stopped.
To implement this I chose saving to Django sessions with db backend which in turns save to the database and this will resolve to a very bad design because I don't want about 2000 users hitting my db every few seconds.
So my question is are there any other ways I can go about implementing this feature that I don't know of. Thanks
If your application is being run on a browser, to be specific, if you are designing a progressive web application, you can make use of browser storage systems, such as localstorage, indexed db, cookies, etc ..
This way, you wouldn't need to send user's updated state back and forth to your backend and you can do the state update based on a specific condition or each n seconds/minutes.
I have a django application that deploys the model logic and data handling through the administration.
I also have in the same project a python file (scriptcl.py) that makes use of the model data to perform heavy calculations that take some time, per example 5 secs, to be processed.
I have migrated the project to the cloud and now I need an API to call this file (scriptcl.py) passing parameters, process the computation accordingly to the parameters and data of the DB (maintained in the admin) and then respond back.
All examples of the django DRF that I've seen so far only contain authentication and data handling (Create, Read, Update, Delete).
Could anyone suggest an idea to approach this?
In my opinion correct approach would be using Celery to perform this calculations asynchronous.
Write a class which inherits from DRF APIView which handles authentication, write whatever logic you want or call whichever function, Get the final result and send back the JsonReposen. But as you mentioned if the Api takes more time to respond. Then you might have to think of some thing else. Like giving back a request_id and hit that server with the request_id every 5seconds to get the data or something like that.
Just to give a feedback to this, the approach that I took was to build another API using flask and normal python scripts.
I also used sqlalchemy to access the database and retrieve the necessary data.
I'll provide more background information first. The question is asked again in the last bullet of "My Thoughts & Questions" section.
Background
I am working on a legacy system which looks kind of like a batch processing system: A job is passed along a series of worker programs that each works on part of the whole job until it is finished. The job status/management information is generated by each worker program along the way and written into local text files with names like "status_info.txt" and "mgmt_info.txt".
This processing system does NOT use any databases, just pure text files on the local file system. Changing their code to use a databases is also kind of expensive which I want to avoid.
I am trying to add a GUI to this system for primarily two purposes. First, let the users view (a read operation) the job status and management information so they can get a big picture of what's going on and whether there is any error in any steps. Second, allow the users to redo one or more steps by changing (a write operation) the job status and management information.
My Current Solution
I am thinking of using Django to develop the GUI because:
Django is fast on development; Django is web-based so almost no installation is required;
I want to enable remote monitoring of the system so a web-based, in-browser GUI makes more sense;
I used some Django before so had some experience.
However, I see Django mostly works with a real database: SQLite, MySQL, PostgreSQL, etc.. The user-defined model will be matched into the tables in these databases by Django automatically. However, the legacy system only produces text files.
My Thoughts & Questions
Fortunately, I noticed that the text files are all in one of the two formats:
Multiple lines of strings;
Multiple lines of key-value pairs.
Both formats look easy to match to a database table design. For example, a "multiple lines of strings" can be considered as a DB table of a single column of text, while a "multiple lines of key-value pairs" as a two-column table.
Therefore, my question is: Can I build my model upon local text files instead of a real database, and somehow override some code somewhere in Django that acts as the interface between the core framework and the external database, so these text files will play the role of a "database" to Django and the reading/writing operations will happen to these files?? I've searched on internet and stackoverflow but wasn't lucky enough. Will appreciate if you can give me some helpful links.
What Not to Do.
If you are going to reproduce an RDBMS using files you are in for a lot and I mean a lot of grief and hard work. Even the simplest RDBMS like sqlite has thousands of man hours of work invested on it. If you were to bring your files into django or any other framework you would need to write a custom backend for it.
What To Do
Create django models backed by an RDBMS and import the files into it. Alternatively since this data appears to be mostly in Key Value pairs, you might be able to use Mongodb or redis.
You can use inotify to monitor the file system to detect when a new file has been created by the batch processing system. When that happens you can invoke a django CLI script to process that file and import it's data into the database.
The rest of it is a straight forward django app.
I am currently trying to figure out he best practice in order to design my web services between a django administrated database (+ images) and a mobile app. My main concern is how to separate a bulk update (send every data in the database and all the files on the server) and a lighter, smaller update with only the new and / or modified objects (images or data.)
I have had access to a working code-base using a cronjob and states for each data field (new, modified, up to date) to generate either a reference data file or an update file. I find it to be very redundant and somewhat unelegant, in contradiction with the DRY spirit of Django (there are tons of lines of code, making it nearly unmaintainable.))
I find it very surprising that this aspect is almost un-documented, since web traffic is a crucial matter in mobile developpment.. Fetching everytime all the data served quickly becomes unsustainable as the database grows..
I would be very grateful for any lead or advice you could give me :-) Thx in advance !
Just have a last_modified DateTimeField in your table, and in your user's profile a last_synchronized DateTimeField. When the mobile app wants to synchronize, send the data which was modified after the last synchronization run, and update the last_synchronized field in the user's profile.
I have a big form on my site. When the users fill it out and submit it, most of the data just gets dumped to the database, and then they get redirected to a new page. However, I'd also like to use the data to query another site, and then parse the results. That might take a bit longer. It's not essential that the user sees these results right away, so I was wondering if it's possible to asynchronously call a function that will handle this, and then return an HttpResponse from my view like usual without making them wait?
If so... how? Any particular libraries I should look at?
User RabbitMQ and Celery with django. If you are deployed on EC2, also look at SQS
You create a message from the request-response cycle and an alternative process or a cron keeps checking off the messages.