There is a view in the Django, for the submit button I say it: printSO
Now, the request is comming to view from two different browsers from the same machine, then how django is handling this?
Question:
Does it use any threading concept to invoke two different executions in parallel?
Considering the below scenario: pseudo code:
def results(request, emp_id):
# if emp_id exists in the database, then delete it.
# send response with message "deleted"
Do we need to have any synchronization mechanism in the above code?
The Django development server is single threaded and not suited for processing more than a request at the same time (I believe this is due to the GIL lock).
However, when combined with a different server , such as Apache, the later handles multithreading (in C). Here is some info (modwsgi) :
Modwsgi
To your final question: no, you don't need to sync anything in most cases
Since Django 1.4 the development server has been multi-threaded
See here
though it is still not a
production level webserver
Related
Consider a POST/PUT REST API (using DRF).
If the server receives request1 and within a couple of ms request2 with identical everything to request1 (duplicate request), is there a way to avoid the request2 to be executed using some Django way? Or Should I deal with it manually by some state?
Any inputs would be much appreciated.
There isn't anything out of the box so you would need to write something your self potentially a piece of custom middleware (https://docs.djangoproject.com/en/3.0/topics/http/middleware/) would be best as then it would run over all of the requests. You would need to capture and exam the requests so you'd need a fast storage of some sort such as a memory store.
You could also look into the python asynco library - https://docs.python.org/3/library/asyncio-sync.html
Another possible solution would be using a FIFO message queue which is configured to support de-duplication based on content. This would turn the request into an deferred process though so it may not be suitable for your needs.
I'm running a system with a few workers that's taking jobs from a message queue, all using Djangos ORM.
In one case I'm actually passing a message along from one worker to another in another queue.
It works like this:
Worker1 in queue1 creates an object (MySQL INSERT) and pushes a message to queue2
Worker2 accepts the new message in queue2 and retrieves the object (MySQL SELECT), using Djangos objects.get(pk=object_id)
This works for the first message. But in the second message worker 2 always fails on that it can't find object with id object_id (with Django exception DoesNotExist).
This works seamlessly in my local setup with Django 1.2.3 and MySQL 5.1.66, the problem occurs only in my test environment which runs Django 1.3.1 and MySQL 5.5.29.
If I restart worker2 every time before worker1 pushes a message, it works fine. This makes me believe there's some kind of caching going on.
Is there any caching involved in Django's objects.get() that differs between these versions? If that's the case, can I clear it in some way?
The issue is likely related to the use of MySQL transactions. On the sender's site, the transaction must be committed to the database before notifying the receiver of an item to read. On the receiver's side, the transaction level used for a session must be set such that the new data becomes visible in the session after the sender's commit.
By default, MySQL uses the REPEATABLE READ isolation level. This poses problems where there are more than one process reading/writing to the database. One possible solution is to set the isolation level in the Django settings.py file using a DATABASES option like the following:
'OPTIONS': {'init_command': 'SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED'},
Note however that changing the transaction isolation level may have other side effects, especially when using statement based replication.
The following links provide more useful information:
How do I force Django to ignore any caches and reload data?
Django ticket#13906
In my django application I am using django-notification to send notifications. However I noticed that in some cases (when sending multiple notifications) my web application is giving delayed responses. Although I am sending notifications through Ajax requests, I still think it would be best if I could implement mailtools library which provide threaded emails.
Has anyone implemented such a thing? Is it easy? How can I use ThreadedMailer from mailtools in django-notification?
or, is there another alternative?
Use Celery for this purpose. It's easy to setup with django and you can use the code you're using right now.
The ajax request puts the email into task queue and returns. You could return your task id if you want to check later if the task succeeded.
Update:
Celery only enables you to call your functions in backgound. Say in ajax view you called:
send_email(…)
Now in tasks.py you should define function:
#task
def send_email(…)
And in the view you will call it by:
send_email.delay(…)
And that's it. The email will be sent by background worker deamon using your existing python code.
This doesn't make django-notification obsolete. Celery does completly different thing and can be used with any lib you can imagine.
The only change is task arguments have to be pickable. It means you have to pass db ids, not whole objects, etc.
Edit
OK I have a long polling from javascript that talks to a django view. The view looks as follows. It loses some messages that I publish from redis client in the channel. Also I should not be connecting to redis for every request (Perhaps the redis variables can be saved in session?)
If someone can point out the changes I need to make this view work with long polling, it would be awesome! Thank you!
def listen (request):
if request.session:
logger.info( 'request session: %s' %(request.session))
channel = request.GET.get('channel', None)
if channel:
logger.info('not in cache - first time - constructing redis object')
r = redis.Redis(host='localhost', port=6379, db=0)
p = r.pubsub()
logger.info('subscribing to channel: %s' %(channel))
p.psubscribe(channel)
logger.info('subscribed to channel: %s' %(channel))
message = p.listen().next()
logger.info('got msg %s' %(message))
return HttpResponse(json.dumps(message));
return HttpResponse('')
----Original question---
I am trying to create a chat application (using django, python) and am trying to avoid the polling mechanism. I have been struggling with this now - so any pointers would be really appreciated!
Since web sockets are not supported in most browsers, I think long polling is the right choice. Right now I am looking for something that scales better than regular polling and is easy to integrate with python django stack. Once I am done with this development, I plan to evaluate other python frameworks (tornado twister, gevent etc.) come to mind.
I did some research and liked the redis pubsub mechanism. The chat message gets published to a channel to which both users have already subscribed to. Following are my questions:
From what I understand, apache would not scale well since long polling would soon run into process/thread limits. Hence I have decided to switch to nginx. Is this rationale correct? Also are there any issues involved in nginx that I am worried about? In particular, I am worried about the latest version not supporting http 1.1 for proxy passing as mentioned in the blog post at http://www.letseehere.com/reverse-proxy-web-sockets?
How do I create the client portion of the subscription of messages on the browser side? In my mind, it would be a url to which the javascript code would "long poll". So at the javascript level, the client would poll a url which gets "blocked" in a "non blocking way" at the server side. When a result (in this case a new chat message) appears, server returns the result. Javascript does what it needs to and then again polls the same url. Is this thinking correct? What happens in between the intervals when the javascript loop is pausing - do we loose any messages from the server side.
In essence, I want to create the following:
From redis, I publish a message to a channel "foo" (can use redis-cli also - easy to incorporate it later in python/django)
I want the same message to appear in two browser windows that use the same js code to poll. Assume that the browser code knows the channel name for test purpose
I publish a second message that again appears in two browser windows.
I am new to real time apps, so apologies for any question that may not make sense.
Thank you!
Well just answering your question partly and mentioning one option out of many: Gunicorn being used with an async worker class is a solution for long-polling/non-blocking requests that is really easy to setup!
I work on manage.py command which creates about 200 threads to check remote hosts. My database setup allows me to use 120 connections, so I need to use some kind of pooling. I've tried using separated thread, like this
class Pool(Thread):
def __init__(self):
Thread.__init__(self)
self.semaphore = threading.BoundedSemaphore(10)
def give(self, trackers):
self.semaphore.acquire()
data = ... some ORM (not lazy, query triggered here) ...
self.semaphore.release()
return data
I pass instance of this object to every check-thread but still getting "OperationalError: FATAL: sorry, too many clients already" inside Pool object after init-ing 120 threads .
I've expected that only 10 database connections will be opened and threads will wait for free semaphore slot. I can check that semaphore works by commenting "release()", in that case only 10 threads will work and other will wait till app termination.
As much as I understand, every thread is opening new connection to database even if actual call is inside different thread, but why? Is there any way to perform all database queries inside only one thread?
Django's ORM manages database connections in thread-local variables. So each different thread accessing the ORM will create its own connection. You can see that in the first few lines of django/db/backends/__init__.py.
If you want to limit the number of database connections made, you must limit the number of different threads that actually access the ORM. A solution could be to implement a service that delegates ORM requests to a pool of dedicated ORM threads. To transmit the requests and their results from and to other threads you will have to implement some sort of message passing mechanism. Since this is a typical producer/consumer problem, the Python docs about threading should give some hints how to achieve this.
Edit: I've just googled for "django connection pooling". There are many people who complain that Django does not provide a proper connection pool. Some of them managed to integrate a separate pooling package. For PostgreSQL, I would take a look at the pgpool middleware.