I would like to use a thread that starts inside of a Django application.
If we use a standart Python thread it could be stopped by a webserver when the request is finished.
Is there a standard way to do this? Or is there a Django library available that provides this functionality?
I use threads intensively for long processes. A better solution is Celery, of course.
To define thread:
from threading import Thread
class afegeixThread(Thread):
def __init__ (self,usuari, expandir=None, alumnes=None,
impartir=None, matmulla = False):
Thread.__init__(self)
self.expandir = expandir
self.alumnes = alumnes
self.impartir = impartir
self.flagPrimerDiaFet = False
self.usuari = usuari
self.matmulla = matmulla
def run(self):
errors = []
try:
...
self.flagPrimerDiaFet = ...
...
def firstDayDone(self):
return self.flagPrimerDiaFet
Calling thread:
from presencia.afegeixTreuAlumnesLlista import afegeixThread
afegeix=afegeixThread(expandir = expandir, alumnes=alumnes,
impartir=impartir, usuari = user, matmulla = matmulla)
afegeix.start()
#Waiting for first day done before return html:
import time
while afegeix and not afegeix.firstDayDone(): time.sleep( 0.5 )
#return html code
return HttpResponseRedirect('/presencia/passaLlista/%s/'% pk )
Related
I created a task in the admin that is costly and should be carried out asynchronously. I would do something like
def costly_task(**kwargs):
def do_task(id):
## do stuff, you know, that is costly
task = ThreadTask.objects.get(pk = id)
task.is_done = True
task.save()
task = ThreadTask()
task.save()
t = threading.Thread(target = do_task, args = [task.id])
t.setDaemon(True)
t.start()
return {"id": task.id}
With a models.py table:
class ThreadTask(models.Model):
task = models.CharField(max_length = 30, blank = True, null = True)
is_done = models.BooleanField(blank = False, default = False)
This works well, but I want to inform the admin of the runnig task, and also when it is finished. There is a very old (django 4.0 incompatible) package called django-async-messages which leverages the normal django-messages package to be used asynchronously. I googled but did not find anything of newer age ... any ideas on how to do this? Can I use djangos async processes to send out two messages:
when task started (that one is easy)
when task finished
I found something similar, but not quite what I am looking for because this would require a lot of implementation effort: Django async send notifications
I am trying to build a timer inside a get method in a DRF View. I have created the timer method inside the GameViewController class and what I am trying to achieve is that a every minute (5 times in a row) a resource object is shown to the user through the get request and a game round object is created. My View works at the moment, however the timer doesn't seem to be doing anything.
I know this isn't exactly how things are done in django but this is how I need to do it for my game API for game logic purposes.
How can I make the timer work? Do I need to use something like request.time or such?
Thanks in advance.
views.py
class GameView(APIView):
def get(self, request, *args, **kwargs):
...
round_number = gametype.rounds
# time = controller.timer()
now = datetime.now()
now_plus_1 = now + timedelta(minutes=1)
while round_number != 0:
while now < now_plus_1:
random_resource = Resource.objects.all().order_by('?').first()
resource_serializer = ResourceSerializer(random_resource)
gameround = Gameround.objects.create(
id=controller.generate_random_id(Gameround),
user_id=current_user_id,
gamesession=gamesession,
created=datetime.now(),
score=current_score
)
gameround_serializer = GameroundSerializer(gameround)
round_number -= 1
return Response({# 'gametype': gametype_serializer.data,
'resource': resource_serializer.data,
'gameround': gameround_serializer.data
})
If you want to jump quickly into this - use huey https://github.com/coleifer/huey
You will need to install Redis as a backend of your queue. It's not complicated.
Huey can run your code by cron, delays or something else complicated:
from huey import RedisHuey, crontab
huey = RedisHuey('my-app', host='redis.myapp.com')
#huey.task()
def add_numbers(a, b):
return a + b
#huey.task(retries=2, retry_delay=60)
def flaky_task(url):
# This task might fail, in which case it will be retried up to 2 times
# with a delay of 60s between retries.
return this_might_fail(url)
#huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
sync_all_data()
Hyue has the Django extentions https://huey.readthedocs.io/en/latest/contrib.html#django
As for me, this was the fastest way to achieve the same tasks and this has been working in production for ~1 year without my support.
I'm currently running on one python Gui development based on Pyqt5. There is quite a lot of file system related operations such as copy, move and delete. I tried to enhance the performance of these operations by utilizing concurrent.futures module and experimented on both Threadpoolexecutor and Processpoolexecutor. However, neither of them could give me a unblocking gui which would be able to go on with user interaction after I submitted the file operations to the pool. These submitted operations are simply static methods of native python classes. I also added callbacks through the Future objects returned by submissions, which are the instance methods of one qt class and which did get called when the corresponding submitted job had completed.
Just unable to figure out a way to solve the problem of freezed ui(even just for seconds' duration) Could anybody help?
from concurrent.futures import ProcessPoolExecutor
class ArchiveResManager:
def __init__(self, resource_dir):
self._path_reference_dir = resource_dir
def get_resource_list(self):
return tuple(
de.path for de in os.scandir(self._path_reference_dir) if de.is_file()
)
def add_callback_resource_added(fn):
self._callback_on_resource_added = fn
def add_resources(self, resources: iter):
src_paths = tuple(p for p in resources)
dest_paths = tuple(self._get_dest_path(p) for p in resources)
with ProcessPoolExecutor(2) as executor:
exe.submit(
ArchiveResManager._do_resource_operations,
src_paths,
dest_paths
).add_done_callback(self._on_resource_added)
#staticmethod
def _do_resource_operations(src_paths, dest_paths):
added_items = list()
for src_path, dest_path in zip(src_paths, dest_paths):
shutil.copyfile(
src_path, dest_path
)
added_items.append(dest_path)
return added_items
def _on_resource_added(self, future):
if future.done():
if self._callback_on_resource_added:
self._callback_on_resource_added(future.result())
# This model class is bound to a instance of QListView in the QDialog of my project
class ArchiveModel(QAbstractListModel):
def __init__(self, archive_dir, parent):
super().__init__(parent)
self._archive_manager = ArchiveResManager(archive_dir)
self._archive_manager.add_callback_resource_added(self._on_archive_operations_completed)
self._archive_items = self._archive_manager.get_resource_list()
def _on_archive_operations_completed(self, result_list):
last_row = self.rowCount() - 1
self.beginInsertRows(QModelIndex(), last_row, last_row + len(result_list))
self._archive_items.extend(result_list)
self.endInsetRows()
I am currently developing a Django application based on django-tenants-schema. You don't need to look into the actual code of the module, but the idea is that it has a global setting for the current database connection defining which schema to use for the application tenant, e.g.
tenant = tenants_schema.get_tenant()
And for setting
tenants_schema.set_tenant(xxx)
For some of the tasks I would like them to remember the current global tenant selected during the instantiation, e.g. in theory:
class AbstractTask(Task):
'''
Run this method before returning the task future
'''
def before_submit(self):
self.run_args['tenant'] = tenants_schema.get_tenant()
'''
This method is run before related .run() task method
'''
def before_run(self):
tenants_schema.set_tenant(self.run_args['tenant'])
Is there an elegant way of doing it in celery?
Celery (as of 3.1) has signals you can hook into to do this. You can alter the kwargs that were passed in, and on the other side, undo your alterations before they're given to the actual task:
from celery import shared_task
from celery.signals import before_task_publish, task_prerun, task_postrun
from threading import local
current_tenant = local()
#before_task_publish.connect
def add_tenant_to_task(body=None, **unused):
body['kwargs']['tenant_middleware.tenant'] = getattr(current_tenant, 'id', None)
print 'sending tenant: {t}'.format(t=current_tenant.id)
#task_prerun.connect
def extract_tenant_from_task(kwargs=None, **unused):
tenant_id = kwargs.pop('tenant_middleware.tenant', None)
current_tenant.id = tenant_id
print 'current_tenant.id set to {t}'.format(t=tenant_id)
#task_postrun.connect
def cleanup_tenant(**kwargs):
current_tenant.id = None
print 'cleaned current_tenant.id'
#shared_task
def get_current_tenant():
# Here is where you would do work that relied on current_tenant.id being set.
import time
time.sleep(1)
return current_tenant.id
And if you run the task (not showing logging from the worker):
In [1]: current_tenant.id = 1234; ct = get_current_tenant.delay(); current_tenant.id = 5678; ct.get()
sending tenant: 1234
Out[1]: 1234
In [2]: current_tenant.id
Out[2]: 5678
The signals are not called if no message is sent (when you call the task function directly, without delay() or apply_async()). If you want to filter on the task name, it is available as body['task'] in the before_task_publish signal handler, and the task object itself is available in the task_prerun and task_postrun handlers.
I am a Celery newbie, so I can't really tell if this is the "blessed" way of doing "middleware"-type stuff in Celery, but I think it will work for me.
I'm not sure what you mean here, is before_submit executed before the task is called by a client?
In that case I would rather use a with statement here:
from contextlib import contextmanager
#contextmanager
def set_tenant_db(tenant):
prev_tenant = tenants_schema.get_tenant()
try:
tenants_scheme.set_tenant(tenant)
yield
finally:
tenants_schema.set_tenant(prev_tenant)
#app.task
def tenant_task(tenant=None):
with set_tenant_db(tenant):
do_actions_here()
tenant_task.delay(tenant=tenants_scheme.get_tenant())
You can of course create a base task that does this automatically,
you can apply the context in Task.__call__ for example, but I'm not sure
if that saves you much if you can just use the with statement explicitly.
I currently have a site setup using Django. I have added Gevent Socketio to add a chat function. I have a need to scale it as there are quite a few users already on the site and can't find a way to do so.
I tried https://github.com/abourget/gevent-socketio/tree/master/examples/django_chat/chat
I am using Gunicorn & the socketio.sgunicorn.GeventSocketIOWorker worker class so at first I thought of increasing the worker count. Unfortunately this seems to fail intermittently. I have started rewriting it to use redis from a few sources I found and have 1 worker on each server which is now being load balanced. However this seems to have the same problem. I am wondering if there is some issue in the gevent socketio code itself which does not allow it to scale.
Here is how I have started which is just the submit message code.
def redis_client():
"""Get a redis client."""
return Redis(settings.REDIS_HOST, settings.REDIS_PORT, settings.REDIS_DB)
class PubSub(object):
"""
Very simple Pub/Sub pattern wrapper
using simplified Redis Pub/Sub functionality.
Usage (publisher)::
import redis
r = redis.Redis()
q = PubSub(r, "channel")
q.publish("test data")
Usage (listener)::
import redis
r = redis.Redis()
q = PubSub(r, "channel")
def handler(data):
print "Data received: %r" % data
q.subscribe(handler)
"""
def __init__(self, redis, channel="default"):
self.redis = redis
self.channel = channel
def publish(self, data):
self.redis.publish(self.channel, simplejson.dumps(data))
def subscribe(self, handler):
redis = self.redis.pubsub()
redis.subscribe(self.channel)
for data_raw in redis.listen():
if data_raw['type'] != "message":
continue
data = simplejson.loads(data_raw["data"])
handler(data)
from socketio.namespace import BaseNamespace
from socketio.sdjango import namespace
from supremo.utils import redis_client, PubSub
from gevent import Greenlet
#namespace('/chat')
class ChatNamespace(BaseNamespace):
nicknames = []
r = redis_client()
q = PubSub(r, "channel")
def initialize(self):
# Setup redis listener
def handler(data):
self.emit('receive_message',data)
greenlet = Greenlet.spawn(self.q.subscribe, handler)
def on_submit_message(self,msg):
self.q.publish(msg)
I used parts of code from https://github.com/fcurella/django-push-demo and gevent-socketio 0.3.5rc1 instead of rc2 and it is working now with multiple workers and load balancing.