I read http://bottlepy.org/docs/dev/tutorial_app.html#server-setup
and running Apache + Bottle + Python
and Bottle + Apache + WSGI + Sessions
and I would like to know if one can run asynchronous rest api calls to bottle on mod_wsgi server to a py function that does not return anything(its a backend logic) and is non blocking - so I looked up gevent but i am haven't found a solution where you can run mod_wsgi with gevents.
Is there any solution to async calls to run on apache server using mod_wsgi or any other alternative?
UPDATE
as per andreans' answer below;
I ran a simple myip address return with bottle + celery. so one has to run a celery as #celery.task and then run(host='localhost', port=8080, debug=True)? does it require to start celery worker on terminal as well? never used celery before [runnin locally] also running bottle with decorator #route(/something) works but app.route doesnt where app = Bottle() possibly due to some .wsgi file error?
Sorry, can't fit into the comment box. Every request must get a response eventually (or fail/time out). If you really don't need to return any data to the client, send back just an empty response with a status code. If the processing of the request takes time, it should run asynchronously, and that's where celery comes in. So a blocking implementation of your request handler:
def request_processor_long_running_blocking_func(request_data):
# process request data, which takes a lot of time
# result is probably written into db
pass
def request_handler_func(request):
request_processor_long_running_blocking_func(request.data)
return HttpResponse(status=200)
If I understood correctly this is what you're trying to avoid, by making the request_processor_long_running_blocking_func run asynchronously, so the request_handler_func won't block. This would be solved with celery like this:
from celery.task import task
#task
def request_processor_long_running_blocking_func(request_data):
# the task decorator wraps your blocking function, with celery's Task class
# which has a delay method available for you to call, which will run your function
# asynchronously on one of your celery background worker processes
pass
def request_handler_func(request):
request_processor_long_running_blocking_func.delay(request.data)
# calling the function with delay won't block, it returns immediately
# and your response is sent back instantly
return HttpResponse(status=200)
One more thing, send these task requests with ajax, so your web interface won't be reloaded or anything, so the user can continue using your app after sending the request
Related
According to Django is synchronous or asynchronous. Django is synchronous. However I tested a blocking view with python manage.py runserver 8100:
import time
#action(detail=False, methods=['get'])
def test(self, request):
time.sleep(5)
return {}
and triggered two requests with postman at 0s, 1s, and they returned at 5s, 6s. That seems not blocking/synchronous. Where am I wrong?
Even synchronous implementations handle usually multiple requests 'in parallel'.
They do so by using multiple processes, multiple threads or a mix of it.
Depending on the server they have a predefined (fixed) amount of processes or threads or they dynamically allocate threads or processes whenever another requests requires one.
An asynchronous server on the other hand can handle multiple requests 'in parallel' within only one thread / process.
The simple development server, that you can start with management.py runserver is using threading by default.
To best visualize this I suggest to change your code to:
import time
import os
import threading
#action(detail=False, methods=['get'])
def test(self, request):
print("START PID", os.getpid(), "TID", threading.get_native_id())
time.sleep(5)
print("STOP PID", os.getpid(), "TID", threading.get_native_id())
return {pid=os.getpid(), tid=threading.get_native_id()}
As #xyres mentioned: There is a command line option to disable threading.
Just run manage.py runserver --nothreading and try again. Now you should be able to visualize the full synchronous behavior.
I am trying to create an asynchronous task using celery, but I am not achieving success.
I have a task that sends emails:
#shared_task()
def send_email_example(email_id):
...
I call it using the delay() method:
class SomeModelExample:
...
def example(self):
...
send_email_example.delay(self.id)
Locally, I run the celery and can use it. However, in my server, when I use the method that calls the function it takes more than 30 seconds and I receive a status code 502.
I hope that my celery setup is ok, because my periodc tasks works.
Solution
Testing it myself I saw that the celery is really ok. The problem was with my tests on the server.
At this stage I would try:
Use decorator without brackets: #shared_task
Increase harakiri time to >60 seconds on your server to see if it is a task problem or a server problem.
This is a generic question that I seek answer to because of a celery task I saw in my company's codebase from a previous employee.
It's a shared task that calls an endpoint like
#shared_task(time_limit=60*60)
def celery_task_here(some_args):
data = get_data(user, url, server_name)
# some other logic to build csv and stuff
def get_data(user, url, server_name):
client = APIClient()
client.force_authenticate(user=user)
response = client.get(some_url, format='json', SERVER_NAME=server_name)
and all the logic resides in that endpoint.
Now what I understand is that this will make the server do all the work and do not utilize celery's advantage, but I do see celery log producing queries when I run this locally. I'd like to know who's actually doing the job in this case, celery or the django server?
If the task is called via celery_task_here.delay, the task will be pushed to a queue, then the worker process that is responsible for handling the queue will actually execute the task, which is not the "Django server". The worker process could potentially be on the same machine as your Django instance, it depends on your environment.
If you were to call the task via celery_task_here.s (or as a normal function) the task would be executed by the Django server.
It depends of how the task is called
If it is meant to be called as celery task with apply_async or delay than it is executed as celery task by celery worker process
You still can call it as normal function without sending it to celery if you just call it as function
I've got a flask app running under gunicorn which handles client requests via REST api with an extremely CPU-intensive backend; some requests take minutes to respond to.
But that creates its own problem. If I, say, run a little script to make a request and kill it (ctrl-C or whatever), the flask app keeps on running despite the fact that no one will hear it when it comes back from the depths of computation and gets its broken pipe.
Is there a way to terminate the API call (even just kill/restart the worker) as soon as the client connection is broken? That feels like a thing Gunicorn could handle, but I'm powerless to find any setting that would do the trick.
Thanks--this has been vexing me!
Killing a flask worker can be done with this code:
from flask import request
def shutdown_server():
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Werkzeug server doesn't run flask')
func()
#app.route('/shutdown', methods=['GET'])
def shutdown():
shutdown_server()
return 'Shutting down...'
For killing a Gunicorn server on Linux, you can use this command, which I tested:
pkill gunicorn
This command works flawlessly on all kinds of Linuxes, which I assume you have installed for server
Or if I give you a Python implementation:
import os
def shutdownGunicorn():
os.system("pkill gunicorn")
I don't think killing after request is done would be smart, because then you couldn't know when you will get next request.
Flask doesn't take much CPU and RAM usage while it's not working!
Hope that gives you an answer!
I need to run celery task only when django request finished.
Is it possible?
I've found that the best way to make sure your task happens after the request is finished is to write a custom middleware. In the process_response method, you can handle any quick actions that don't impact page load time or performance too much. Anything else, you can hand off to Celery. Any saving or database transactions are completed by the time process_response is called (AFAICT).
Try something like this:
Django sends request_finished at the end of every request.
You can access request object through sender argument,
from django.dispatch import receiver
from django.core.signals import request_finished
from app.tasks import my_task
#receiver(request_finished)
def add_celery_task(sender):
if sender.__name__ != 'StaticFilesHandler':
my_task.delay()
If you are running server in development environment it's good to check sender's name to avoid adding too many celery task for every static file you are serving.
You can run the task in the background, using delay method of celery. I mean just before returning the response you can call the delay method to put the task in the background.
Some thing like this:
task_name.delay(arg1, arg2, ...)
By doing this your task will be put into background and run asynchronously, this is not going to block the request response cycle .