BACKGROUND: I have a Django application and i want to write tests for it. I have written a docker-compose.yml file so that i can be deployed on server quickly and easily. The application works with the help of celery and redis as its broker. The entire process of selenium testing is also being done via docker where i deploy selenium hub and nodes of chrome and firefox.
PROBLEM: When i run the python manage.py test app_name.tests.test_forms my fixtures are loaded and selenium connection is successful, class of the test inherits from StaticLiveServerTestCase. But when i call a celery task from the app_name.tasks.task_name.delay(), obviously after importing it, I noticed that the tasks runs on the my celery worker container that is for the production app and makes the changes that the task is supposed to do in the test database, it does it in the production database.
I have tried multiple possible solutions, like adding this above the test class #override_settings(CELERY_TASK_EAGER_PROPAGATES=True,CELERY_TASK_ALWAYS_EAGER=True,BROKER_BACKEND='memory')
also tried to starting celery_worker in the test's thread using the following in the setUpClass:
cls.celery_worker = start_worker(app,perform_ping_check=False)
cls.celery_worker.__enter__()
and in tearDownCass:
cls.celery_worker.__exit__(None, None, None)
and get this following error:
ERROR/MainProcess] pidbox command error: AttributeError("'NoneType' object has no attribute 'groups'")
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kombu/pidbox.py", line 104, in dispatch
reply = handle(method, arguments)
File "/usr/local/lib/python3.7/site-packages/kombu/pidbox.py", line 126, in handle_cast
return self.handle(method, arguments)
File "/usr/local/lib/python3.7/site-packages/kombu/pidbox.py", line 120, in handle
return self.handlers[method](self.state, **arguments)
File "/usr/local/lib/python3.7/site-packages/celery/worker/control.py", line 279, in enable_events
if dispatcher.groups and 'task' not in dispatcher.groups:
AttributeError: 'NoneType' object has no attribute 'groups'
But even though this error is encountered, the changes in the production database still happen.
I am missing something, can anyone help me with how to make celery task make changes in test database.
Am i following a bad practice? I just hope that i can make my selenium fill the forms required for testing and the celery tasks do their job and update the database after some processing and i can assert them in the test class.
Thank you, any help will be highly appreciated.
NOTE: The celery tasks are nested, i.e they call each other via .delay() or .apply_async() calls.
Related
I am making a price tracker.My project structure is this:
Myapp-folder
manage.py-from flask script module
subApp-folder
__init__.py
form.py
models.py
views.py
pricemonitor-folder
main.py
__init__.py
send_email.py
price_compare_sendemail.py-with class Compare_sendemail and start_monitor function
In the main.py, I have an interation to compare the prices every 60s and send-email if needed.
from app.PriceMonitor.price_compare_sendmail import Compare_sendemail
break_time = 60 # set waiting time for one crawl round
monitor = Compare_sendemail()
monitor.start_monitor(break_time)
The manage.py is as below:
from flask_script import Manager, Server
from app import app, db
manager = Manager(app)
manager.add_command("runserver",Server(host='127.0.0.1', port=5000, use_debugger=True))
if __name__ == '__main__':
manager.run()
But the iteration doesn't work when I run python manage.py runserver while I directly run the main.py successfully. How can I make up code to run the flask server with the compare_sendemail iteration running at the background? Thanks.
I think you are looking for Celery.
you can use Celery background task. If your application has a long running task, such as processing some uploaded data or sending email, you don’t want to wait for it to finish during a request. Instead, use a task queue to send the necessary data to another process that will run the task in the background while the request returns immediately.
here you can find documentation for celery
https://flask.palletsprojects.com/en/1.1.x/patterns/celery/
and if you want to wait for Task to complete you can use Coroutines and Tasks
https://docs.python.org/3/library/asyncio-task.html
there are other options for flask background task
like
RQ
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xxii-background-jobs
some other Alternatives
https://smirnov-am.github.io/background-jobs-with-flask/
Threads
uWSGI thread
uWSGI spooler
uSWGI spooler is great for simple tasks. like sending OTP SMS or Email.
I answer part of my own question.
In the main.py, I used while loop and time module to iterate the price_compare_sendemail.py
every 60s. While this is not an ideal background task handler, this project is currently just for my own usage so it is OK for me. My original thought was using the flask script manager to handle all the python commands-I don't know if it is the right thought though because I just started to learn Flask.
After some google search, I found the way to use manager.
from subapp.pricemonitor.main import Start_monitor
Monitor=Start_monitor()
#manager.command
def monitor_start():
break_time=10
Monitor.start_monitoring(break_time)
Then use the command 'python manage.py monitor_start' to start the background task. I don't know if it is useful but at least it fit my original thought.
I am just starting to learn about Django and have just discovered celery to run async background tasks.
I have a dummy project which I pilfered off the internet with a sample task as follows:
from djcelery import celery
from time import sleep
#celery.task
def sleeptask(i):
sleep(i)
return i
Now in my view, I have the following:
def test_celery(request):
result = tasks.sleeptask.delay(10)
return HttpResponse(result.task_id)
This runs fine and when I point the browser to it, I get some random string like 93463e9e-d8f5-46b2-8544-8d4b70108b0d which I am guessing is the task id.
However, when I do this:
def test_celery(request):
result = tasks.sleeptask.delay(10)
return HttpResponse(result.get())
The web browser goes in a loop with the message "Connecting..." and never returns. I was under the impression, this will block till the task is run and give the result but that does not seem to be the case. What am I doing wrong?
Another question is the way I am doing it, is it going to run it asynchronously i.e. not block while the task is running?
EDIT
In my settings.py file I have:
import djcelery
# Setup celery
djcelery.setup_loader()
BROKER_URL = 'redis://localhost:6379/0'
On the Django side, I do not get any errors:
System check identified no issues (0 silenced).
September 27, 2016 - 18:13:12
Django version 1.9.5, using settings 'myproject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Thanks to the hints in the comments, I finally was able to solve the problem. I had to add the following:
CELERY_IMPORTS('myproject.tasks') to my settings.py file.
I also needed to run the worker as:
python manage.py celery worker
I got started with flask and I tried out the Flaskr example. On the execution of a certain python script, I would like to update one row of my database.
I am a newbie here and would like to understand: am I going to update the DB from inside that python script or I am going to wait for a signal from the flask WSGI script:
I have referred to this thread but am not sure how I am going to interact with the external script. Any help or hints are appreciated.
WSGI handles HTTP requests/responses. A script won't be issuing those. Instead, import your Flask app in the script and make an application context:
from my_project import my_app
ctx = my_app.app_context()
ctx.push()
# ... my code
db.session.commit()
ctx.pop()
Relevant docs: http://flask.pocoo.org/docs/appcontext/, http://flask.pocoo.org/docs/shell/
Or consider using Flask-Script to add command line functions to your application, if the function doesn't need to be a separate script.
I am trying to use django-celery in my project
In settings.py I have
CELERY_RESULT_BACKEND = "amqp"
The server started fine with
python manage.py celeryd --setting=settings
But if I want to access a result from a delayed task, I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\celery\result.py", line 108, in ready
return self.status in self.backend.READY_STATES
File "C:\Python27\lib\site-packages\celery\result.py", line 196, in status
return self.state
File "C:\Python27\lib\site-packages\celery\result.py", line 191, in state
return self.backend.get_status(self.task_id)
File "C:\Python27\lib\site-packages\celery\backends\base.py", line 404, in _is
_disabled
raise NotImplementedError("No result backend configured. "
NotImplementedError: No result backend configured. Please see the documentation
for more information.
It is very strange because when I just run celeryd (with the same celery settings), it works just fine. Has anyone encountered this problem before?
Thanks in advance!
I had the same problem while getting the result back from the celery task although the celery task was executed ( console logs). What i found was, i had the same setting CELERY_RESULT_BACKEND = "redis" in django settings.py but i had also instantiated celery in the tasks.py
celery = Celery('tasks', broker='redis://localhost') - which i suppose overrides the settings.py property and hence it was not configuring the backend server for my celery instance which is used to store the results.
i removed this and let django get celery get properties from settings.py and the sample code worked for me.
If you're just running the samples from http://www.celeryproject.org/tutorials/first-steps-with-celery/, you need to run the console via manage.py:
% python manage.py shell
For those who are in a desperate search for a solution like I was.
Put this line at the end of the settings.py script:
djcelery.setup_loader()
Looks like django-celery is not going to consider it's own settings without a strict order.
In my case, the problem was that I was passing the CELERY_RESULT_BACKEND argument to the celery constructor:
Celery('proj',
broker = 'amqp://guest:guest#localhost:5672//',
CELERY_RESULT_BACKEND='amqp://',
include=['proj.tasks'])
The solution was to use the backend argument instead:
Celery('proj',
broker = 'amqp://guest:guest#localhost:5672//',
backend='amqp://',
include=['proj.tasks'])
Some how the console has to have django environment set up in order to pick up the settings. For example, in PyCharm you can run django console, in which everything works as expected.
See AMQP BACKEND SETTINGS for better understanding
Note The AMQP backend requires RabbitMQ 1.1.0 or higher to
automatically expire results. If you are running an older version of
RabbitMQ you should disable result expiration like this:
CELERY_TASK_RESULT_EXPIRES = None
Try adding the below line to your settings.py:
CELERY_TASK_RESULT_EXPIRES = 18000 # 5 hours
I am using Lighttpd + FastCGI + Django on a dev machine. I start FastCGI server via manage.py's command line option.
Problem is that I do make changes to sources quite often and I need to get FastCGI to pick up those changes automatically, just as "./manage.py runserver" does.
Is there a command-line option for that, perhaps, or any other solutions?
Have you looked at the code in the runserver part of manage.py that does the monitoring? I see no reason you could not just copy-paste that code into a custom manage.py script and set it to run the lighty restart command when changes were detected.
Alternatively, you could run a separate python program that did the restart using a package like pyinotify:
http://trac.dbzteam.org/pyinotify
I'm wondering if anyone has ever gotten this to work? I have tried implementing a reload mechanism using django's autoreload.py, unfortunately, I get errors when the fork occurs like:
django/core/servers/fastcgi.py", line 180, in runfastcgi
WSGIServer(WSGIHandler(), **wsgi_opts).run()
File "build/bdist.freebsd-6.4-RELEASE-p9-amd64/egg/flup/server/fcgi_fork.py", line 140, in run
File "build/bdist.freebsd-6.4-RELEASE-p9-amd64/egg/flup/server/preforkserver.py", line 119, in run
File "build/bdist.freebsd-6.4-RELEASE-p9-amd64/egg/flup/server/preforkserver.py", line 450, in _installSignalHandlers
ValueError: signal only works in main thread
My ideal setup would be to be able to reload/kill my fcgi process and start a new one after each time a code change is detected, similar to how django does this with their internal server. I also tried removing the threading from autoreload.py that would get past this error, but it does not seem to run the server properly (still investigating that).
Perhaps someone has tried CherryPies autoreload.py in the settings.py file for django?