I have Celery set up with Django. I'm using RabbitMQ as my broker. I'm trying to set up sqlalchemy as my result back-end with a sqlite database separate from the Django database. I have RabbitMQ, Django, & Celery all running without any issues. I put in my settings.py CELERY_RESULT_BACKEND = 'db+sqlite:///celery_results.sqlite3' & on Celery worker startup it shows the back-end configured correctly in the log output.
The problem is that my database isn't being created. Why is this happening?
Hopefully this will save someone some head scratching or ruining their configuration. Everything was set up correctly and running fine. The issue was that I didn't have any tasks running since I wanted as little going on as possible during setup. Once I ran a task the sqlite database & tables were created successfully. I was also able to write a script to confirm that the task results were being stored in the database.
Related
I'm working on a Django project with docker which use different legacy Microsoft SQL Server database. Since I didn't want to alter those legacy databases I use inspectdb and Django router to create models and then use the data there to read and write a default Postgres database with all the Django specific tables.
Everything was working fine, each instance of my model objects have the initial data from the SQL server db populating into the Postgres db. However whenever anything is updated in the SQL server db the data in the is not updating in Postgres db unless the docker container is stop and restarted. I'm wondering is there a way to autoupdate my postgres db when changes are made in the connected SQL Server db's with stopping the container?
Would I need to use something like celery or a cron job to schedule updates? I've used cron jobs before but only to pull from an api. Is it possibly to run celery or a cron job just on a database and not an api? Or is there a solution that's simplier?
I'm also running with nginx and gunicorn on an Ubuntu server. Using volumes to persist db.
I have a Django application and a Celery worker - each running on it's own server.
Currently, Django app uses SQLite to store the data.
I'd like to access the database using Django's ORM from the worker.
Unfortunately, it is not completely clear to me; thus I have some questions.
Is it possible without hacks/workarounds? I'd like to have a simple solution (I would not like to implement REST interface to object access). I imagine that achieving this could be done if I started using PostgreSQL instance which is accessible from both servers.
Which project files (there's just Django + tasks.py file) are required on the worker's machine?
Could you provide me with an example or tutorial? I tried looking it up but found just tutorials/answers bound to a problem of local Celery workers.
I have been searching for ways to do this simply but... Your best option is to attached a kind of callback to the task function that will call another function on the django server to carry out the database update
I am building a small financial web app with django. The app requires that the database has a complete history of prices, regardless of whether someone is currently using the app. These prices are freely available online.
The way I am currently handling this is by running simultaneously a separate python script (outside of django) which downloads the price data and records it in the django database using the sqlite3 module.
My plan for deployment is to run the app on an AWS EC2 instance, change the permissions of the folder where the db file resides, and separately run the download script.
Is this a good way to deploy this sort of app? What are the downsides?
Is there a better way to handle the asynchronous downloads and the deployment? (PythonAnywhere?)
You can write the daemon code and follow this approach to push data to DB as soon as you get it from Internet. Since your daemon would be running independently from the Django, you'd need to take care of data synchronisation related issues as well. One possible solution could be to use DateTimeField in your Django model with auto_now_add = True, which will give you idea of time when data was entered in DB. Hope this helps you or someone else looking for similar answer.
My celery database backend settings are:
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "mysqlite.db"
I am using RabbitMQ as my messager.
It doesn't seem like any results are getting stored in the db, and yet I can read the results after the task is complete. Are they in memory or a RabbitMQ cache?
I haven't tried reading the same result multiple times, so maybe its a read once then poof!
CELERY_RESULT_DBURI is for the sqlalchemy result backend, not the Django one.
The Django one always uses the default database configured in the DATABASES setting (or the DATABASE_* settings if on older Django versions)
my celery daemons work just fine, but I'm having difficulties with collecting task results. task_result.get() leads a timeout. and task.state is always PENDING..(but jobs are completed) i tried separate sqlite dbs, a single postgres db shared by workers. but i still cant get results. CELERY_RESULT_DBURI seems useless to me (for celery 2.5 )i think it's a newer configuration. Any suggestions are welcomed...
EDIT: it's all my fault:
i give extra parameters to my tasks with decorators, ignore_results=True Parameter create this problem. I deleted this key and it works like a charm :)
I am starting a project using django celery and I am making ajax calls to the task urls provided by 'djcelery.urls'.
I would like to know a few things about this data:
Where is that information being stored? Is it called from the djcelery tables in my django projects database or is it kept on the RabbitMQ server? My understanding of the djcelery tables in my database is that they are only for monitoring the usage using the camera.
If it is being stored on the RabbitMQ server, how long will the tasks status report be available? How much memory does this data consume?
Do I need to flush the task status reports periodically to prevent a memory leak? How would this be done? By restarting the rabbitmq server?
Thanks.
The results are stored in the CELERY_RESULT_BACKEND, which is disabled by default.
You can get the result of a task by creating a new celery.result.AsyncResult with the appropriate task_id: How do I get the result of a task if I have the ID that points there?.
Unless you set CELERY_AMQP_TASK_RESULT_EXPIRES, tasks will never expire. You can manually remove the result of a task using AsyncResult.forget().