I have a Django project that needs to send out around 600 emails. I have Celery set up and it works, for the most part. I have the Django project set up to use my Google Apps (Business version -- ie: paid for) email account as the sending account. For testing purposes, I have every email sent to me -- not to the client.
The issue I am having is that Celery seems to randomly skip people in the list. When I start the process of sending all 600 emails, Celery works away, sending emails (I can see them show up in my inbox) but I only receive a total of about 420 emails. When Celery finishes, there are still 180 or so people that need the email. If I click "send emails" again with ONLY the remaining 180 people, it will finish the job and, at the end of two attempts, will have sent emails to all 600 people.
Why would Celery be skipping people?
Yes, you will get those gmail errors and it's not particularly predictable.
You could just use django-mailer instead -- I do, and deal with those gmail connection errors by letting django-mailer automatically retry the failed sending attempts until they succeed.
Check out this SO question for more folks suggesting you just use django-mailer vs celery for mail.
Advice on Python/Django and message queues
Related
I have different REST-API views where I either send a mail (if an account exists) or do not send a mail.
For example, the user can input the email in the forgot-password form and a mail is sent if the account exists.
I am using from django.core.mail import send_mail to send the mail.
The problem is, that this takes some time, and so requests for valid emails are generally longer than requests for non-exiting emails.
This allows an attacker to compare the request times to find out if an account exists or not.
Is there any way that I can call send_mail() without sending the mail?
Or what would be the fix to make request times equally long for both cases?
Note: I could check how long send_mail() needs on average and wait this time if I do not send the mail. As the app runs on different servers with different configs, this can not be generally done in my case. I would rather not store the average execution time per server in a database to solve this.
It's a common practice to use celery for tasks that require some time to be finished. Celery will run a task in a separate thread and a user doesn't need to wait while it is finished. In your specific case what will happen if you use celery:
You send a task send_mail to celery and immediately return a successful response to a user.
Celery receives a task and runs it in a separate thread.
In this way, the response time for both cases will be the same.
So this is something similar to an issue I had, and my solution was actually to always send the email, but the email reads something like You tried to reset your password, but this email isn't registered to an account with us. if they don't have an account.
From a user's perspective, it can be annoying to have to wait for an email that may or may not arrive, and spend time checking spam/junk etc. Telling them they don't have an account with that email address is quicker and cleaner for them.
We saw a big drop in users enquiring with us about why they hadn't received a PW reset email.
(Sorry for not actually answering the question, I dislike it when people do this on SO but since I experienced the same issue, I thought I'd weigh in.)
Perhaps it is just because I've never set up an e-mail system on Django before, or maybe I'm missing it... but does anyone have any insight on how to properly configure django post-office for sending queued e-mails?
I've got a mailing list of 1500 + people, and am hosting my app on heroku - using the standard email system doesn't work because I need to send customized emails to each user, and to connect to the server one by one leads to a timeout.
I've installed django-post_office via pip install, installed the app in settings.py, I've even been able to get an email to send by going:
mail.send(['recipient'],'sender',subject='test',message='hi there',priority='now')
However, if I try to schedule for 30 seconds from now let's say:
nowtime = datetime.datetime.now()
sendtime = nowtime + datetime.timedelta(seconds=30)
and then
mail.send(['recipient'],'sender',subject='test',message='hi there',scheduled_time=sendtime)
Nothing happens... time passes, and the e-mail is still listed as queued, and I don't receive any emails.
I have a feeling it's because I need to ALSO have Celery / RQ / Cron set up??? But the documentation seems to suggest that it should work out of the box. What am I missing?
Thanks folks
Actually, you can find this in the documentation (at the time I'm writing this comment):
Usage
If you use post_office’s EmailBackend, it will automatically queue emails sent using django’s send_mail in the database.
To actually send them out, run python manage.py send_queued_mail. You can schedule this to run regularly via cron:
* * * * * (/usr/bin/python manage.py send_queued_mail >> send_mail.log 2>&1)
my app has list of events with start time (date and time). I want to make a scheduled task to send reminder via email to all user participate in event 1 hour before event start. (Note: Admin can change time of event).
I currently use celery to send email to list of participants when admin change the time of event.
Please suggest me some solution for this. Thanks.
Here's a recent(ish) discussion where a potential solution is proposed for celery: https://github.com/celery/celery/issues/4522.
I built Posthook to make solving these kinds of problems easier for developers. In your case, when a new event is created or the event time changes you can schedule a request back to your app for 1 hour before the start time. Then when you get the request from Posthook you can send out the reminder after validating that it still needs to be sent out.
I have now succesfully setup Django-celery to check after my existing tasks to remind the user by email when the task is due:
#periodic_task(run_every=datetime.timedelta(minutes=1))
def check_for_tasks():
tasks = mdls.Task.objects.all()
now = datetime.datetime.utcnow().replace(tzinfo=utc,second=00, microsecond=00)
for task in tasks:
if task.reminder_date_time == now:
sendmail(...)
So far so good, however what if I wanted to also display a popup to the user as a reminder?
Twitter bootstrap allows creating popups and displaying them from javascript:
$(this).modal('show');
The problem is though, how can a celery worker daemon run this javascript on the user's browser? Maybe I am going a complete wrong way and this is not possible at all. Therefore the question remains can a cronjob on celery ever be used to achieve a ui notification on the browser?
Well, you can't use the Django messages framework, because the task has no way to access the user's request, and you can't pass request objects to the workers neither, because they're unpickable.
But you could definitely use something like django-notifications. You could create notifications in your task and attach them to the user in question. Then, you could retrieve those messages from your view and handle them in your templates to your liking. The user would see the notification on their next request (or you could use AJAX polling for real-time-ish notifications or HTML5 websockets for real real-time [see django-websocket]).
Yes it is possible but it is not easy. Ways to do/emulate server to client communication:
polling
The most trivial approach would be polling the server from javascript. Your celery task could create rows in your database that can be fetched by a url like /updates which checks for new updates, marks the rows as read and returns them.
long polling
Often referred to as comet. The client does a request to the server which pends until the server decides to return something. See django-comet for example.
websocket
To enable true server to client communication you need an open connection from the client to the server. django-socketio and django-websocket are examples of reusable apps that make this possible.
My advice judging by your question's context: either do some basic polling or stick with the emails.
Here is my setup right now:
connection = mail.get_connection()
maillist = []
# my real setup is a little more complex for-loop, but basicly I add all recipients to a list.
for person in object_list:
mail_subject = "Mail subject here"
mail_body = "Mail body text...bla bla"
email_sender = "me#example.com"
maillist.append((mail_subject, mail_body, email_sender, [person.email]))
#send_mass_mail wants a tuple, so we convert the list
mailtuple = tuple(maillist)
mail.send_mass_mail(mailtuple, fail_silently=False, connection=connection)
However, the forloop iterates over 1000+ objects/persons and when I try this method I'm able to send 101 emails, and then it stops. No errors (as I can see) anywhere.
A fellow developer mentioned that maybe the POST size was too big? Any ideas from the SO-community?
Your SMTP server probably has some send limits. For example, I believe Gmail limits outgoing mail to 100 recipients.
As Micah suggested, there is a good chance you are hitting server limits.
Generally, when dealing with mass mail, it is always a good idea to throttle the sending. Doing 50 mails every 5 seconds for 300 seconds beats 3000 mails at once for many practical reasons including smtp server limitations.
Since you mentioned a POST limit - do you send out the emails in a view? I'm wondering how you handle canceled requests in your setup.
I'm using a management command to send out 1000+ newsletters. But instead of send_mass_mail i use the normal send method in a loop. It takes about 5 minutes (haven't a correct count atm) to send out the mails and i haven't run into any server limits yet.
My plan is to switch to celery to handle sending through a web interface. Perhaps you want to have a look at it in case you haven't already.
http://celeryproject.org/