We are using AWS SES for sending mails. Amazon SES sends bounce and complaint notifications through emails or AWS SNS. We would like to automatically process the bounce and complaint notifications (coming from email or AWS SNS) to extract the email ids, so that these emails can be removed from the original list.
One way to automate is to send these notifications to a topic in AWS SNS, then subscribe to the topic using AWS SQS and finally read the messages in the AWS SQS. SNS supports subscription over the following protocols - HTTP/HTTPS/EMail/EMail(JSON)/SMS/SQS. This is feasible, but I find it too cumbersome for a simple task of automatically processing the bounce and complaint notifications.
Is there any elegant way of tacking this problem?
I have found a blog entry from Amazon with the code in C#. Is there a better solution?
I find that directly subscribing to SNS using an HTTP endpoint is the most straightforward approach. You literally have to write only a few lines of code. Here's my django example:
def process(request):
json = request.raw_post_data
js = simplejson.loads(json)
info = simplejson.loads(js["Message"])
type = info["notificationType"] # "Complaint" or "Bounce"
email = info["mail"]["destination"][0]
# do whatever you want with the email
I think the way you describe IS probably the most elegant way. You already have very appropriate services in SNS and SQS that have associated SDK's in most major languages to allow you to do what you need easily. The hardest part is writing the code to update/remove the records in your mailing lists.
Through trial an error I have come up with this one - it is for Django but does a decent job for me.
First the models, then the request handler...
class ComplaintType:
ABUSE = 'abuse'
AUTH_FAILURE = 'auth-failure'
FRAUD = 'fraud'
NOT_SPAM = 'not-spam'
OTHER = 'other'
VIRUS = 'virus'
COMPLAINT_FEEDBACK_TYPE_CHOICES = [
[ComplaintType.ABUSE, _('Unsolicited email or some other kind of email abuse')],
[ComplaintType.AUTH_FAILURE, _('Unsolicited email or some other kind of email abuse')],
[ComplaintType.FRAUD, _('Some kind of fraud or phishing activity')],
[ComplaintType.NOT_SPAM, _('Entity providing the report does not consider the message to be spam')],
[ComplaintType.OTHER, _('Feedback does not fit into any other registered type')],
[ComplaintType.VIRUS, _('A virus was found in the originating message')]
]
class SES_Complaint(models.Model):
subject = models.CharField(max_length=255)
message = models.TextField()
email_address = models.EmailField(db_index=True)
user_agent = models.CharField(max_length=255)
complaint_feedback_type = models.CharField(max_length=255, choices=COMPLAINT_FEEDBACK_TYPE_CHOICES)
arrival_date = models.DateTimeField()
timestamp = models.DateTimeField()
feedback_id = models.CharField(max_length=255)
created = models.DateTimeField(auto_now_add=True)
modified = models.DateTimeField(auto_now=True)
class Meta:
verbose_name = 'SES Complaint'
verbose_name_plural = 'SES Complaints'
def get_reason(self):
return self.get_complaint_feedback_type_display()
class BounceType:
UNDETERMINED = 'Undetermined'
PERMANENT = 'Permanent'
TRANSIENT = 'Transient'
class BounceSubType:
UNDETERMINED = 'Undetermined'
GENERAL = 'General'
NO_EMAIL = 'NoEmail'
SUPPRESSED = 'Suppressed'
MAILBOX_FULL = 'MailboxFull'
MESSAGE_TOO_LARGE = 'MessageToolarge'
CONTENT_REJECTED = 'ContentRejected'
ATTACHMENT_REJECTED = 'AttachmentRejected'
BOUNCE_TYPE_CHOICES = [
[BounceType.UNDETERMINED, _('Unable to determine a specific bounce reason')],
[BounceType.PERMANENT, _('Unable to successfully send')],
[BounceType.TRANSIENT, _('All retry attempts have been exhausted')],
]
BOUNCE_SUB_TYPE_CHOICES = [
[BounceSubType.UNDETERMINED, _('Unable to determine a specific bounce reason')],
[BounceSubType.GENERAL, _('General bounce. You may be able to successfully retry sending to that recipient in the future.')],
[BounceSubType.NO_EMAIL, _('Permanent hard bounce. The target email address does not exist.')],
[BounceSubType.SUPPRESSED, _('Address has a recent history of bouncing as invalid.')],
[BounceSubType.MAILBOX_FULL, _('Mailbox full')],
[BounceSubType.MESSAGE_TOO_LARGE, _('Message too large')],
[BounceSubType.CONTENT_REJECTED, _('Content rejected')],
[BounceSubType.ATTACHMENT_REJECTED, _('Attachment rejected')]
]
class SES_Bounce(models.Model):
subject = models.CharField(max_length=255)
message = models.TextField()
bounce_type = models.CharField(max_length=255, choices=BOUNCE_TYPE_CHOICES)
bounce_sub_type = models.CharField(max_length=255, choices=BOUNCE_SUB_TYPE_CHOICES)
timestamp = models.DateTimeField()
feedback_id = models.CharField(max_length=255)
status = models.CharField(max_length=255)
action = models.CharField(max_length=255)
diagnostic_code = models.CharField(max_length=255)
email_address = models.EmailField(db_index=True)
created = models.DateTimeField(auto_now_add=True)
modified = models.DateTimeField(auto_now=True, db_index=True)
class Meta:
verbose_name = 'SES Bounce'
verbose_name_plural = 'SES Bounces'
def get_reason(self):
return '%s - %s' % (self.get_bounce_type_display(), self.get_bounce_sub_type_display())
And here is the request handler:
#csrf_exempt
def aws_sns(request):
logger.debug('Incoming SNS')
if request.method == 'POST':
logger.debug('Incoming SNS is POST')
sns_message_type = request.META.get('HTTP_X_AMZ_SNS_MESSAGE_TYPE', None)
if sns_message_type is not None:
logger.debug('Incoming SNS - %s', sns_message_type)
json_body = request.body
json_body = json_body.replace('\n', '')
js = loads(json_body)
if sns_message_type == "SubscriptionConfirmation":
subscribe_url = js["SubscribeURL"]
logger.debug('Incoming subscription - %s', subscribe_url)
urllib.urlopen(subscribe_url)
elif sns_message_type == "Notification":
message = js.get("Message", None)
message = message.replace('\n', '')
message = loads(message)
notification_type = message.get("notificationType", None)
if notification_type == 'AmazonSnsSubscriptionSucceeded':
logger.debug('Subscription succeeded')
elif notification_type == 'Bounce':
logger.debug('Incoming bounce')
bounce = message['bounce']
bounce_type = bounce['bounceType']
bounce_sub_type = bounce['bounceSubType']
timestamp = bounce['timestamp']
feedback_id = bounce['feedbackId']
bounce_recipients = bounce['bouncedRecipients']
for recipient in bounce_recipients:
status = recipient.get('status')
action = recipient.get('action')
#diagnostic_code = recipient['diagnosticCode']
email_address = recipient['emailAddress']
SES_Bounce.objects.filter(email_address=email_address).delete()
SES_Bounce.objects.create(
message=message,
bounce_type=bounce_type,
bounce_sub_type=bounce_sub_type,
timestamp=timestamp,
feedback_id=feedback_id,
status=status,
action=action,
#diagnostic_code=diagnostic_code,
email_address=email_address
)
elif notification_type == 'Complaint':
logger.debug('Incoming complaint')
complaint = message['complaint']
user_agent = complaint.get('userAgent')
complaint_feedback_type = complaint.get('complaintFeedbackType')
arrival_date = complaint.get('arrivalDate')
timestamp = complaint['timestamp']
feedback_id = complaint['feedbackId']
recipients = complaint['complainedRecipients']
for recipient in recipients:
email_address = recipient['emailAddress']
SES_Complaint.objects.filter(email_address=email_address).delete()
SES_Complaint.objects.create(
#subject=subject,
message=message,
email_address=email_address,
user_agent=user_agent,
complaint_feedback_type=complaint_feedback_type,
arrival_date=arrival_date,
timestamp=timestamp,
feedback_id=feedback_id
)
else:
logger.exception('Incoming Notification SNS is not supported: %s', notification_type)
return HttpResponse()
else:
logger.exception('Incoming SNS did not have the right header')
for key, value in request.META.items():
logger.debug('Key: %s - %s', key, value)
else:
logger.exception('Incoming SNS was not a POST')
return HttpResponseBadRequest()
Recently, I was able to get this working using an HTTP Endpoint via SNS. I use python/django to consume the notification. You have to process the subscription message first before you consume the notifications; you can read about subscriptions in the SNS documentation.
I think if you have a smaller application that doesn't send to many emails; http endpoint should work fine. This code requires that you have a notification model created.
#process an amazon sns http endpoint notification for amazon ses bounces and complaints
#csrf_exempt
def process_ses_notification(request):
if request.POST:
json_body = request.body
#remove this control character(throws an error) thats present inside the test subscription confirmation
js = loads(json_body.replace('\n', ''))
if js["Type"] == "SubscriptionConfirmation":
subscribe_url = js["SubscribeURL"]
urllib.urlopen(subscribe_url)
return HttpResponse(status=200)
elif js["Type"] == "Notification":
#process message from amazon sns
arg_info = loads(js["Message"]) # may need to use loads(js["Message"]) after testing with amazon
arg_notification_type = arg_info["notificationType"]
if arg_notification_type == 'Bounce':
#required bounce object fields
arg_emails=arg_info["bounce"]["bouncedRecipients"]
arg_notification_subtype=arg_info["bounce"]["bounceType"]
arg_feedback_id=arg_info["bounce"]["feedbackId"]
arg_date_recorded=arg_info["bounce"]["timestamp"]
elif arg_notification_type == 'Complaint':
#required complaint object fields
arg_emails=arg_info["complaint"]["complainedRecipients"]
arg_feedback_id=arg_info["complaint"]["feedbackId"]
arg_date_recorded=arg_info["complaint"]["timestamp"]
#check if feedback type is inside optional field name
if "complaintFeedbackType" in arg_info["complaint"]:
arg_notification_subtype=arg_info["complaint"]["complaintFeedbackType"]
else:
arg_notification_subtype=""
else:
HttpResponse(status=400)
#save notifications for multiple emails
for arg_email in arg_emails:
notification = SES_Notification(info=json_body, notification_type=arg_notification_type,
email=arg_email["emailAddress"], notification_subtype=arg_notification_subtype,
date_recorded=arg_date_recorded, feedback_id=arg_feedback_id)
notification.save()
return HttpResponse(status=200)
return HttpResponse(status=400)
all the answers above are great, but just a small and important addition:
you first need to verify that the request is from amazon sns: (as described at Verifying the Signatures of Amazon SNS Messages)
for the python code that validate the signature - a good example is here
Related
this is my first time writing a flask project and I find it easy to understand. I am working on a subscription base project and I stumbled a problem that I cannot get my head around.
I really appreciate it if anyone can help. Thanks so much. Looking forward to your response.
#payments.route('/webhook', methods=['POST'])
def webhook_received():
# You can use webhooks to receive information about asynchronous payment events.
# For more about our webhook events check out https://stripe.com/docs/webhooks.
webhook_secret = os.getenv('STRIPE_WEBHOOK_SECRET')
request_data = json.loads(request.data)
if webhook_secret:
# Retrieve the event by verifying the signature using the raw body and secret if webhook signing is configured.
signature = request.headers.get('stripe-signature')
try:
event = stripe.Webhook.construct_event(
payload=request.data, sig_header=signature, secret=webhook_secret)
data = event['data']
except Exception as e:
return e
# Get the type of webhook event sent - used to check the status of PaymentIntents.
event_type = event['type']
else:
data = request_data['data']
event_type = request_data['type']
data_object = data['object']
print('event ' + event_type)
if event_type == 'checkout.session.completed':
# Handle the checkout.session.completed event
session = data['data']['object']
#find and update the user's subscription
user = UserMixin.query.filter_by(username=current_user.username).first()
user.subscriptionStatus = True
user.stripe_customer_id = session['customer']
user.subscriptionItem = session['subscription_items'][0]['price']['lookup_key']
user.price = session['subscription_items'][0]['plan']['amount']
if user.subscriptionItem == 'Starter':
user.num_words = 20000
elif user.subscriptionItem == 'Professional':
user.num_words = 50000
db.session.commit()
print('🔔 Payment succeeded!')
return jsonify({'status': 'success'})
We are using sendgrid 6.0.5 python2.7 on google app engine standard.
The following code works
subject = data_sent_obj["subject"]
body_html = data_sent_obj["body_html"]
body_text = data_sent_obj["body_text"]
email_id_variable = "info#mycompany.com"
to_email = "info#mycompany.com" # THIS WORKS
# to_email = Email(email_id_variable) # THIS DOES NOT WORK
email_message = Mail(
from_email = 'info#mycompany.com',
to_emails = to_email,
subject = subject,
html_content = body_html)
personalization = Personalization()
personalization.add_to(Email(to_email))
bcc_list = bcc_email_list
for bcc_email in bcc_list:
personalization.add_bcc(Email(bcc_email))
email_message.add_personalization(personalization)
try:
sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY'))
response = sg.send(email_message)
When we use
to_email = Email(email_id_variable) we get the following error.
ValueError('Please use a To, Cc or Bcc object.',)
Essentially we would like to send email to an address that is in a variable.
It seems the issue itself is not using the variable but the Mail implementation removing the Email object as a possibility for the list in to_emails, so instead use the To object:
from sendgrid.helpers.mail import To
...
to_email = To(email_id_variable)
The same could work with cc and bcc objects.
I´m using signals to send mails to the users depending of some actions. In one of my signals I need to send the same mail to multiple users, my signal use post_save so for the parameter [recipient_list] I use this instance.email_list email_list is where I store the list of email addresses to send the mail, they are stored in this format user1#mail.com, user2#mail.com, user3#mail.com.
To send the emails I use EMAIL_HOST = 'smtp.gmail.com' the problem is that the only user who recive the email is the first one in the list. So I 'log in' in the gmail account to check the send emails section and actually in the "to" area of the email shows that was send to all the users in the list but the emails never arrive just for the first email address in the list.
I read that if google detect that the account sends a lot of messages could be blocked but only send like 5 emails at the same time and when I'm in the account never shows me some alert or something.
So the problem is how I send the emails or maybe some bad configuration of the gmail account?
Any help is really appreciated.
Sorry for my bad grammar.
EDIT: Here's my code.
forms.py
class MyForm(forms.Form):
userslist = forms.ModelChoiceField(queryset = User.objects.filter(here goes my condition to show the users), empty_label='List of users', label='Users', required=False)
emailaddress = forms.CharField(max_length=1000, label='Send to:', required=False)
comment = forms.CharField(widget=CKEditorUploadingWidget(), label="Comment:")
That form display a list of users to select the email address in the field emailaddress store the values. This is my Ajax to bring the email address:
views.py
class mails(TemplateView):
def get(self, request, *args, **kwargs):
id_user = request.GET['id']
us = User.objects.filter(id = id_user)
data = serializers.serialize('json', us, fields=('email'))
return HttpResponse(data, content_type='application/json')
And here's the <script> I use to populate the emailaddres field:
<script>
$('#id_userlist').on('change', concatenate);
function concatenate() {
var id = $(this).val();
$.ajax({
data: { 'id': id },
url: '/the_url_to_get_data/',
type: 'get',
success: function (data) {
var mail = ""
for (var i = 0; i < data.length; i++) {
mail += data[i].fields.email;
}
var orig = $('#id_emailaddress').val();
$('#id_emailaddress').val(orig + mail + ',');
}
})
}
</script>
The signal I use to send the mail is this:
#receiver(post_save, sender=ModelOfMyForm, dispatch_uid='mails_signal')
def mails_signal(sender, instance, **kwargs):
if kwargs.get('created', False):
if instance.emailaddress:
#Here goes the code for the subject, plane_message,
#from_email and template_message.
send_mail(subject, plane_message, from_email, [instance.emailaddress], fail_silently=False, html_message=template_message)
So if I select 4 users the info is save in this way in the database:
Then I 'log in' in the account to check the 'Sent Mail' section and check the detail of the mail and shows that was send to the 4 users but the only user who recibe the mail was the first in the list.
Your problem is that you are passing a comma-separated string of email addresses inside instance.emailaddress (first#gmail.com, second#hotmail.com, third#hotmail.com etc). Django expects a Python list of addresses, not a comma separated string. It will just ignore everything after the first comma.
Change your code as follows and it will work:
def mails_signal(sender, instance, **kwargs):
if kwargs.get('created', False):
if instance.emailaddress:
#Here goes the code for the subject, plane_message,
#from_email and template_message.
recipients = [r.strip() for r in instance.emailaddress.split(',')]
send_mail(subject, plane_message, from_email, recipients, fail_silently=False, html_message=template_message)
I'm trying to call Amazon Ses in python to send an email with attachments. If the receiver is a #gmail account, it works fine. However, if the receiver is #yahoo or some other email services, attachments do not get sent. What am I doing wrong?
def build_msg_html(cls, sender, receiver, subject, txt, html, attachment):
msg = MIMEMultipart('alternative')
msg['Subject'] = subject
msg['From'] = sender
msg['To'] = receiver
msg.attach(MIMEText(txt, 'plain', 'utf-8'))
if html is not None:
msg.attach(MIMEText(html, 'html', 'utf-8'))
if attachment is not None:
msg.attach(MIMEApplication(
attachment.file.read(),
Content_Disposition='attachment; filename="%s"' % attachment.filename,
Name=attachment.filename
))
return msg
I managed to solve this by following the answer from this question Java Mail attachment not shown in outlook :
I replaced MIMEMultipart('alternative') with MIMEMultipart()
I'm using django-allauth for my authentication system. I need that when the user sign in, the profile module get populated with the provider info (in my case facebook).
I'm trying to use the pre_social_login signal, but I just don't know how to retrieve the data from the provider auth
from django.dispatch import receiver
from allauth.socialaccount.signals import pre_social_login
#receiver(pre_social_login)
def populate_profile(sender, **kwargs):
u = UserProfile( >>FACEBOOK_DATA<< )
u.save()
Thanks!!!
The pre_social_login signal is sent after a user successfully
authenticates via a social provider, but before the login is actually
processed. This signal is emitted for social logins, signups and when
connecting additional social accounts to an account.
So it is sent before the signup is fully completed -- therefore this not the proper signal to use.
Instead, I recommend you use allauth.account.signals.user_signed_up, which is emitted for all users, local and social ones.
From within that handler you can inspect whatever SocialAccount is attached to the user. For example, if you want to inspect Google+ specific data, do this:
user.socialaccount_set.filter(provider='google')[0].extra_data
UPDATE: the latest development version makes this a little bit more convenient by passing along a sociallogin parameter that directly contains all related info (social account, token, ...)
Here is a Concrete example of #pennersr solution :
Assumming your profile model has these 3 fields: first_name, email, picture_url
views.py:
#receiver(user_signed_up)
def populate_profile(sociallogin, user, **kwargs):
if sociallogin.account.provider == 'facebook':
user_data = user.socialaccount_set.filter(provider='facebook')[0].extra_data
picture_url = "http://graph.facebook.com/" + sociallogin.account.uid + "/picture?type=large"
email = user_data['email']
first_name = user_data['first_name']
if sociallogin.account.provider == 'linkedin':
user_data = user.socialaccount_set.filter(provider='linkedin')[0].extra_data
picture_url = user_data['picture-urls']['picture-url']
email = user_data['email-address']
first_name = user_data['first-name']
if sociallogin.account.provider == 'twitter':
user_data = user.socialaccount_set.filter(provider='twitter')[0].extra_data
picture_url = user_data['profile_image_url']
picture_url = picture_url.rsplit("_", 1)[0] + "." + picture_url.rsplit(".", 1)[1]
email = user_data['email']
first_name = user_data['name'].split()[0]
user.profile.avatar_url = picture_url
user.profile.email_address = email
user.profile.first_name = first_name
user.profile.save()
If you are confused about those picture_url variable in each provider. Then take a look at the docs:
facebook:
picture_url = "http://graph.facebook.com/" + sociallogin.account.uid + "/picture?type=large" Docs
linkedin:
picture_url = user_data['picture-urls']['picture-url'] Docs
twitter:
picture_url = picture_url.rsplit("_", 1)[0] + "." + picture_url.rsplit(".", 1)[1] Docs And for the rsplit() take a look here
Hope that helps. :)
I am doing in this way and taking picture (field) url and google provider(field) as an example.
socialaccount_obj = SocialAccount.objects.filter(provider='google', user_id=self.user.id)
picture = "not available"
if len(socialaccount_obj):
picture = socialaccount_obj[0].extra_data['picture']
make sure to import : from allauth.socialaccount.models import SocialAccount
There is an easier way to do this.
Just add the following to your settings.py. For example, Linked in...
SOCIALACCOUNT_PROVIDERS = {
'linkedin': {
'SCOPE': [
'r_basicprofile',
'r_emailaddress'
],
'PROFILE_FIELDS': [
'id',
'first-name',
'last-name',
'email-address',
'picture-url',
'public-profile-url',
]
}
The fields are automatically pulled across.