We have a service responsible for sending emails using AWS SES. This is working pretty well since we deployed it. But one strange thing start to happen a day ago (April 22, 2020). We have change nothing from our side and start to receives a lot of emails from Amazon SES:
What we already know:
As it is happening with almost all emails we sent, not all users all
are clicking in the "unsubscribe" link
The users are receiving the emails, once we know they are clicking in
the links inside of the emails
The emails we sent two days ago are exactly the same emails we are sending today. Both content and configuration
If anyone have past for this kind of problem, any help would be great
I haven't encountered the report abuse but have encountered the related bounce email issue several times. Not much is useful from FAQ (https://aws.amazon.com/ses/faqs/) but it does mention the reputation dashboard which you should be following to see if you are on the road to recovery.
Your tasks include:
1) Investigating if you send an email that could be considered abuse/spam under local laws of the receiver
At a minimum, you need to make sure you are offering the capability to unsubscribe and actually unsubscribe users in timely fashion. But also review content with an eye on local laws.
2) Ensure that users who do not want to receive email from you are removed.
This should be part of above.
3) Build up your reputation by increasing the percentage of valid emails.
This has been an issue for us in systems that send a small amount of email...it takes time to build up from a dip.
Remember - AWS wants to ensure it's multi-tenant mail servers remain whitelisted and that other AWS customers aren't impacted by any one potential bad actor.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
I am using AWS SES to send out emails automatically through my application.
I have configured the Identity management as following:
DKIM is setup correctly. I have no issues sending emails from my domain except for GMX and WEB.de emails where I receive the following error:
Action: failed
Final-Recipient: rfc822; xyz#web.de
(mxweb111) Nemesis ESMTP Service not available
554-No SMTP service
554-Reject due to policy restrictions
Looking at further documentation, it seems that emails coming from my domain are classified as Spam by their servers.
I have done research and found that I might need to configure Reverse-DNS but as it looks like, AWS SES does not support this?
What else can I do to make my emails get through WEB.de and GMX servers?
Thank you.
I was in deep conversations with AWS SES support regarding this issue. This is the outcome:
I also would like to update you that SES internal team were able to confirm a deliverability issue with the recipient ISP and are actively working towards a resolution but we do not have an exact ETA at this time. Due to the nature of the shared IP pool, these types of blocks can happen periodically and we make every effort to resolve these issues as fast as possible. To prevent impact from these types of issues, it is always recommended to use dedicated ips for higher volume sending.
It means that the shared IP addresses used by AWS SES are blacklisted with GMX and WEB.de
AWS SES wants to resolve this.
In the meantime, they recommend to use dedicated IP addresses to solve this issue.
Please note that these IP addresses have to be "warmed up" in order to not cause trouble on the recipient end (e.g. spam folder issues).
Unfortunately, my sending volume is not that high (yet) so I have my fingers crossed I can get those emails send out easily. Otherwise I have to find another solution or need to wait for AWS so solve the blacklist issue.
I hope this helps anyone else.
Edit January 2021
I was able to send to GMX/WEB.de although my IP was only starting to warm up. Now after one month I am nearly at 100% with not many emails per day sendout volume.
Several e-mail services operated by United Internet (at least GMX, Web.de) seem to have blocked Amazon SES IPs. Validity of DKIM, SPF, DMARC does not seem to have any impact on the block. I'm seeing these rejections in my logfiles as far back as 2020-10-05.
The alternative of using a the dedicated IP address has its own challenges. Managing and warming up new IP addresses for delivery can be very painful (e.g., Outlook.com was known to accept and then silently discard e-mails after IP changes).
I would suggest to write to mailsecurity#info.gmx.net or use their contact form https://postmaster.gmx.net/en/contact. I've received a response from them, although they didn't seem to fully grasp the issue. Maybe more contacts will help them see the importance of addressing this. Until then I am informing my users per banner of the issue (and recommending alternative e-mail services).
Amazon support has not been helpful for me. I've received one first-level response which indicated the responder had not understood the issue at all, but promised to forward it to SES support. Since then I haven't heard anything for a week.
Edit: since 2021-03-24, there are no more 554-Reject due to policy restrictions failures in my logs. Seems that either GMX or Amazon have done something to address this problem.
On Monday morning messages sent via MailGun to our Office 365 account started to fail with the following message:
5.7.511 Access denied, banned sender[198.61.254.54]. To request removal from this list please forward this message to delist#messaging.microsoft.com. For more information please go to http://go.microsoft.com/fwlink/?LinkId=526653.
I checked on MX Toolbox and found their IP 198.61.254.54 listed in SORBS.
I cannot delist the IP from SORBS as the request has to come from the IP address which is listed.
I emailed the delist# address at Microsoft, who replied to say that the ISP/ESP is responsible for delistings.
I opened a ticket on Tuesday at 15:11pm with Mailgun support, asking them to delist the IP or change us to a different IP. It's now 15:42 on Wednesday and they have not acknowledged or replied to my ticket.
We send around 5,000 emails per month so are well under their recommended 50k+ threshold for a dedicated IP address.
Mailgun's control panel says:
For technical questions we recommend asking the community on Stack Overflow.
So here I am. What's my next move?
Mailgun support responded after 51 hours and said: "We have reviewed the error, and we have made adjustments on our side that will help solve this issue moving forward".
They had changed the shared IP address my domains were allocated to, and now mails are being delivered again correctly.
Another possible solution to this would have been for me to purchase a dedicated sending IP, although I was close to the lower recommended limit for this so I'm not sure if they would have improved deliverability in my case. I'm not sure if the purchase process for dedicated IPs is automated, or if I would have had to wait for a support agent to assign the new IP in any case.
I use django sendmail to send mail notifications through gmail.
I allways set fail_silently to false. Obviously, this a way to know if I can connect to gmail and if gmail is able to send the message, but not is enough to know if this e mail address is undeliverable.
I want to keep clean email people table. How can I do to know (by code, not by hand) If an email has been returned as undeliverable.
First of all, the bounce is an asynchronous event. In some cases it might take up to 3 days (or even more) for an undeliverable email to bounce.
In order to detect bounces, your mail server must provide support for it one or another way. Gmail doesn't do this, but you can come up with a heuristic solution using a background process that periodically scans inbox for bounces and parses them using for example imaplib. However, gmail has a sending limit which you will hit sooner or later, so I'm not sure if it's worth investing time in this solution.
Alternatively, you can use an email sending service with bounce tracking capabilities such as Postmark
I am running a web service that currently sends confirmation emails out to new users via the gmail smtp servers. As I'm only getting a few new users each day, this hasn't been a problem.
I've recently added new features to the webapp that will require a customized message to be sent out to each user every day. Think of this as similar to the regular messages LinkedIn sends out that give you a status report on the activity in your network. Every user's message will be different. With thousands of users, this means thousands of unique messages will be sent each day.
Edit: I've since found that these types of email are called "transactional or relationship messages". Spamtacular has a good article on differentiating between marketing and transactional email.
I don't think using gmail's smtp servers will cut it anymore, but I don't know that for sure. I don't know what gmail's maximum outgoing messages per account is (it might be 100/day), but they limit outgoing mail to 500 recipients per message. I'm not sending a single message to 500 recipients, but I'm going to be sending 1000's of customized messages with each recipient getting one per day.
I'm interested to learn any best practices for doing this (especially for Java-based webapps). Here are some of my thoughts and concerns on it:
Should I set up my own outgoing mail server? If I do this, it seems like I'll have all sorts of other issues to worry about, such as preventing mail server abuse, monitoring bounces, allowing ways to opt-out of emails, etc. Are there any tools or services to help with this? Maybe something like OpenEMM or a services like MailChimp? But those seem focused more toward email marketing campaigns.
I don't think I should have the webapp itself handle sending emails as it currently is for new user signups. I'm thinking I should setup a separate messaging server that can access the same backend/datastore as the webapp. Thoughts on this?
Should I consider setting up some sort of message queueing service to help with this, such as JMS, RabbitMQ, ActiveMQ, etc.?
Do I need to provide users a way to opt-out? Do I need to flag these as bulk messages? I don't really consider these email marketing messages, but I'm unsure what is considered appropriate or proper netiquette.
Any advice is appreciated. I'm also very interested in open source tools or web services that simplify things and could help me to ramp up as quickly as possible.
Thanks!
With regard to your first question, yes, you should set up your own mail server. Using gmail to do this might work for a while, but they are likely to shut you down in short order when they see this kind of activity. You could sign up for a business account and use app engine to send messages. Here's a link with information about mail quotas for that service.
Regarding your second and third questions, It would be a good idea to have messages queued by the web app and sent out by a centralized service rather than having the app send out the messages on its own.
Usually I would just use a database table as a queue - the web app inserts rows for each message it wants to send. A service/scheduled task app would grab new messages out of the table and send them off. This gives you lots of flexibility if you want to switch mail servers later, better reliability if the mail server is down, easier diagnostics if there are problems with recipients not getting messages, and the ability to resend messages. As for using JMS/MQ to do this - probably not necessary. IMO a database table used as a queue would give you more flexibility here than an actualy JMS-based queue system.
As for opt outs, YES - you should give people a way to opt out. I don't think you need to flag the messages as bulk though.
On the architecture side of things I would definitely consider decoupling the sending of the emails from the main service via some form of asynchronous message queuing (or facsimile thereof using database as an intermediary). Another benefit of this approach is that if the SMPT server\network is down you could build in retry semantics, additionally for future scalability you could implement multiple mail senders reading from the same queue or implement sending throttling or scheduling (i.e send n messages per hour), etc etc.
From few months ago when i was using twitter, i was able to send twitter a SMS and when i go home and check my twitter page, i see the SMS i sent is on the website as a tweet. "That was great"
Now, i want to make the same in my website, so someone will send my website SMS as a command and my website will save this command in the db for future processing.
My problem that i don't know where to begin.
1- How users will send from around the world while there are different mobile companies in each country, or thats not a problem?
2- How my website will receive and read these SMS? there is a service for that?
3- Do you know any articles which simplify these tasks for me?
If someone worked on something like that before, please advice, any info will be helpful.
Contact your messaging provider, they will have solutions for each country they support.
In practice these things need to be agreed on a per-country bases (e.g. shortcodes etc), but the providers will do a lot to help.
Depending on what countries you want to cover, a single provider will probably do it - if you need absolutely every country with a mobile network, then you might need several, in which case integration is more complicated.
Typically they send either a HTTP POST, or a SMTP email to your server when they receive a message to your company's shortcode or shortcode prefix. But the integration options that exist are agreed per provider; there is no real standard or de-facto standard.
Well, first of all you need a sms-gateway. This is a service which you can buy a lot of places with varying prices. Your site can communicate with this gateway in different ways depending on the gateway-host.
Now, you can send messages to the number you bought on the sms-server and poll them (or push, again depending on your sms-provider) to your site. Just as with any other sms "IRL", you can use country codes to send a very costly SMS from around the globe. If you wish to keep this price lower, you need to rent a SMS-gateway which is internationalized or you need to rent one in each country...
.. In conclusion, doing this is not really a feasable option for your small "hobby-type" project :) Renting a SMS-gateway is rather cheap though, so the problem is really in your "multiple countries" request ..
I have created a web service for sending and receiving SMS messages. We are connecting through VPN to the SMS gateway of the local GSM operator: they have assigned us an public number as well as the option to send messages worldwide.
It doesn't matter if we send sms worldwide or receive from anyone - it just work :)
International sms might be a bit more expensive to send.
Edit:
theoretically there is a possibility to send sms thgrough an sip provider (like betamax /voipdiscount.com/) but this is not so fast and reliable comparing to traditional service.
I've had some success in the past with http://www.aspsms.com/
This is a paid service (per SMS) and be aware that you need to pay and FAX (yes, FAX...) your identity information to the before you get an API key.