The Configs
We did the usual main.cf (postfix) relay/sasl entries:
# Amavisd + SpamAssassin + ClamAV
#
content_filter = smtp-amavis:[127.0.0.1]:10024
# Concurrency per recipient limit.
smtp-amavis_destination_recipient_limit = 1
relayhost = [smtp.mailgun.org]:587
smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:login#domain.com:password
smtp_sasl_security_options = noanonymous
This actually works perfectly fine, and mail deliveries are properly logged under Mailgun too.
The Problem
If we send an email within our domain, it's not triggering Mailgun. Means: if JohnDoe#ourXYZDomain.com sends an email to JaneDoe#ourXYZDomain, it is delivered via postfix. If JohnDoe#ourXYZDomain.com sends an email to JohnDoe#someOtherDomain.com, it is delivered and logged via Mailgun smtp.
The Analysis
Mails within same domain are sent and no errors are thrown. When looking into header of received email, it shows that postfix didn't even bother using Mailgun. See localhost [127.0.0.1] in line 6:
Subject:test - 00:11
Contact photo
From johndoe#ourXYZDomain.com Date Mon 00:11
Return-Path: <johndoe#ourXYZDomain.com>
Delivered-To: janedoe#ourXYZDomain.com
Received: from mail.ourXYZDomain.com (localhost [127.0.0.1])
by mail.ourXYZDomain.com (Postfix) with ESMTP id 49KwDw97hggXdtN
for <janedoe#ourXYZDomain.com>; Sun, 10 May 2020 20:11:12 +0000 (UTC)
Authentication-Results: mail.ourXYZDomain.com (amavisd-new); dkim=pass
reason="pass (just generated, assumed good)"
header.d=ourXYZDomain.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=
ourXYZDomain.com; h=user-agent:message-id:subject:subject
:to:from:from:date:date:content-transfer-encoding:content-type
:content-type:mime-version; s=dkim; t=1589141471; x=1591733472;
bh=WonWKNs0MVBQ4Md9bT3TQ0-----=; b=1lp9qX-----YE
HQwrRVwjLjcPcP/jkjhgjghgfWisfODNZ5xHnQto5Xa
D6/Wj8fAEpwiu8uG5Ujhugz778gjNZ8UhFXtJf2aK
1B8iZembDuiIsjg6fKj6snRjA=
X-Virus-Scanned: amavisd-new at mail.ourXYZDomain.com
Received: from mail.ourXYZDomain.com ([127.0.0.1])
by mail.ourXYZDomain.com (mail.ourXYZDomain.com [127.0.0.1]) (amavisd-new, port 10026)
with ESMTP id xzds0121548c for <janedoe#ourXYZDomain.com>;
Sun, 10 May 2020 20:11:11 +0000 (UTC)
Received: from _ (localhost [127.0.0.1])
by mail.ourXYZDomain.com (Postfix) with ESMTPSA id 49KwDv54101252XdtL
for <janedoe#ourXYZDomain.com>; Sun, 10 May 2020 20:11:11 +0000 (UTC)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
format=flowed
Content-Transfer-Encoding: 7bit
Date: Mon, 11 May 2020 00:11:11 +0400
From: johndoe#ourXYZDomain.com
To: janedoe#ourXYZDomain.com
Subject: test - 00:11
Message-ID: <1003f6125a2ff654101214#ourXYZDomain.com>
X-Sender: johndoe#ourXYZDomain.com
We have no explanation for this behaviour. Maybe we missed something?
Thank you for any hint
When You send mail from one $mydomain virtual mailbox to another $mydomain virtual mailbox postfix will not relay it to Mailgun and transport it locally.
You need to create multiple instances of Postfix where one instance will relay all mails to Mailgun without local delivery and second instance will listen 25 port for income mail and will deliver it to virtual boxes.
Explanation of this solution You can find there: http://www.postfix.org/MULTI_INSTANCE_README.html
Mailgun is one of many Mail Relay service mostly used to relay mails to outside networks. Mail relay is the process of transferring an email from one server to another for delivery. For example, if you work for Company A and send an email to someone at Company B, you connect to your company's SMTP server which then relays your email to the server owned by Company B.
To send mail on same network you do not need a relayhost. Mails here are deliver locally by postfix to other user's account. Local mails depend on "myorigin" "mydestination" mynetworks" settings in main.cf. The following mailhost configuration is an example which will make things clear to you.
/etc/postfix/main.cf:
myorigin = $mydomain
mydestination = $myhostname localhost.$mydomain localhost $mydomain
mynetworks = 127.0.0.0/8 10.0.0.0/24
relayhost = [smtp.mailgun.org]:587
# Optional: forward all non-local mail to firewall
#relayhost = [firewall.example.com]
You can find more information in following link
Postfix Standard Configuration Examples
Related
I am trying to have Airflow email me using AWS SES whenever a task in my DAG fails to run or retries to run. I am using my AWS SES credentials rather than my general AWS credentials too.
My current airflow.cfg
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = emailsmtpserver.region.amazonaws.com
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
smtp_user = REMOVEDAWSACCESSKEY
smtp_password = REMOVEDAWSSECRETACCESSKEY
smtp_port = 25
smtp_mail_from = myemail#myjob.com
Current task in my DAG that is designed to intentionally fail and retry:
testfaildag_library_install_jar_jdbc = PythonOperator(
task_id='library_install_jar',
retries=3,
retry_delay=timedelta(seconds=15),
python_callable=add_library_to_cluster,
params={'_task_id': 'cluster_create', '_cluster_name': CLUSTER_NAME, '_library_path':s3000://fakepath.jar},
dag=dag,
email_on_failure=True,
email_on_retry=True,
email=’myname#myjob.com’,
provide_context=True
)
Everything works as designed as the task retries the set number of times and ultimately fails, except no emails are being sent. I have checked the logs in the task mentioned above too, and smtp is never mentioned.
I've looked at the similar question here, but the only solution there did not work for me. Additionally, Airflow's documentation such as their example here does not seem to work for me either.
Does SES work with Airflow's email_on_failure and email_on_retry functions?
What I am currently thinking of doing is using the on_failure_callback function to call a python script provided by AWS here to send an email on failure, but that is not the preferable route at this point.
Thank you, appreciate any help.
--updated 6/8 with working SES
here's my write up on how we got it all working. There is a small summary at the bottom of this answer.
Couple of big points:
We decided not to use Amazon SES, and rather use sendmail We now have SES up and working.
It is the airflow worker that services the email_on_failure and email_on_retry features. You can do journalctl –u airflow-worker –f to monitor it during a Dag run. On your production server, you do NOT need to restart your airflow-worker after changing your airflow.cfg with new smtp settings - it should be automatically picked up. No need to worry about messing up currently running Dags.
Here is the technical write-up on how to use sendmail:
Since we changed from ses to sendmail on localhost, we had to change our smtp settings in the airflow.cfg.
The new config is:
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = localhost
smtp_starttls = False
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
#smtp_user = not used
#smtp_password = not used
smtp_port = 25
smtp_mail_from = myjob#mywork.com
This works in both production and local airflow instances.
Some common errors one might receive if their config is not like mine above:
socket.error: [Errno 111] Connection refused -- you must change your smtp_host line in airflow.cfg to localhost
smtplib.SMTPException: STARTTLS extension not supported by server. -- you must change your smtp_starttls in airflow.cfg to False
In my local testing, I tried to simply force airflow to show a log of what was going on when it tried to send an email – I created a fake dag as follows:
# Airflow imports
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
# General imports
from datetime import datetime,timedelta
def throwerror():
raise ValueError("Failure")
SPARK_V_2_2_1 = '3.5.x-scala2.11'
args = {
'owner': ‘me’,
'email': ['me#myjob'],
'depends_on_past': False,
'start_date': datetime(2018, 5,24),
'end_date':datetime(2018,6,28)
}
dag = DAG(
dag_id='testemaildag',
default_args=args,
catchup=False,
schedule_interval="* 18 * * *"
)
t1 = DummyOperator(
task_id='extract_data',
dag=dag
)
t2 = PythonOperator(
task_id='fail_task',
dag=dag,
python_callable=throwerror
)
t2.set_upstream(t1)
If you do the journalctl -u airflow-worker -f, you can see that the worker says that it has sent an alert email on the failure to the email in your DAG, but we were still not receiving the email. We then decided to look into the mail logs of sendmail by doing cat /var/log/maillog. We saw a log like this:
Jun 5 14:10:25 production-server-ip-range postfix/smtpd[port]: connect from localhost[127.0.0.1]
Jun 5 14:10:25 production-server-ip-range postfix/smtpd[port]: ID: client=localhost[127.0.0.1]
Jun 5 14:10:25 production-server-ip-range postfix/cleanup[port]: ID: message-id=<randomMessageID#production-server-ip-range-ec2-instance>
Jun 5 14:10:25 production-server-ip-range postfix/smtpd[port]: disconnect from localhost[127.0.0.1]
Jun 5 14:10:25 production-server-ip-range postfix/qmgr[port]: MESSAGEID: from=<myjob#mycompany.com>, size=1297, nrcpt=1 (queue active)
Jun 5 14:10:55 production-server-ip-range postfix/smtp[port]: connect to aspmx.l.google.com[smtp-ip-range]:25: Connection timed out
Jun 5 14:11:25 production-server-ip-range postfix/smtp[port]: connect to alt1.aspmx.l.google.com[smtp-ip-range]:25: Connection timed out
So this is probably the biggest "Oh duh" moment. Here we are able to see what is actually going on in our smtp service. We used telnet to confirm that we were not able to connect to the targeted IP ranges from gmail.
We determined that the email was attempting to be sent, but that the sendmail service was unable to connect to the ip ranges successfully.
We decided to allow all outbound traffic on port 25 in AWS (as our airflow production environment is an ec2 instance), and it now works successfully. We are now able to receive emails on failures and retries (tip: email_on_failure and email_on_retry are defaulted as True in your DAG API Reference - you do not need to put it into your args if you do not want to, but it is still good practice to explicitly state True or False in it).
SES now works. Here is the airflow config:
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = emailsmtpserver.region.amazonaws.com
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
smtp_user = REMOVEDAWSACCESSKEY
smtp_password = REMOVEDAWSSECRETACCESSKEY
smtp_port = 587
smtp_mail_from = myemail#myjob.com (Verified SES email)
Thanks!
Similar case here, I tried to follow the same debugging process but got no log output. Also, the outbound rule for my airflow ec2 instance is open to all ports and ips, so it should be some other causes.
I noticed that when you create the SMTP credential from SES, it will also create an IAM user. I am not sure how is airflow running in your case (bare metal on ec2 instance or wrapped in containers), and how that user access is set up.
I will try to setup outgoing mail server in odoo 9 .so i fill all the field and test connection and the connection also success , but at the time of send mail it will generate an error .
Field fill like that:-
Name : sendmail
Priority: 10
SMTP Server : smtp.office365.com
SMTP Port:25
Debugging: enable
Connection Security:TLS (STARTTLS)
Username:my yser name
Password:password
But, when we send any mail then it will generate the below error
16-12-06 10:04:28,440 426 INFO test openerp.addons.base.ir.ir_mail_server: Mail delivery failed via SMTP server 'smtp.office365.com'.
SMTPDataError: 550
5.7.60 SMTP; Client does not have permissions to send as this sender
2016-12-06 10:04:28,443 426 ERROR test openerp.addons.mail.models.mail_mail: failed sending mail (id: 136) due to Mail Delivery Failed
Mail delivery failed via SMTP server 'smtp.office365.com'.
SMTPDataError: 550
5.7.60 SMTP; Client does not have permissions to send as this sender
Traceback (most recent call last):
File "/usr/lib/python2.7/dist- packages/openerp/addons/mail/models/mail_mail.py", line 262, in send
res = IrMailServer.send_email(msg, mail_server_id=mail.mail_server_id.id)
File "/usr/lib/python2.7/dist-packages/openerp/api.py", line 248, in wrapper
return new_api(self, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/openerp/api.py", line 490, in new_api
result = method(self._model, cr, uid, *args, **old_kwargs)
File "/usr/lib/python2.7/dist-packages/openerp/addons/base/ir/ir_mail_server.py", line 483, in send_email
raise MailDeliveryException(_("Mail Delivery Failed"), msg)
MailDeliveryException: (u'Mail Delivery Failed', u"Mail delivery failed via SMTP server 'smtp.office365.com'.\nSMTPDataError: 550\n5.7.60 SMTP; Client does not have permissions to send as this sender")
So i tried too much for this , but i am not getting any solution , if you have any solutin , please share with me .
Use port 587.
The error message tells you that the sender is invalid - you can only send as the mailbox owner (primary SMTP address) or as one of the proxy addresses associated with the mailbox.
Removing all catchall parameters (mail.catchall.domain and mail.catchall.alias) under "Settings" -> "Technical" -> "Parameters" -> "System Parameters" and it work like charm .
WORKS LIKE CHARM :
Removing all catchall parameters (mail.catchall.domain and mail.catchall.alias) under "Settings" -> "Technical" -> "Parameters" -> "System Parameters" and it work like charm . TY Debasish
I'm on Odoo V12
It was not suffisient for my problem, I had to delete the alias domain but there another thing to check :
I initialy created my odoo installation with a GMAIL address, worked a bit but had to switch for a pro e-mail because all my invitations e-mail was blocked by Google Bot beacause it look liked suspicious. It did this only in Odoo v12 because there is more links in the mail.
So I configured my real smtp server in Odoo but get the error 550. Odoo kept in the COMPANY settings the primary gmail address and tried to send on my other smtp server with the gmail name. The other server didn't accepted it so sent me back error 550.
Once i putted my new e-mail address in the company description, and deleted alias domain it worked !!
PS : Don't try to edit ir_mail_server.py to put in bruteforce your e-mail ... Doesn't work ..
I have been asked to do some system admin and to move a legacy PHP web application to an Amazon EC2 instance running Debian. I have done this, and emails are successfully being sent from postfix.
Concern was expressed by the previous system admin that the server was not using an email relay, and a request to use SES seemed straight forward. I have implemented a mail relay using Mailgun from a Rackspace instance, and though not trivial, I got this done in a couple of hours.
I have not found the SES process quite so simple, and I suspect this is because I am unfamiliar with using certificates.
Initially I set up the service using the instructions here
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix.html
Elastic IP set up for server
Credentials created for SMTP server
Created IAM user and got a username and password for SMTP at
email-smtp.us-west-2.amazonaws.com
I created an /etc/postfix/sasl_passwd file with
[email-smtp.us-west-2.amazonaws.com]:25 USERNAME:PASSWORD
I then ran
postmap hash:/etc/postfix/sasl_passwd
to create the sasl_passwd.db
/etc/postfix/master.cf did not have smtp_fallback_relay in it
I created a certificate by installing apt-get install sasl2-bin and
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
and pointing postfix to this in my main.cf (at the end of this post).
I am using sendmail to send an email via Python
SENDMAIL = "/usr/sbin/sendmail" # sendmail location
FROM = "andy#travelinsurancequotes.com.au"
#TO = ["kirstie#travelinsurancequotes.com.au", "jason#slatescience.com"]
TO = ["jason#slatescience.com"]
SUBJECT = "Artog SMTP server is working!"
TEXT = "Sending emails on the TIQ webserver is working"
# Prepare actual message
message = """\
From: %s
To: %s
Subject: %s
%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)
# Send the mail
import os
p = os.popen("%s -f %s -t -i" % (SENDMAIL, FROM), "w")
p.write(message)
status = p.close()
if status:
print "Sendmail exit status", stat
but I keep getting a time out error on sending:
Feb 26 03:18:19 lamp postfix/error[23414]: 5DE3240508: to=<jason#slatescience.com>, relay=none, delay=0.02, delays=0.02/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to email-smtp.us-west-2.amazonaws.com[54.187.123.10]:25: Connection timed out
I can connect via port 25
root#lamp /home/www# telnet email-smtp.us-west-2.amazonaws.com 25
Trying 54.149.142.243...
Connected to ses-smtp-us-west-2-prod-14896026.us-west-2.elb.amazonaws.com.
Escape character is '^]'.
220 email-smtp.amazonaws.com ESMTP
My main.cf file is
myhostname = travelinsurancequotes.com.au
mydomain = travelinsurancequotes.com.au
inet_interfaces = all
mynetworks_style = host
local_destination_recipient_limit = 300
local_destination_concurrency_limit = 5
recipient_delimiter=+
smtpd_banner = $myhostname
smtpd_sasl_auth_enable = yes
smtp_sasl_mechanism_filter = plain
smtpd_sasl_local_domain = $myhostname
broken_sasl_auth_clients = yes
smtpd_helo_required = yes
smtp_use_tls = yes
smtpd_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtpd_tls_key_file = /etc/postfix/sslcerts/server.key
smtpd_tls_cert_file = /etc/postfix/sslcerts/server.crt
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom
relayhost = [email-smtp.us-west-2.amazonaws.com]:25
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
AWS EC2 has some sort of limit on mail being sent ..
I had that error, and Amazon Support told me to fill this form out to remove the limit.
https://aws.amazon.com/forms/ec2-email-limit-rdns-request
I hope this helps
Why am I getting a 500 error? I have not been able to successfully receive an email into my gmail account in developent
Here what is in my server log:
tarted POST "/contacts" for ::1 at 2015-04-06 12:23:42 -0500
Processing by ContactsController#create as */*
Parameters: {"utf8"=>"✓","authenticity_token"=>"2rAtjm6kGBtqrHwLM1oCdrjzJUhYuSR2eX+/4fJpgcCF2TvDs+jKtOPgXmSPOV/eQ9eIRlftynyB6VZKqHw4Ew==", "contact"=>{"name"=>"ahdiwhfjfhew", "email"=>"iahfiwhef#yahoo.com", "message"=>"kfbjwefawjbfjwbeakfekwjfjk3fwoefowqou"}, "commit"=>"Send Message"}
DEPRECATION WARNING: `#deliver` is deprecated and will be removed in Rails 5. Use `#deliver_now` to deliver immediately or `#deliver_later` to deliver through Active Job. (called from create at /Users/corneliusmurphy/spartan_strength_and_power/app/controllers/contacts_controller.rb:9)
Rendered /usr/local/lib/ruby/gems/2.2.0/gems/mail_form-1.5.0/lib/mail_form/views/mail_form/contact.erb (0.6ms)
MailForm::Notifier#contact: processed outbound mail in 83.5ms
Sent mail to neil.murphy0600#gmail.com (817.1ms)
Date: Mon, 06 Apr 2015 12:23:42 -0500
From: ahdiwhfjfhew <iahfiwhef#yahoo.com>
To: neil.murphy0600#gmail.com
Message-ID: <5522c11eb5b6b_596c3fc77b83f2c055624#CorneliussMBP3.mail>
Subject: My Contact Form
Mime-Version: 1.0
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit
<h4 style="text-decoration:underline">My Contact Form</h4>
<p><b>Name:</b>
ahdiwhfjfhew</p>
<p><b>Email:</b>
iahfiwhef#yahoo.com</p>
<p><b>Message:</b>
kfbjwefawjbfjwbeakfekwjfjk3fwoefowqou</p>
Completed 500 Internal Server Error in 902ms
Net::SMTPAuthenticationError (534-5.7.14 <https://accounts.google.com/ContinueSignIn?sarp=1&scc=1&plt=AKgnsbt4U):app/controllers/contacts_controller.rb:9:in `create'
The answer was Because Google was an issue with Google security feature that can be enable/disabled within Gmail
I'm using dcramer's fork of django-paypal, but I always encounter an invalid IPN while working with my sandbox accounts.
I receive the following IPN:
Invalid postback. (INVALID)
I tried everything that showed up on google:
checked seller & buyer emails
sandbox accounts are both verified
I use form.sandbox to render the paypal form
tried removing custom values
there is no non-ascii character in the request
When manually checking the request with https://www.sandbox.paypal.com/cgi-bin/webscr, I also get INVALID.
Did someone encounter this issue ? Is there any more-verbose page to validate ipn requests ?
Yes, I also get errors on post-back starting yesterday (18 June):
Opened POST Back Socket to PayPal.
PayPal Post Back returns HTTP/1.0 400 Bad Request
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 216
Expires: Mon, 18 Jun 2012 22:18:00 GMT
Date: Mon, 18 Jun 2012 22:18:00 GMT
Connection: close
<HTML><HEAD>
<TITLE>Invalid URL</TITLE>
</HEAD><BODY>
<H1>Invalid URL</H1>
The requested URL "/cgi-bin/webscr", is invalid.<p>
....
</BODY></HTML>
: not handled.
I use my own IPN integration. It tries to handle all replies from PayPal, which is why I get the last message (: not handled.) I made a package upgrade yesterday, so I'm not quite sure it is a PayPal problem though.