I want to append a runbook url to the airflow email alerts on failure. The default email sends other useful information such as airflow log link and I dont want to loose that. How can I achieve this.
I explored email operator but seems like it creates a new email and now I get 2 emails one for dag that failed and other from the email operator.
Is there a way to go about it ?
failure_email = EmailOperator (
task_id='failure-email',
name='failure-email',
to=['oncall#gmail.com'],
subject="T3 Failed",
html_content='<a>https://path/to/runbook</a>',
namespace= ,
dag=dag
)
You can customize the airflow email content by setting the html_content_template variable in the conf, the value of this var should be a path to a jinja2 file.
for example:
airflow.cfg -
[email]
html_content_template = /path/to/template
/path/to/template -
Try {{try_number}} out of {{max_tries + 1}}<br>
Exception:<br>{{exception_html}}<br>
Log: Link<br>
Host: {{ti.hostname}}<br>
Log file: {{ti.log_filepath}}<br>
Mark success: Link<br>
Runbook: <a>https://path/to/runbook</a>
(The default content of the airflow emails was taken from https://github.com/apache/airflow/blob/6e269570cd6dd2d434a3c63ca0db9f1a59dea847/airflow/models/taskinstance.py#L1263).
Related
I'm trying to design a PowerShell script that utilizes AWS SES to send a RawEmail with a .csv attachment, but information is scant on how to actually attach the attachment:
PS C:\Users\jsmith_prod\Desktop\SSMSScript> ls
Directory: C:\Users\jsmith_prod\Desktop\SSMS Script
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2/16/2023 2:00 PM 1096846 Inventory.csv
PS C:\Users\jsmith_prod\Desktop\SSMSScript> Send-SES2Email -FromEmailAddress jsmith#schwarma.com -Destination_ToAddress jsmith#schwarma.com -Raw_Data .\Inventory.csv
0100018661480af8-6a5939bd-5f9a-4f2f-9aa8-084cd872b9a8-000000
PS C:\Users\jsmith_prod\Desktop\SSMSScript>
When I try the above simple command, I only end up with an empty email, rather than an email with the attached .csv. According to AWS's documentation, attachments should be included under Raw_Data in order to highlight that this is a raw email. However, without any helpful error message, I can't figure out what's going wrong.
According to the Send-SES2Email documentation, the -Raw_Data argument is the raw email message itself, not an attachment. As you mentioned, attachments should be included under Raw_Data. The client must ensure that the message format complies with Internet email standards regarding email header fields, MIME types, and MIME encoding. The documentation defines criteria for raw messages and refers to RFC 5321 in that regard.
Please note, that the message body must contain a properly formatted, raw email message, with appropriate header fields and message body encoding. Although it's possible to construct the raw message manually within an application, it's much easier to do so using existing mail libraries. Check out this documentation for examples of how properly construct raw email message with an attachment. With Python it can look something like this (take note of how an attachment is handled):
import os
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication
# Replace sender#example.com with your "From" address.
# This address must be verified with Amazon SES.
SENDER = "Sender Name <sender#example.com>"
# Replace recipient#example.com with a "To" address. If your account
# is still in the sandbox, this address must be verified.
RECIPIENT = "recipient#example.com"
# Specify a configuration set. If you do not want to use a configuration
# set, comment the following variable, and the
# ConfigurationSetName=CONFIGURATION_SET argument below.
CONFIGURATION_SET = "ConfigSet"
# If necessary, replace us-west-2 with the AWS Region you're using for Amazon SES.
AWS_REGION = "us-west-2"
# The subject line for the email.
SUBJECT = "Customer service contact info"
# The full path to the file that will be attached to the email.
ATTACHMENT = "path/to/customers-to-contact.xlsx"
# The email body for recipients with non-HTML email clients.
BODY_TEXT = "Hello,\r\nPlease see the attached file for a list of customers to contact."
# The HTML body of the email.
BODY_HTML = """\
<html>
<head></head>
<body>
<h1>Hello!</h1>
<p>Please see the attached file for a list of customers to contact.</p>
</body>
</html>
"""
# The character encoding for the email.
CHARSET = "utf-8"
# Create a multipart/mixed parent container.
msg = MIMEMultipart('mixed')
# Add subject, from and to lines.
msg['Subject'] = SUBJECT
msg['From'] = SENDER
msg['To'] = RECIPIENT
# Create a multipart/alternative child container.
msg_body = MIMEMultipart('alternative')
# Encode the text and HTML content and set the character encoding. This step is
# necessary if you're sending a message with characters outside the ASCII range.
textpart = MIMEText(BODY_TEXT.encode(CHARSET), 'plain', CHARSET)
htmlpart = MIMEText(BODY_HTML.encode(CHARSET), 'html', CHARSET)
# Add the text and HTML parts to the child container.
msg_body.attach(textpart)
msg_body.attach(htmlpart)
# Define the attachment part and encode it using MIMEApplication.
att = MIMEApplication(open(ATTACHMENT, 'rb').read())
# Add a header to tell the email client to treat this part as an attachment,
# and to give the attachment a name.
att.add_header('Content-Disposition','attachment',filename=os.path.basename(ATTACHMENT))
# Attach the multipart/alternative child container to the multipart/mixed
# parent container.
msg.attach(msg_body)
# Add the attachment to the parent container.
msg.attach(att)
# Print message
print(msg.as_string())
I have verified my domain with AWS SES and as a result my site successfully does send password verification emails (via allauth).
I would, however, also like to be able to send emails based on local scripts. To do this I have been using django shell_plus:
from django.core.mail import send_mail
send_mail("It works!", "This will get sent through anymail",
"me#mysite.com", ["me#mysite.com"])
I get output of '1' suggesting this the email has been sent successfully but I do not receive an email
I think my config is correct given that the site does successfully send emails:
EMAIL_BACKEND = "anymail.backends.amazon_ses.EmailBackend"
ANYMAIL = {
"AMAZON_SES_CLIENT_PARAMS": {
"aws_access_key_id": AWS_ACCESS_KEY_ID,
"aws_secret_access_key": AWS_SECRET_ACCESS_KEY,
"region_name": "us-east-1",
},
}
Can anyone explain what I need to do to send emails from the terminal i.e. without being directly logged into an AWS server?
Was a basic error - I had the required settings in my production config file, but not in my dev config file
I am able to get inbox emails and also able to get emails from specific folder but i am unable to get "starred" emails.
I tried below code. and i am expecting emails with "starred flag" in response.
from imap_tools import MailBox, A
# Create your views here.
def temp(request):
#Get date, subject and body len of all emails from INBOX folder
with MailBox('smtp.gmail.com').login('admin#gmail.com', 'password', 'INBOX') as mailbox:
temp="empty"
for msg in mailbox.fetch():
temp = (msg.date, msg.subject, msg.html)
return HttpResponse(temp)
https://github.com/ikvk/imap_tools/blob/master/examples/search.py
mailbox.fetch(AND(flagged=True))
(A long time ago in a galaxy far far away) star looked like a flag.
I am trying to configure airflow to send an email using AWS SES(HTTP) email backend. The configuration is done according to https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html. But I always get an error "Invalid type for parameter Source". This comes from AWS API where it validates source/from parameter should not be null. But, airflow email API does not accept the parameter code snippet below(ref: https://github.com/apache/airflow/blob/main/airflow/utils/email.py). So my question is how to send the source/from-email parameter. Is there any configuration parameter airflow config file or it takes from AWS IAM credential. I am using AWS IAM credential set with AIRFLOW__EMAIL__EMAIL_CONN_ID parameter in airflow.
def send_email(
to: Union[List[str], Iterable[str]],
subject: str,
html_content: str,
files: Optional[List[str]] = None,
dryrun: bool = False,
cc: Optional[Union[str, Iterable[str]]] = None,
bcc: Optional[Union[str, Iterable[str]]] = None,
mime_subtype: str = 'mixed',
mime_charset: str = 'utf-8',
conn_id: Optional[str] = None,
**kwargs,
):
It took me a while, but one of the main things I missed was that you have to restart the scheduler, not the airflow webserver (but, since my services are called airflow and airflow-scheduler, I forgot the first one is really just the webserver)
Here's an example of my airflow.cfg file:
[email]
# Configuration email backend and whether to
# send email alerts on retry or failure
# Email backend to use
#email_backend = airflow.utils.email.send_email_smtp
email_backend = airflow.providers.amazon.aws.utils.emailer.send_email
# aws_ses is a connection I created in the airflow web UI
email_conn_id = aws_ses
from_email = Name <me#example.com>
As for the AWS side, I set up an access key and secret (which I entered into the airflow web UI to create the aws_ses connection). I also did a few other things related to roles, though I'm not sure if they were needed.
I didn't get the same error you did, but I kept getting an error on sending the email so maybe this will help.
I am trying to login to gmail and read the messages from python package. But getting an Error : [ALERT] Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)
import imaplib
import email
m = imaplib.IMAP4_SSL("imap.gmail.com", 993)
m.login("username#gmail.com","password")
m.select('"[Gmail]/All Mail"')
result, data = m.uid('search', None, "ALL") # search all email and return uids
if result == 'OK':
for num in data[0].split():
result, data = m.uid('fetch', num, '(RFC822)')
if result == 'OK':
email_message = email.message_from_bytes(data[0][1]) # raw email text including headers
print('From:' + email_message['From'])
m.close()
m.logout()
It's work for me:
Turn ON https://myaccount.google.com/lesssecureapps
If still get error then visit https://accounts.google.com/b/4/DisplayUnlockCaptcha (where 4 is your authuser id in list logged in account), and click Continue
Just turn this ON for your Gmail account - https://myaccount.google.com/lesssecureapps