AWS SeS From Lambda (works with Python, not with .Net) - amazon-web-services

I have a .NET lambda that sends emails via SES. I have a VPCe for SeS coupled with all of the required port, subnet & ACL config.
I created a Python lambda to do exactly the same as my .NET Lambda. It's joined to the same subnets, same security group & uses the same SMTP user and password.
The Python code works fine, connects to SES and sends the emails without fail but my .NET Lambda doesn't.
The last thing in my logs is:
ConnectAsync is going to be called
When calling the uri/path it says "Service Unavailable, 500."
I know the code works as if it's in debug on my dev machine pointing at the www-ses endpoint, it works fine.
I've also tried Nat gateways as an alternative (same issue) but I've settled with VPCe as I know my Python lambda works fine.
.NET:
try
{
using var smtp = new SmtpClient();
LambdaLogger.Log($"smtp ConnectAsync is going to be called for host {Configuration["SmtpHost"]} and port {Configuration["SmtpPort"].To<int>()} ");
//await smtp.ConnectAsync(Configuration["SmtpHost"], Configuration["SmtpPort"].To<int>(), SecureSocketOptions.StartTlsWhenAvailable);
await smtp.ConnectAsync("vpce-0af6f6a54b8c6d652-ltbm2mcx.email-smtp.eu-west-2.vpce.amazonaws.com", 587, SecureSocketOptions.StartTlsWhenAvailable);
LambdaLogger.Log($"SMTP connection created on {Configuration["SmtpHost"]} and port {Configuration["SmtpPort"].To<int>()}.");
await smtp.AuthenticateAsync(Configuration["SESUserName"], Configuration["SESPassword"]);
LambdaLogger.Log($"User auth completed for user .");
await smtp.SendAsync(email);
LambdaLogger.Log("email sent.");
emailMessage.isDeleted = true;
_applicationContext.EmailMessages.Update(emailMessage);
smtp.Disconnect(true);
}
catch (Exception ex)
{
emailMessage.Retries = emailMessage.Retries++;
_applicationContext.EmailMessages.Update(emailMessage);
LambdaLogger.Log($"Email Sending Error: {ex.ToFullMessage()}");
}
await _applicationContext.SaveChangesAsync();
Python:
import json
import smtplib
from email.mime.text import MIMEText
def lambda_handler(event, context):
sender = '<removd>'
receivers = ['<removed>']
port = 587
msg = MIMEText('This is test mail')
msg['Subject'] = 'Test mail'
msg['From'] = '<removed>'
msg['To'] = '<remvoed>'
with smtplib.SMTP('vpce-0af6f6a54b8c6d652-ltbm2mcx.email-smtp.eu-west-2.vpce.amazonaws.com', 587) as server:
server.starttls()
server.login('<removed>', '<removed>')
server.sendmail(sender, receivers, msg.as_string())
print("Successfully sent email")

Related

Flask mqtt works fine in the web app console (pythonanywhere) but it doesn't work in the web app itself,

I use pythonanywhere for my IoT flask server. My MQTT code runs locally (visualstudio) but fails under pythonanywhere (code below copied from my similar question on pythonanywhere forum).
I configured MQTT credentials and set tls to false and 5 seconds keepalive then I instantiate mqtt=Mqtt(app). In #mqtt.onConnect() function I printed values when rc is zero but it doesn't print in any log file (even I used sys.stderr) meaning that means it doesn't go in this function, therefore publish and onMessage() doesn't work too.
However if I open python in the bash console and import MQTT instance from app file it connects to my broker and when I call the function that publish from the console it publish the message to my broker.
I tried to mqtt=Mqtt() then in main scope mqtt.init_app(app) and also tried in main mqtt.run() didn't work.
app1.config['MQTT_BROKER_URL'] = 'mybroker'
app1.config['MQTT_BROKER_PORT'] = 1883
app1.config['MQTT_USERNAME'] = ' '
app1.config['MQTT_PASSWORD'] = ' '
app1.config['MQTT_KEEPALIVE'] = 5
app1.config['MQTT_TLS_ENABLED'] = False
mqtt_client = Mqtt(app1)
import sys as syss
#mqtt_client.on_connect()
def handle_connect(client,userdata,flags,rc):
if rc == 0:
print('Connected successfully', file = syss.stderr)
mqtt_client.subscribe('esp/copra')
else:
print('Bad connection. Code:', rc , file=syss.stderr)
testmqtt="a"
#mqtt_client.on_message()
def handle_mqtt_message(client,userdata,message):
payload = message.payload.decode()
print("payload is " + payload)
#app1.route("/testmqtt")
def testmqtt():
print("test mqtt here")
mqtt_client.publish('esp/copra',"pythonanywhere")
return{"mqtt": "mqtt"}
if name == "main":
#app.config['SESSION_TYPE'] = 'filesystem'
mqtt_client.init_app(app1)
app1.run()

opensearch authentication with opensearch-py on aws lambda

I am trying to connect to AWS OpenSearch domain from AWS Lambda using the opensearch python client (development purposes, non production).
I was trying the following:
from opensearchpy import OpenSearch
import boto3
from requests_aws4auth import AWS4Auth
import os
import config
my_region = os.environ['AWS_REGION']
service = 'es' # still es???
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, my_region, service, session_token=credentials.token)
openSearch_endpoint = config.openSearch_endpoint
# sth wrong here:
openSearch_client = OpenSearch(hosts = [openSearch_endpoint], auth = awsauth)
as per the following blogs:
https://aws.amazon.com/blogs/database/indexing-metadata-in-amazon-elasticsearch-service-using-aws-lambda-and-python/
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/search-example.html
but it does not work (it does not want to authenticate, "errorMessage":"AuthorizationException(403, '')" . However if I don't use the python client but simply go through requests instead:
import requests
host = config.openSearch_endpoint
url = host + '/' +'_cat/indices?v'
# this one works:
r = requests.get(url, auth=awsauth)
, my lambda function does communicate with the OpenSearch domain.
I consulted the OpenSearch() documentation but it is not clear to me how its parameters map to boto3 session credentials, and/or to AWS4Auth. So what should this line
openSearch_client = OpenSearch(hosts = [openSearch_endpoint], auth = awsauth)
be?
actually managed to find the solution a couple of hours later:
from opensearchpy import OpenSearch, RequestsHttpConnection
my_region = os.environ['AWS_REGION']
service = 'es' # still es?
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, my_region, service, session_token=credentials.token)
host = config.openSearch_endpoint
openSearch_client = OpenSearch(
hosts=[openSearch_endpoint],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
ssl_assert_hostname = False,
ssl_show_warn = False,
connection_class=RequestsHttpConnection
)
OpenSearch Service requires port 443 for incoming requests therefore you need to add a new Inbound Rule under Security Group attached to your OpenSearch Service domain.
Try the rule:
Type: HTTPS
Protocol: TCP
Port range: 443
Source: 0.0.0.0/0 (Anywhere-IPv4)
Additionally, you should have a Resource-based policy for your Lambda function to perform requests to your OpenSearch Service domain.

Connect to RabbitMQ instance remotely (created on AWS)

I'm having trouble connecting to a RabbitMQ instance (it's my first time doing so). I've spun one up on AWS, and been given access to an admin panel which I'm able to access.
I'm trying to connect to the RabbitMQ server in python/pika with the following code:
import pika
import logging
logging.basicConfig(level=logging.DEBUG)
credentials = pika.PlainCredentials('*******', '**********')
parameters = pika.ConnectionParameters(host='a-25c34e4d-a3eb-32de-abfg-l95d931afc72f.mq.us-west-1.amazonaws.com',
port=5671,
virtual_host='/',
credentials=credentials,
)
connection = pika.BlockingConnection(parameters)
I get pika.exceptions.IncompatibleProtocolError: StreamLostError: ("Stream connection lost: ConnectionResetError(54, 'Connection reset by peer')",) when I run the above.
you're trying to connect through AMQP protocol and AWS is using AMQPS, you should add ssl_options to your connection parameters like this
import ssl
logging.basicConfig(level=logging.DEBUG)
credentials = pika.PlainCredentials('*******', '**********')
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
parameters = pika.ConnectionParameters(host='a-25c34e4d-a3eb-32de-abfg-l95d931afc72f.mq.us-west-1.amazonaws.com',
port=5671,
virtual_host='/',
credentials=credentials,
ssl_options=pika.SSLOptions(context)
)
connection = pika.BlockingConnection(parameters)

Why is my lambda not able to talk to elasticache?

I have an Redis ElastiCache cluster that has a FQDN for the primary node in the format: master.clustername.x.euw1.cache.amazonaws.com. I also have a Route53 record with the CNAME pointing at that FQDN.
I have a .net core lambda in the same VPC as the cluster, with access to the cluster via security groups. The lambda talks to the cluster using the Redis library developed by Stack Overflow (Github repo here for reference).
If I give the lambda the hostname the FQDN for the Redis cluster (the one that starts with master) I can connect, save data and read it.
If I give the lambda the CNAME (and that CNAME gives the same IP address as the FQDN when I ping it from my local machine and also if I use Dns.GetHostEntry within the lambda) it doesn't connect and I get the following error message:
One or more errors occurred. (It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING): AggregateException
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at lambda_method(Closure , Stream , Stream , LambdaContextInternal )
at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(Func`1 multiplexerFactory, TextWriter log) in c:\code\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 890
at lambda.Redis.RedisClientBuilder.Build(String redisHost, String redisPort, Int32 redisDbId) in C:\BuildAgent\work\91d24911506461d0\src\Lambda\Redis\RedisClientBuilder.cs:line 9
at lambda.Ioc.ServiceBuilder.GetRedisClient() in C:\BuildAgent\work\91d24911506461d0\src\Lambda\IoC\ServiceBuilder.cs:line 18
at lambda.Ioc.ServiceBuilder.GetServices() in C:\BuildAgent\work\91d24911506461d0\src\Lambda\IoC\ServiceBuilder.cs:line 11
at Handlers.OrderHandler.Run(SNSEvent request, ILambdaContext context) in C:\BuildAgent\work\91d24911506461d0\src\Lambda\Handlers\OrderHandler.cs:line 26
Has anyone seen anything similar to this?
It turned out that because I was using an SSL certificate on the elasticache cluster and the SSL certificate was bound the the master. endpoint whereas I was trying to connect to the CNAME, the certificate validation was failing.
So I ended up querying the Route53 record within the code to get the master endpoint and it worked.
Possible workaround to isolate problems from your client lib -- follow AWS' tutorial and rewrite your Lambda to something like the code below (example in Python).
from __future__ import print_function
import time
import uuid
import sys
import socket
import elasticache_auto_discovery
from pymemcache.client.hash import HashClient
#elasticache settings
elasticache_config_endpoint = "your-elasticache-cluster-endpoint:port"
nodes = elasticache_auto_discovery.discover(elasticache_config_endpoint)
nodes = map(lambda x: (x[1], int(x[2])), nodes)
memcache_client = HashClient(nodes)
def handler(event, context):
"""
This function puts into memcache and get from it.
Memcache is hosted using elasticache
"""
#Create a random UUID... this will the sample element we add to the cache.
uuid_inserted = uuid.uuid4().hex
#Put the UUID to the cache.
memcache_client.set('uuid', uuid_inserted)
#Get item (UUID) from the cache.
uuid_obtained = memcache_client.get('uuid')
if uuid_obtained.decode("utf-8") == uuid_inserted:
# this print should go to the CloudWatch Logs and Lambda console.
print ("Success: Fetched value %s from memcache" %(uuid_inserted))
else:
raise Exception("Value is not the same as we put :(. Expected %s got %s" %(uuid_inserted, uuid_obtained))
return "Fetched value from memcache: " + uuid_obtained.decode("utf-8")
Reference: https://docs.aws.amazon.com/lambda/latest/dg/vpc-ec-deployment-pkg.html

code runs indefinitely while trying to send mail via python

I am trying to send mail using python via my own setup mail server
import smtplib
SERVER='myserverdomain.com'
server = smtplib.SMTP(SERVER,587)
server.starttls()
server.login(USER,PASS)
server.sendmail(FROM, TO, message)
server.quit()
But when I run the above code after line 3 it is not executing.Kind of like goes into an infinite loop.Why could this be
I am able to ping my server successfully.
import smtplib
SERVER='myserverdomain.com'
server = smtplib.SMTP_SSL(SERVER,587)
server.login(USER,PASS)
server.sendmail(FROM, TO, message)
server.quit()
This answer worked