Queued Emails Fail in AWS VPC Environment - amazon-web-services

I have a system that sends out emails and queues them if volume increases.
If it's 1 or 2 emails per hour, it sends out fine.
On busier times, the queue gets backed up and soon email sending starts to fail.
I've load-tested the SMTP server from my local - no problems there.
However, when I try to do the load-test from my EC2 server
- which is in a private subnet of a VPC
- where outgoing traffic goes through a NAT
- and where we've set up our own DNS server
the first few go out fine - and the rest starts to throw errors.
Here's the C# exception ToString()
System.Net.Mail.SmtpException: Failure sending mail. ---> System.Net.WebExcept
ion: Unable to connect to the remote server ---> System.Net.Sockets.SocketExcept
ion: A connection attempt failed because the connected party did not properly re
spond after a period of time, or established connection failed because connected
host has failed to respond [SMTPServerIP]:25
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddre
ss socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Sock
et s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state,
IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object ow
ner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket
6)
at System.Net.PooledStream.Activate(Object owningObject, Boolean async, Gener
alAsyncDelegate asyncCallback)
at System.Net.PooledStream.Activate(Object owningObject, GeneralAsyncDelegate
asyncCallback)
at System.Net.ConnectionPool.GetConnection(Object owningObject, GeneralAsyncD
elegate asyncCallback, Int32 creationTimeout)
at System.Net.Mail.SmtpConnection.GetConnection(ServicePoint servicePoint)
at System.Net.Mail.SmtpTransport.GetConnection(ServicePoint servicePoint)
at System.Net.Mail.SmtpClient.GetConnection()
at System.Net.Mail.SmtpClient.Send(MailMessage message)
--- End of inner exception stack trace ---
at System.Net.Mail.SmtpClient.Send(MailMessage message)
at ConsoleApplication2.Program.Main(String[] args) in c:\ConsoleApplication2\
Program.cs:line 49
I'm pretty sure it's not a code issue - most likely a setting on the NAS or DNS.
Can someone point me in the right direction?

After some more googling - it seems to be a throttle on AWS' end
Source:
http://aws.amazon.com/ec2/faqs/
Q: Are there any limitations in sending email from EC2 instances?
Yes. In order to maintain the quality of EC2 addresses for sending email, we enforce default limits on the amount of email that can be sent from EC2 accounts. If you wish to send larger amounts of email from EC2, you can apply to have these limits removed from your account by filling out this form .
Better details here:
Amazon EC2/SES SMTP Timeout

Related

SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: (null) when communicating with Chef Automate server

I am having difficulty connecting to my Chef Automate server, hosted on AWS OpsWorks.
I am usually connecting to it at least once per day, however since the start of the week I have been unable to.
There is some weekly maintenance performed on the server on a Friday, however this seems to go unnoticed.
When I try and communicate with the server I get the following error:
knife environment from file environments/production.json
ERROR: SSL Validation failure connecting to host: crmpicco-production-lay0vgyp4ighjsxv.us-east-1.opsworks-cm.io - SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: (null)
ERROR: SSL Error connecting to https://crmpicco-production-lay0vgyp4ighjsxv.us-east-1.opsworks-cm.io/organizations/rfc1872/environments/production, retry 1/5
In the events, I can see the following:
2022-08-26T12:25:26Z Maintenance completed successfully
2022-08-26T12:24:54Z Updating stack arn:aws:cloudformation:us-east-1:367114569123:stack/aws-opsworks-cm-instance-mc-prod-chef-1661515433111/27c16c50-2537-22ed-80ab-12a4e5696267 to associate EIP 2.51.125.211
2022-08-26T12:24:23Z Updating stack arn:aws:cloudformation:us-east-1:367114569123:stack/aws-opsworks-cm-instance-mc-prod-chef-1660910626222/fad95750-1fb6-22ed-817f-0aca43928f1d to disassociate EIP 2.51.125.211
2022-08-26T12:24:11Z Checking health of new instance
I have tried a knife ssl fetch, but that is also unable to communicate with the server.

boost::asio::connect reports success on wrong subnet

Using Boost v1.74:
int main()
{
auto ctx = boost::asio::io_context{};
auto socket = boost::asio::ip::tcp::socket{ctx};
auto ep = boost::asio::ip::tcp::endpoint{
boost::asio::ip::make_address_v4("192.168.0.52"),
80};
boost::asio::connect(socket, std::array{std::move(ep)});
std::cout << "Success!" << std::endl;
}
The IP address of my machine on my local network is 192.168.0.31/24, and so trying to connect to a non-existent address in the same subnet with the above code gives:
10:24:55: Starting /home/cmannett85/workspace/build-scratch-Desktop-Debug/scratch ...
terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
what(): connect: No route to host
10:24:59: The program has unexpectedly finished.
This is all expected. If I change the bottom octet of the subnet in the address (e.g. 192.168.1.52), then the app just waits for a few minutes - presumably because it sent messages to any routers to see if they own the requested subnet. There aren't any routers on my network, so it eventually times out:
10:27:39: Starting /home/cmannett85/workspace/build-scratch-Desktop-Debug/scratch ...
terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
what(): connect: Connection timed out
10:29:49: The program has unexpectedly finished.
Again, as expected. If I change the next octet (e.g. 192.167.0.52) instead, I would expect this to behave exactly the same as it is an equally unknown subnet as the previous. But it suceeds!
10:31:22: Starting /home/cmannett85/workspace/build-scratch-Desktop-Debug/scratch ...
Success!
This address is definitely not on my network:
$ ping 192.167.0.52
PING 192.167.0.52 (192.167.0.52) 56(84) bytes of data.
^C
--- 192.167.0.52 ping statistics ---
17 packets transmitted, 0 received, 100% packet loss, time 16368ms
So why is the code reporting that it is connected? And why is changing the second octet different to the third?
Any IP address of the form 192.168.xx.xx is a non-internet-routable network. This means no internet routers will route it. So the only way packets get routed off your subnet is if you configure a route on your own router or host. 192.167.xx.xx is an internet routable network, Presumable there is a host out there on the internet that uses the address you specified. So if you can connect your host to the internet, some internet router will get your packet to the address specified.
It's something related to my VPN. I didn't think it was relevant as the tunnel address is 10.17.0.60/16, but disabling it makes the above code work as expected.
Thanks to a suggestion by #dewaffled, Curl is showing that there is something on the otherside of this connection that is completing the TCP handshake, but after a timeout of a few minutes closes the connection.
$ curl -v http://192.167.0.52
* Trying 192.167.0.52:80...
* Connected to 192.167.0.52 (192.167.0.52) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.167.0.52
> User-Agent: curl/7.74.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
I know nothing about how VPNs work, but I suspect this is an implementation detail of my particular provider. Hopefully this 'answer' will limit confusion for anyone else!

Frequent HttpRequestException when talking to Cosmos DB from Web App

We have a web service (Azure App Service) deployed to Azure that talks to our Azure Cosmos DB via the standard C# SDK for Cosmos DB/Document DB.
Both - app service and Cosmos DB account/collections - are in the same resource group and in the same location in Azure.
For certain bulk operations where the web service performs a burst of requests to Cosmos DB we are frequently getting errors in the web service when talking to the database:
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.InternalBind(EndPoint localEP)
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
[...]
at Microsoft.Azure.Documents.Query.ProxyDocumentQueryExecutionContext.<ExecuteNextAsync>d__0.MoveNext()<---
Each of our ApiController instances statically allocates a single repository class, which in turn fetches a IReliableReadWriteDocumentClient instance in its constructor and holds it for its entire lifetime via
IDocumentDbInitializer dbinit = new DocumentDbInitializer();
Client = dbinit.GetClient(endpointUrl, myAuthKey, connectionPolicy);
So in my understanding we should use only 2 Document DB clients for our 2 repositories in the entire web service.
Things we've tried so far:
throttle the requests on the client during the bulk operation to less than 3/s
reduce the clients ConnectionPolicy.MaxConnectionLimit from default (50) to 20
increase the apps ServicePointManager.DefaultConnectionLimit
None of these measures significantly reduced the number of exceptions we're experiencing.
Any suggestions how to avoid this error in the first place?
Additional Cosmos DB SDK functionality to tune/configure/adapt for our use case..?

How do I set timeout for TIdHTTPProxyServer (not connection timout)

I am using TIdHTTPProxyServer and now I want to terminate connection when it is success to connect to the target HTTP server but receive no response for a long time(i.g. 3 mins)
Currently I find no related property or event about it. And even if the client terminate the connection before the proxy server receive the response from the HTTP server. OnException Event will not be fired until the proxy server receive the response. (That is, if the proxy server still receive no response from HTTP Server, I even do not know the client has already terminate the connection...)
Any help will be appreciated.
Thanks!
Willy
Indy uses infinite timeouts by default. To do what you are asking for, you need to set the ReadTimeout property of the outbound connection to the target server. You can access that connection via the TIdHTTPProxyServerContext.OutboundClient property. Use the OnHTTPBeforeCommand event, which is triggered just before the OutboundClient connects to the target server, eg:
#include "IdTCPClient.hpp"
void __fastcall TForm1::IdHTTPProxyServer1HTTPBeforeCommand(TIdHTTPProxyServerContext *AContext)
{
static_cast<TIdTCPClient*>(AContext->OutboundClient)->ReadTimeout = ...;
}

SmtpException from WebException: Unable to connect to the remote server

I am trying to send mail using smtp server but i am getting following Exception.
Please give me some advice..
System.Net.Mail.SmtpException: Failure sending mail.
---> System.Net.WebException: Unable to connect to the remote server
---> System.Net.Sockets.SocketException: A socket operation was attempted to an unreachable network 184.xx.xxx.xx:25
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket 6, Int32 timeout)
at System.Net.PooledStream.Activate(Object owningObject, Boolean async, Int32 timeout, GeneralAsyncDelegate asyncCallback)
at System.Net.PooledStream.Activate(Object owningObject, GeneralAsyncDelegate asyncCallback)
at System.Net.ConnectionPool.GetConnection(Object owningObject, GeneralAsyncD elegate asyncCallback, Int32 creationTimeout)
at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port)
at System.Net.Mail.SmtpTransport.GetConnection(String host, Int32 port)
at System.Net.Mail.SmtpClient.GetConnection()
at System.Net.Mail.SmtpClient.Send(MailMessage message)
--- End of inner exception stack trace ---
at System.Net.Mail.SmtpClient.Send(MailMessage message) at testEmail.Module1.Main() in E:\testEmail\testEmail\Module1.vb:line 17
From this messagge
A socket operation was attempted to an unreachable network 184.xx.xxx.xx:25
it seems that you cannot reach the SMTP address.
Check your internet connection and then check if the address is correct.
A Ping to the address or a tracerout can help you understand why you cannot contact the server, if the address is correct
The message tells me that your webserver has problems connectiong to your SMTP server. Are you sure the SMTP address is right?
Remember that some SMTP-servers need logon
Also some SMTP servers will not let you connect if your use a different IPS
I would change the SMTP to the one your ISP has.
You can try to connect to the SMTP with Telnet to check your connection
telnet YourSmtpAddress 25