TLS Connection Issue with AWS Global Accelerator - amazon-web-services

Question

I have a working Load-Balancer (Sydney Australia) / Global Accelerator setup, however, I am having a bit of trouble with clients in the Central Americas region (specifically El Salvador and Costa-Rica) utilising this through the AWS Global Accelerator.
For myself (Sydney Australia) and other users in the US, there are no issues connecting through the Global Accelerator.
The clients that I am working with are pretty green and not comfortable performing a lot of command line scripts. From what I have been able to test with them, it appears that something is getting dropped on the TLS handshake.
Here's a non-working Global Accelerator curl
curl -Ikv https://GAIP1
* Rebuilt URL to: https://GAIP1/
* Trying GAIP1...
* TCP_NODELAY set
* Connected to GAIP1 (GAIP1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP1:443
* stopped the pause stream!
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP1:443
curl -Ikv https://GAIP2
* Rebuilt URL to: https://GAIP2/
* Trying GAIP2...
* TCP_NODELAY set
* Connected to GAIP2 (GAIP2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP2:443
* stopped the pause stream!
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP2:443
Here's an working load balancer curl
curl -Ikv https://loadbalancer.dns.name
* Rebuilt URL to: https://loadbalancer.dns.name/
* Trying ALBIP1...
* TCP_NODELAY set
* Connected to loadbalancer.dns.name (ALBIP1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=valid.domain
* start date: Apr 9 00:00:00 2020 GMT
* expire date: May 9 12:00:00 2021 GMT
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbdd4004800)
> HEAD / HTTP/2
> Host: loadbalancer.dns.name
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 302
HTTP/2 302
< date: Tue, 12 May 2020 23:36:34 GMT
date: Tue, 12 May 2020 23:36:34 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 200
content-length: 200
< location: http://loadbalancer.dns.name/Error?aspxerrorpath=/
location: http://loadbalancer.dns.name/Error?aspxerrorpath=/
< set-cookie: AWSALB=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/
set-cookie: AWSALB=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/
< set-cookie: AWSALBCORS=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/; SameSite=None; Secure
set-cookie: AWSALBCORS=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/; SameSite=None; Secure
< server: Microsoft-IIS/8.0
server: Microsoft-IIS/8.0
< x-powered-by: ASP.NET
x-powered-by: ASP.NET
< p3p: CP="NO COMPACT POLICY DEFINED"
p3p: CP="NO COMPACT POLICY DEFINED"
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
<
* Connection #0 to host loadbalancer.dns.name left intact
I've tried working with the AWS support, however, they have asked me to run some utils my client does not have / have indicated they are out of their conform zone for running any further commands, so I'm now at a bit of a loss as to what the problem COULD be, and how I am supposed to proceed.
Here's what AWS Support want the Client to run
1) curl outputs with HTTP and HTTPs while running tcp packets
capture at the same time. It's very important to run the tcp packets
capture first and then perform the tests:
curl -Ikv http://GAIP1
curl -Ikv http://GAIP2
curl -Ikv https://GAIP1
curl -Ikv https://GAIP2
To capture packets, run this command:
sudo tcpdump -n -vvv -s 65535 -i any -w GA.pcap
2) Can you provide the outputs of these commands:
hping3 -S -c 50 -p 443 -V GAIP1
hping3 -S -c 50 -p 443 -V GAIP2
Thanks for any help in advance
--Edit
Add Guatemala to the list of affected countries
List Of Affected Countries / Regions
El Salvador
Costa Rica
Guatemala
Map for regional reference

Answer 1

I finally have an answer to this. I'm still looking for some additional clarification from AWS as to why their configuration is causing this. Will update if i hear back.
I had some custom settings on the Network ACL for my subnets on the Application Load Balancers. Removing the settings from them (making them the default permissive allow all) resolved the issue.
The issue itself was to do the the maximum MTU of the hops from the affected regions. From those regions their maximum MTU size was below 1420 which is the default size for a TLS handshake (at least on my computer).
Unfortunately the custom ACL settings on the subnets was causing the ICMP fragmentation communications to be dropped and so any packets larger than 1391 bytes would just be silently dropped.
Removing the custom rules on the ACL for the subnets of the load balancer, allows packets to fragment properly.