Istio 1.14.5, sidecar injection fails due invalid certificate - istio

I'm using Istio 1.14.5 in my AWS EKS (1.21.14-eks-fb459a0).
I'm injecting my istio sidecar with:
istioctl kube-inject -f my-deployment.yaml | kubectl apply -f -
But, when I scaleout my deployment to 1 (create ReplicaSet), I'm getting this event in the ReplicaSet:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 72s (x17 over 6m40s) replicaset-controller Error creating: Internal error occurred: failed calling webhook "object.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": Address is not allowed
I'm already injecting the sidecar in another namespaces withou problems.
Using anothe POD in that namespace, accessing the shell I tried to perform a request to that URL with curl -v:
$ curl https://istiod.istio-system.svc:443 -v
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55de240c1f50)
* Connected to istiod.istio-system.svc (10.100.100.235) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
How can I solve this? What is the source of this problem?

Related

Connecting to AWS CloudFront: Secure Connection Failed

Firefox https://static.allfile.net/lib/main.js :
Secure Connection Failed
An error occurred during a connection to static.allfile.net. Cannot communicate securely with peer: no common encryption algorithm(s).
Error code: SSL_ERROR_NO_CYPHER_OVERLAP
$ curl -v https://static.allfile.net/lib/main.js
* Trying 13.226.2.2:443...
* Connected to static.allfile.net (13.226.2.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:0A00010B:SSL routines::wrong version number
* Closing connection 0
curl: (35) error:0A00010B:SSL routines::wrong version number
Why? It worked yesterday.
I have TLSv1.2_2021 (recommended) in CloudFront settings. I also have:
[x] HTTP/2
[ ] HTTP/3
"Legacy clients support - $600/month prorated charge applies. Most customers do not need this." is off.

TLS Connection Issue with AWS Global Accelerator

I have a working Load-Balancer (Sydney Australia) / Global Accelerator setup, however, I am having a bit of trouble with clients in the Central Americas region (specifically El Salvador and Costa-Rica) utilising this through the AWS Global Accelerator.
For myself (Sydney Australia) and other users in the US, there are no issues connecting through the Global Accelerator.
The clients that I am working with are pretty green and not comfortable performing a lot of command line scripts. From what I have been able to test with them, it appears that something is getting dropped on the TLS handshake.
Here's a non-working Global Accelerator curl
curl -Ikv https://GAIP1
* Rebuilt URL to: https://GAIP1/
* Trying GAIP1...
* TCP_NODELAY set
* Connected to GAIP1 (GAIP1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP1:443
* stopped the pause stream!
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP1:443
curl -Ikv https://GAIP2
* Rebuilt URL to: https://GAIP2/
* Trying GAIP2...
* TCP_NODELAY set
* Connected to GAIP2 (GAIP2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP2:443
* stopped the pause stream!
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to GAIP2:443
Here's an working load balancer curl
curl -Ikv https://loadbalancer.dns.name
* Rebuilt URL to: https://loadbalancer.dns.name/
* Trying ALBIP1...
* TCP_NODELAY set
* Connected to loadbalancer.dns.name (ALBIP1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=valid.domain
* start date: Apr 9 00:00:00 2020 GMT
* expire date: May 9 12:00:00 2021 GMT
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbdd4004800)
> HEAD / HTTP/2
> Host: loadbalancer.dns.name
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 302
HTTP/2 302
< date: Tue, 12 May 2020 23:36:34 GMT
date: Tue, 12 May 2020 23:36:34 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 200
content-length: 200
< location: http://loadbalancer.dns.name/Error?aspxerrorpath=/
location: http://loadbalancer.dns.name/Error?aspxerrorpath=/
< set-cookie: AWSALB=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/
set-cookie: AWSALB=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/
< set-cookie: AWSALBCORS=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/; SameSite=None; Secure
set-cookie: AWSALBCORS=####; Expires=Tue, 19 May 2020 23:36:34 GMT; Path=/; SameSite=None; Secure
< server: Microsoft-IIS/8.0
server: Microsoft-IIS/8.0
< x-powered-by: ASP.NET
x-powered-by: ASP.NET
< p3p: CP="NO COMPACT POLICY DEFINED"
p3p: CP="NO COMPACT POLICY DEFINED"
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
<
* Connection #0 to host loadbalancer.dns.name left intact
I've tried working with the AWS support, however, they have asked me to run some utils my client does not have / have indicated they are out of their conform zone for running any further commands, so I'm now at a bit of a loss as to what the problem COULD be, and how I am supposed to proceed.
Here's what AWS Support want the Client to run
1) curl outputs with HTTP and HTTPs while running tcp packets
capture at the same time. It's very important to run the tcp packets
capture first and then perform the tests:
curl -Ikv http://GAIP1
curl -Ikv http://GAIP2
curl -Ikv https://GAIP1
curl -Ikv https://GAIP2
To capture packets, run this command:
sudo tcpdump -n -vvv -s 65535 -i any -w GA.pcap
2) Can you provide the outputs of these commands:
hping3 -S -c 50 -p 443 -V GAIP1
hping3 -S -c 50 -p 443 -V GAIP2
Thanks for any help in advance
--Edit
Add Guatemala to the list of affected countries
List Of Affected Countries / Regions
El Salvador
Costa Rica
Guatemala
Map for regional reference
I finally have an answer to this. I'm still looking for some additional clarification from AWS as to why their configuration is causing this. Will update if i hear back.
I had some custom settings on the Network ACL for my subnets on the Application Load Balancers. Removing the settings from them (making them the default permissive allow all) resolved the issue.
The issue itself was to do the the maximum MTU of the hops from the affected regions. From those regions their maximum MTU size was below 1420 which is the default size for a TLS handshake (at least on my computer).
Unfortunately the custom ACL settings on the subnets was causing the ICMP fragmentation communications to be dropped and so any packets larger than 1391 bytes would just be silently dropped.
Removing the custom rules on the ACL for the subnets of the load balancer, allows packets to fragment properly.

"curl: (52) Empty reply from server" on Windows only

We are able to publish data to AWS IoT Core using cURL from a Linux desktop, yet when we try from Windows desktops we receive the response "curl: (52) Empty reply from server". We are convinced the issue is with the default Windows configuration yet the error seems to suggest a server-side issue?
I have seen somewhere that it may be due to WinSSL, and we should try switching to OpenSSL, though I cannot find any information on how this is achieved.
As part of our debugging process:
We are not behind a proxy
Host firewall was disabled as a test
There is no network firewall (home workers)
Version of cURL on Windows (not working):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
Version of cURL on Linux (working):
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
Verbose output of cURL on Windows (resulting in empty reply):
C:\dev\agent\aws-test>curl --tlsv1.2 --cacert root-CA.pem --cert aaaaaaaaa-certificate.pem --key aaaaaaaaa-private.pem -X POST -d "{ \"message\": \"Hello, world\" }" "https://aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com:8443/topics/test/1" --verbose
Note: Unnecessary use of -X or --request, POST is already inferred.
5. Trying 52.208.99.5...
6. TCP_NODELAY set
7. Connected to aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com (52.208.99.5) port 8443 (#0)
8. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 1/3)
9. schannel: checking server certificate revocation
10. schannel: sending initial handshake data: sending 211 bytes...
11. schannel: sent initial handshake data: sent 211 bytes
12. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 2/3)
13. schannel: failed to receive handshake, need more data
14. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 2/3)
15. schannel: encrypted data got 4096
16. schannel: encrypted data buffer: offset 4096 length 4096
17. schannel: received incomplete message, need more data
18. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 2/3)
19. schannel: encrypted data got 1024
20. schannel: encrypted data buffer: offset 5120 length 5120
21. schannel: received incomplete message, need more data
22. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 2/3)
23. schannel: encrypted data got 229
24. schannel: encrypted data buffer: offset 5349 length 6144
25. schannel: sending next handshake data: sending 1414 bytes...
26. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 2/3)
27. schannel: encrypted data got 51
28. schannel: encrypted data buffer: offset 51 length 6144
29. schannel: SSL/TLS handshake complete
30. schannel: SSL/TLS connection with aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com port 8443 (step 3/3)
31. schannel: stored credential handle in session cache
> POST /topics/testmmcdonald/1 HTTP/1.1
> Host: aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com:8443
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
>
32. upload completely sent off: 29 out of 29 bytes
33. schannel: client wants to read 102400 bytes
34. schannel: encdata_buffer resized 103424
35. schannel: encrypted data buffer: offset 0 length 103424
36. schannel: encrypted data got 31
37. schannel: encrypted data buffer: offset 31 length 103424
38. schannel: server closed the connection
39. schannel: schannel_recv cleanup
40. Empty reply from server
41. Connection #0 to host aaaaaaaaaaaaaa-ats.iot.eu-west-1.amazonaws.com left intact
curl: (52) Empty reply from server
Verbose output of functioning cURL on Linux desktop:
curl --tlsv1.2 --cacert root-CA.crt --cert aaaaaaaaa-certificate.pem.crt --key aaaaaaaaa-private.pem.key -X POST -d "{ \"message\": \"Now?\" }" "https://aaaaaaaaa-ats.iot.eu-west-1.amazonaws.com:8443/topics/test/1" --verbose
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 52.213.138.243...
* TCP_NODELAY set
* Connected to aaaaaaaaa-ats.iot.eu-west-1.amazonaws.com (52.213.138.243) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: root-CA.crt
CApath: /etc/ssl/certs
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.iot.eu-west-1.amazonaws.com
* start date: Jul 2 00:00:00 2019 GMT
* expire date: Jun 28 12:00:00 2020 GMT
* subjectAltName: host "aaaaaaaaa-ats.iot.eu-west-1.amazonaws.com" matched cert's "*.iot.eu-west-1.amazonaws.com"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
> POST /topics/test/1 HTTP/1.1
> Host: aaaaaaaaa-ats.iot.eu-west-1.amazonaws.com:8443
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 21
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 21 out of 21 bytes
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 65
< date: Tue, 27 Aug 2019 14:39:59 GMT
< x-amzn-RequestId: 903a3d180147
< connection: keep-alive
<
* Connection #0 to host aaaaaaaaa-ats.iot.eu-west-1.amazonaws.com left intact
{"message":"OK","traceId":"903a3d180147"}
Huh.... it's working. I just downloaded curl.exe from https://curl.haxx.se/windows/ instead of using the one pre-built in to Windows.

Kubernetes ingress on GKE results in 502 response on http / SSL_ERROR_SYSCALL on https

I've tested my configuration on minikube where it works perfectly, however on GKE I run into an error of HTTP responding with 502 while the HTTPS gets the connection terminated?
I have no idea how to diagnose this issue, which logs could I look at?
Here is a verbose curl log when accessing over https://
* Expire in 0 ms for 1 (transfer 0x1deb470)
* Expire in 0 ms for 1 (transfer 0x1deb470)
* Expire in 0 ms for 1 (transfer 0x1deb470)
* Trying 35.244.154.110...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x1deb470)
* Connected to chrischrisexample.de (35.244.154.110) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to chrischrisexample.de:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to chrischrisexample.de:443
To solve it I had to:
Respond with a HTTP 200 on the health check (from the Google load balancer!)
Set a SSL certificate secret in the ingress (even if a self signed one)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Sync 14m (x20 over 157m) loadbalancer-controller Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce
Warning Translate 3m56s (x18 over 9m24s) loadbalancer-controller error while evaluating the ingress spec: could not find port "80" in service "default/app"; could not find port "80" in service "default/app"; could not find port "80" in service "default/app"; could not find port "80" in service "default/app"
These errors were shown on the kubectl describe ingress... Still doesn't make sense why it would error on the SSL handshake / connection though.

DropBox upload using files_put returns authentication failed

I'm trying to upload a file (simple text file with one line of text in it) using with my C/C++ application using libCurl.
There are a few things that I have noticed.
First...
The DropBox API appears to want a '&' instead of a '?' before the list of arguments.
Second...
It is unclear if the body of the request needs to be included in computing the oauth_signature.
Ultimately, I get an "Authentication failed" response from the DropBox API.
I've included the debug trace from libCurl for more information...
SSLv3, TLS alert, Client hello (1):
About to connect() to api-content.dropbox.com port 443 (#0)
Trying 107.22.243.22... Connected to api-content.dropbox.com (107.22.243.22) port 443 (#0)
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server key exchange (12):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using DHE-RSA-AES256-SHA
Server certificate:
subject: C=US; ST=California; L=San Francisco; O=Dropbox, Inc.; OU=IT; CN=*.dropbox.com
start date: 2010-01-06 00:00:00 GMT
expire date: 2012-01-06 23:59:59 GMT
common name: *.dropbox.com (matched)
issuer: C=ZA; ST=Western Cape; L=Cape Town; O=Thawte Consulting cc; OU=Certification Services Division; CN=Thawte Premium Server CA; emailAddress=premium-server#thawte.com
SSL certificate verify ok.
PUT /1/files_put/dropbox/1&file=test%2Etxt&overwrite=true&oauth_consumer_key=asfewasdfas&oauth_nonce=1323293220d0&oauth_signature=asmoa4YE2c%2FuwjDKJRKFILpcn8%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1323293220&oauth_token=fafsesnj13iguxnh&oauth_version=1.0 HTTP/1.1
Host: api-content.dropbox.com
Accept: */*
Content-type: application/json
Content-Length: 24
Expect: 100-continue
HTTP/1.1 100 Continue
Test upload to DropBox.
HTTP/1.1 401 Unauthorized
Server: dbws
Date: Wed, 07 Dec 2011 21:27:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
22
{"error": "Authentication failed"}
0
Try using the OAuth "PLAINTEXT" authentication mode using the HTTP "Authorization" header.
PLAINTEXT OAuth is much less error prone (since there's no canonicalizattion or hashing).