awk input file has space space in quotes - regex

Good day.
Trying to learn some AWK to convert some Juniper firewall configs to Cisco or Palo configs. Part of that is to parse the configuration. I have a sample here:
set service "RDP" protocol tcp src-port 0-65535 dst-port 3389-3389
set service "LDAPS" protocol tcp src-port 0-65535 dst-port 636-636
set service "SOAPS" protocol tcp src-port 0-65535 dst-port 444-444
set service "KEYS-ADMIN" protocol tcp src-port 0-65535 dst-port 9000-9000
set service "WSUS-MDM" protocol tcp src-port 0-65535 dst-port 8530-8530
set service "WSUS-MDM" + tcp src-port 0-65535 dst-port 8531-8531
set service "WSUS-MDM" + tcp src-port 0-65535 dst-port 8531-8531
set service "HTTPS-MDM" protocol tcp src-port 0-65535 dst-port 8443-8443
set service "IPSEC - 4500" protocol udp src-port 0-65535 dst-port 4500-4500
set service "IPSEC - 4500" + tcp src-port 0-65535 dst-port 1433-1433
set service "IPSEC - 4500" + tcp src-port 0-65535 dst-port 1433-1433
set service "OKFTP" protocol tcp src-port 0-65535 dst-port 2169-2169
set service "Bomgar 8200" protocol tcp src-port 0-65535 dst-port 8200-8200
set service "Cisco VPN" protocol tcp src-port 0-65535 dst-port 10000-10000
set service "Cisco VPN 2" protocol tcp src-port 0-65535 dst-port 10000-10000
set service "Cisco VPN 2" + udp src-port 0-65535 dst-port 10000-10000
set service "Cisco VPN 2" + udp src-port 0-65535 dst-port 500-500
set service "Cisco VPN 2" + udp src-port 0-65535 dst-port 4500-4500
set service "Cisco VPN 2" + 50 src-port 0-65535 dst-port 0-65535
set service "Cisco VPN 2" + udp src-port 0-65535 dst-port 10000-10000
set service "Cisco VPN 2" + udp src-port 0-65535 dst-port 500-500
set service "Cisco VPN 2" + udp src-port 0-65535 dst-port 4500-4500
set service "TrendMicro8080" protocol tcp src-port 0-65535 dst-port 8080-8080
set service "TrendMicro26980" protocol tcp src-port 0-65535 dst-port 26980-26980
set service "TrendMicro26980" + udp src-port 0-65535 dst-port 26980-26980
set service "PenPal Test" protocol tcp src-port 0-65535 dst-port 522-522
set service "HTTP8080" protocol tcp src-port 0-65535 dst-port 8080-8080
set service "HTTPS445" protocol tcp src-port 0-65535 dst-port 445-445
set service "MOBILEIRON-TLS" protocol tcp src-port 0-65535 dst-port 9997-9997
set service "MOBILEIRON-TLS" + tcp src-port 0-65535 dst-port 9998-9998
I saved this snippet of lines to a file named test1 and ran this command from the command line:
awk -F " " 'BEGIN {OFS=","} {print $3,$5,$7,$9}' test1
Although it MOSTLY worked out, the spaces contained inside the " " where seen by awk as valid spaces. The output:
"RDP",tcp,0-65535,3389-3389
"LDAPS",tcp,0-65535,636-636
"SOAPS",tcp,0-65535,444-444
"KEYS-ADMIN",tcp,0-65535,9000-9000
"WSUS-MDM",tcp,0-65535,8530-8530
"WSUS-MDM",tcp,0-65535,8531-8531
"WSUS-MDM",tcp,0-65535,8531-8531
"HTTPS-MDM",tcp,0-65535,8443-8443
"IPSEC,4500",udp,0-65535
"IPSEC,4500",tcp,0-65535
"IPSEC,4500",tcp,0-65535
"OKFTP",tcp,0-65535,2169-2169
"Bomgar,protocol,src-port,dst-port
"Cisco,protocol,src-port,dst-port
"Cisco,2",tcp,0-65535
"Cisco,2",udp,0-65535
"Cisco,2",udp,0-65535
"Cisco,2",udp,0-65535
"Cisco,2",50,0-65535
"Cisco,2",udp,0-65535
"Cisco,2",udp,0-65535
"Cisco,2",udp,0-65535
Ideally, I would like to have awk ignore the spaces in the " ". I guess I could add it as a regular expression? Do I use the '!' somehow? Not sure. Any help would be appreciated.

There are likely many ways to achieve your end result (maybe even one that is awk inclusive):
awk -F\" 'BEGIN {OFS=","} {split($3,F," ");print $2,F[2],F[4],F[6]}' test1
Another way that is possible is to use sed:
sed 's/\("[^"]*"\)* \("[^"]*"\)*/\1,\2/g' test1
...or piped to awk:
sed 's/\("[^"]*"\)* \("[^"]*"\)*/\1,\2/g' test1 | awk -F ',' 'BEGIN {OFS=","} {print $3,$5,$7,$9}'
Output:
"RDP",tcp,0-65535,3389-3389
"LDAPS",tcp,0-65535,636-636
"SOAPS",tcp,0-65535,444-444
"KEYS-ADMIN",tcp,0-65535,9000-9000
"WSUS-MDM",tcp,0-65535,8530-8530
"WSUS-MDM",tcp,0-65535,8531-8531
"WSUS-MDM",tcp,0-65535,8531-8531
"HTTPS-MDM",tcp,0-65535,8443-8443
"IPSEC - 4500",udp,0-65535,4500-4500
"IPSEC - 4500",tcp,0-65535,1433-1433
"IPSEC - 4500",tcp,0-65535,1433-1433
"OKFTP",tcp,0-65535,2169-2169
"Bomgar 8200",tcp,0-65535,8200-8200
"Cisco VPN",tcp,0-65535,10000-10000
"Cisco VPN 2",tcp,0-65535,10000-10000
"Cisco VPN 2",udp,0-65535,10000-10000
"Cisco VPN 2",udp,0-65535,500-500
...
The awk solution was discovered after learning of this excellent example.

Related

How to connect to an EKS service from outside the cluster using a LoadBalancer in a private VPC

I am trying to expose An EKS deployment of Kafka outside the cluster, within the same VPC.
In terraform I added an ingress rule for the Kafka security group:
ingress {
from_port = 9092
protocol = "tcp"
to_port = 9092
cidr_blocks = [
"10.0.0.0/16",
]
}
This is the service yaml
apiVersion: v1
kind: Service
metadata:
name: bootstrap-external
namespace: kafka
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "10.0.0.0/16"
service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-0....d,sg-0db....ae"
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 9092
targetPort: 9092
selector:
app: kafka
When trying to connect from another instance, belonging to one of the security groups in the yaml,
I seem to be able to establish a connection through the load balancer but not get referred to Kafka:
[ec2-user#ip-10-0-4-47 kafkacat]$ nc -zvw10 internal-a08....628f-1654182718.us-east-2.elb.amazonaws.com 9092
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.3.151:9092.
Ncat: 0 bytes sent, 0 bytes received in 0.05 seconds.
[ec2-user#ip-10-0-4-47 kafkacat]$ nmap -Pn internal-a0837....a0e628f-1654182718.us-east-2.elb.amazonaws.com -p 9092
Starting Nmap 6.40 ( http://nmap.org ) at 2021-02-28 07:19 UTC
Nmap scan report for internal-a083747ab.....8f-1654182718.us-east-2.elb.amazonaws.com (10.0.2.41)
Host is up (0.00088s latency).
Other addresses for internal-a083747ab....36f0a0e628f-1654182718.us-east-2.elb.amazonaws.com (not scanned): 10.0.3.151 10.0.1.85
rDNS record for 10.0.2.41: ip-10-0-2-41.us-east-2.compute.internal
PORT STATE SERVICE
9092/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds
[ec2-user#ip-10-0-4-47 kafkacat]$ kafkacat -b internal-a083747abf4....-1654182718.us-east-2.elb.amazonaws.com:9092 -t models
% Auto-selecting Consumer mode (use -P or -C to override)
% ERROR: Local: Host resolution failure: kafka-2.broker.kafka.svc.cluster.local:9092/2: Failed to resolve 'kafka-2.broker.kafka.svc.cluster.local:9092': Name or service not known
% ERROR: Local: Host resolution failure: kafka-1.broker.kafka.svc.cluster.local:9092/1: Failed to resolve 'kafka-1.broker.kafka.svc.cluster.local:9092': Name or service not known
% ERROR: Local: Host resolution failure: kafka-0.broker.kafka.svc.cluster.local:9092/0: Failed to resolve 'kafka-0.broker.kafka.svc.cluster.local:9092': Name or service not known
^C[ec2-user#ip-10-0-4-47 kafkacat]$
``
We solved the Kafka connection by:
Adding ingress rule to the Kafka worker security group (We use Terraform)
ingress {
from_port = 9094
protocol = "tcp"
to_port = 9094
cidr_blocks = [
"10.0.0.0/16",
]
}
Provisioning each broker a load balancer service in Kubernetes YAML (note that the last digit in the nodePort corresponds to the broker stateful set ID).
apiVersion: v1
kind: Service
metadata:
name: bootstrap-external-0
namespace: kafka
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "10.0.0.0/16"
service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: sg-....d,sg-0db14....e,sg-001ce.....e,sg-0fe....15d13c
spec:
type: LoadBalancer
ports:
-
protocol: TCP
targetPort: 9094
port: 32400
nodePort: 32400
selector:
app: kafka
kafka-broker-id: "0"
Retrieving load balancer name by parsing kubctl -n kafka get svc bootstrap-external-0.
Adding DNS name by convention using Route 53.
We plan to automate by terraforming the Route53 after load balancer is created.

TLS origination from sidecar proxy instead of the Egress Gateway

I am trying to setup mTLS for outgoing connections, but instead of originating the TLS traffic from the egress gateway, I’m trying to do it from the sidecar proxy itself. We want to originate the TLS connection from proxy and not the egress gateway.
I took care of mounting the client certs in my sidecar proxy container and verified that the client certs are available in the expected path. My API resources look something like below
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: external-host-mtls
spec:
hosts:
- external-host-example.com
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: external-mtls
spec:
hosts:
- external-host-example.com
tls:
- match:
- port: 443
sniHosts:
- external-host-example.com
route:
- destination:
host: external-host-example.com
port:
number: 443
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: external-mtls
spec:
host: external-host-example.com
trafficPolicy:
tls:
mode: MUTUAL
clientCertificate: /etc/client-certs/client.pem
privateKey: /etc/client-certs/client.key
caCertificates: /etc/client-certs/ca.pem
when I try to curl to external-host-example.com, I am hoping that Istio will add the client certs to the connection.
I’m not sure if that’s happening, because I’m running into errors.
curl -H "Host: external-host-example.com" --tlsv1.2 -v https://external-host-example.com
* About to connect() to external-host-example.com port 443 (#0)
* Trying x.x.x.x...
* Connected to external-host-example.com (x.x.x.x) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file
Looking at the debug logs, I see this
|2020-11-17T15:53:58.226367Z|debug|envoy filter|[external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:148] tls:onServerName(), requestedServerName: external-host-example.com|
|2020-11-17T15:53:58.226443Z|debug|envoy filter|[external/envoy/source/common/tcp_proxy/tcp_proxy.cc:251] [C161] new tcp proxy session|
|2020-11-17T15:53:58.226480Z|debug|envoy filter|[external/envoy/source/common/tcp_proxy/tcp_proxy.cc:395] [C161] Creating connection to cluster outbound|443||external-host-example.com|
|2020-11-17T15:53:58.226509Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:83] creating a new connection|
|2020-11-17T15:53:58.226550Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:364] [C162] connecting|
|2020-11-17T15:53:58.226557Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:727] [C162] connecting to x.x.x.x:443|
|2020-11-17T15:53:58.226641Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:736] [C162] connection in progress|
|2020-11-17T15:53:58.226656Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:109] queueing request due to no available connections|
|2020-11-17T15:53:58.226662Z|debug|envoy conn_handler|[external/envoy/source/server/connection_handler_impl.cc:411] [C161] new connection|
|2020-11-17T15:53:58.252446Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:592] [C162] connected|
|2020-11-17T15:53:58.252555Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C162] handshake expecting read|
|2020-11-17T15:53:58.277388Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C162] handshake expecting read|
|2020-11-17T15:53:58.277417Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C162] handshake expecting read|
|2020-11-17T15:53:58.277595Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:176] [C162] handshake complete|
|2020-11-17T15:53:58.277633Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:285] [C162] assigning connection|
|2020-11-17T15:53:58.277661Z|debug|envoy filter|[external/envoy/source/common/tcp_proxy/tcp_proxy.cc:624] TCP:onUpstreamEvent(), requestedServerName:external-host-example.com|
|2020-11-17T15:53:58.303804Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C162]|
|2020-11-17T15:53:58.303830Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:558] [C162] remote close|
|2020-11-17T15:53:58.303834Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:200] [C162] closing socket: 0|
|2020-11-17T15:53:58.303853Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:298] [C162] SSL shutdown: rc=-1|
|2020-11-17T15:53:58.303855Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C162]|
|2020-11-17T15:53:58.303880Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:124] [C162] client disconnected|
|2020-11-17T15:53:58.303894Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:109] [C161] closing data_to_write=0 type=0|
|2020-11-17T15:53:58.303900Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:200] [C161] closing socket: 1|
|2020-11-17T15:53:58.303985Z|debug|envoy conn_handler|[external/envoy/source/server/connection_handler_impl.cc:111] [C161] adding to cleanup list|
Any idea what am I doing wrong? How do I debug this further?

Stunnel for Elasticchache Redis(cluster mode enabled)

I have spin up Elasticcache Redis cluster mode enabled cluster on AWS. I am having 3 master shards and 1 replica for each(total 3 replicas). I have turn on in-transit encryption. For this I have installed stunnel on my EC2 instance and my config file looks like. 3001 is my cluster port
[redis-cli]
client = yes
accept = 127.0.0.1:3001
connect = master1_url:3001
[redis-cli1]
client = yes
accept = 127.0.0.1:3002
connect = replica1_url.com:3001
[redis-cli2]
client = yes
accept = 127.0.0.1:3003
connect = master2_url:3001
[redis-cli3]
client = yes
accept = 127.0.0.1:3004
connect = replica2_url.com:3001
[redis-cli4]
client = yes
accept = 127.0.0.1:3005
connect = master3_url:3001
[redis-cli3]
client = yes
accept = 127.0.0.1:3006
connect = replica3_url.com:3001
----------------------------------------------
sudo netstat -tulnp | grep -i stunnel
tcp 0 0 127.0.0.1:3001 0.0.0.0:* LISTEN 32272/stunnel
tcp 0 0 127.0.0.1:3002 0.0.0.0:* LISTEN 32272/stunnel
tcp 0 0 127.0.0.1:3003 0.0.0.0:* LISTEN 32272/stunnel
tcp 0 0 127.0.0.1:3004 0.0.0.0:* LISTEN 32272/stunnel
tcp 0 0 127.0.0.1:3005 0.0.0.0:* LISTEN 32272/stunnel
tcp 0 0 127.0.0.1:3006 0.0.0.0:* LISTEN 32272/stunnel
When I connect using localhost (src/redis-cli -c -h localhost -p 3001)my connection is successful. But when I hit "get key" it stuck with following
localhost:3001> get key
-> Redirected to slot [12539] located at master3_url:3001
If I change the cluster to single shard and single replica all works fine. What setting I am missing when using multi shards cluster. My Ec2 instance is open to accept connection on all ports. Redis Cluster is open to accept connection from Ec2 instance on all ports.
This is my first question on stackoverflow :)

nginx-ingress doesn't work with AWS ELB when use aws-load-balancer-backend-protocol: "https"

I use AWS EKS as my k8s control plane, and deployed a 3-node autoscaling group as my work nodes(K8s Nodes.) This autoscaling group sits in my VPC. And I made sure security groups are open at least permissive enough for peer node and ELB to communicate.
I am trying to use nginx-ingress for routing traffic from outside k8s cluster. I use helm to deploy my nginx-ingress using a values.yaml.
My values.yaml looks like this:
serviceAccount:
create: true
name: nginx-ingress-sa
rbac:
create: true
controller:
kind: "Deployment"
service:
type: "LoadBalancer"
# targetPorts:
# http: 80
# https: http
loadBalancerSourceRanges:
- 1.2.3.4/32
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:123456789:certificate/my-cert
service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: sg-12345678
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/enable-access-log: "true"
config:
log-format-escape-json: "true"
log-format-upstream: '{"real_ip" : "$the_real_ip", "remote_user": "$remote_user", "time_iso8601": "$time_iso8601", "request": "$request", "request_method" : "$request_method", "status": "$status", "upstream_addr": $upstream_addr", "upstream_status": "$upstream_status"}'
extraArgs:
v: 3 # NGINX log level
My ingress yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress-1
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/enable-access-log: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
rules:
- host: "s1.testk8.dev.mydomain.net"
http:
paths:
- path: /
backend:
serviceName: s1-service
servicePort: 443
- host: "s2.testk8.dev.mydomain.net"
http:
paths:
- path: /
backend:
serviceName: s2-service
servicePort: 443
tls:
- hosts:
- "s1.testk8.dev.mydomain.net"
- "s2.testk8.dev.mydomain.net"
secretName: "testk8.dev.mydomain.net"
Note that this secret is a self-signed TLS cert on the domain *.mydomain.net.
The behavior right now with this setting is that if enter
https://s1.testk8.dev.mydomain.net in Chrome, it just hangs. It says waiting for s1.testk8.dev.mydomain.net on the lower left corner.
If I use:
curl -vk https://s1.testk8.dev.mydomain.net
It returns:
* Trying x.x.x.x...
* TCP_NODELAY set
* Connected to s1.testk8.dev.mydomain.net (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.mydomain.net
* start date: Apr 25 00:00:00 2018 GMT
* expire date: May 25 12:00:00 2019 GMT
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: s1.testk8.dev.steelcentral.net
> User-Agent: curl/7.54.0
> Accept: */*
>
And it also appears to wait for server response.
I also tried to tweak the values.yaml, and when I change
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http" # instead of https as above
then I hit the https://s1.testk8.dev.mydomain.net URL, I can at least see the HTTP 400 message(plain HTTP request sent to HTTPS port) from the ingress controller pod.
If uncomment these lines in the values.yaml:
# targetPorts:
# http: 80
# https: http
I am able to reach to my backend pod(controlled by a statefulset, not listed here.), I can see access log of my backend pod having new entries.
Not sure whether my use case is weird here since I see lot of folks use nginx-ingress on AWS they terminate the TLS at ELB. But I need to let my backend pod to terminate the TLS.
I also tried the ssl-passthrough flag, didn't help. When the backend-protocal is https, my request doesn't even seem to reach the ingress controller, so talking ssl-passthrough might still be meanless.
Thank you in advance if you just read all the way through here!!
As far as I can tell, even with the current master of nginx-ingress, it is not possible to use self-signed certificates. The template https://github.com/kubernetes/ingress-nginx/blob/master/rootfs/etc/nginx/template/nginx.tmpl is missing any of the needed directives like:
location / {
proxy_pass https://backend.server.ip/;
proxy_ssl_trusted_certificate /etc/nginx/sslcerts/backend.server.pem;
proxy_ssl_verify off;
... other proxy settings
}
So try to use e.g. a let's encrypt certificate.
My guess is that your backend services are using HTTPS and the in-between traffic is being sent through HTTP. This line in your values.yaml seems odd:
targetPorts:
http: 80
https: http
Can you try something like this?
targetPorts:
http: 80
https: 443

connection issue with hazelcast on amazon AWS

I am using Hazelcast v3.6 on two amazon AWS virtual machines (not using the AWS specific settings for hazelcast). The connection is supposed to work via TCP/IP connection settings (not multicasting). I have opened 5701-5801 address for connection on the virtual machines.
I have tried using iperf on the two virtual machines using which I can see that the client on one VM connects to the server on another VM (and vice versa when I switch the client server setup for iperf).
When I launch two Hazelcast servers on different VM's, the connection is not established. The log statements and the hazelcast.xml config are given below (I am not using the programmatic settings for Hazelcast). I have changed the IP addresses below:
20160401-16:41:02.812 [cached2] InitConnectionTask INFO - [45.46.47.48]:5701 [dev] [3.6] Connecting to /22.23.24.25:5701, timeout: 0, bind-any: true
20160401-16:41:02.812 [cached3] InitConnectionTask INFO - [45.46.47.48]:5701 [dev] [3.6] Connecting to /22.23.24.25:5703, timeout: 0, bind-any: true
20160401-16:41:02.813 [cached1] InitConnectionTask INFO - [45.46.47.48]:5701 [dev] [3.6] Connecting to /22.23.24.25:5702, timeout: 0, bind-any: true
20160401-16:41:02.816 [cached1] InitConnectionTask INFO - [45.46.47.48]:5701 [dev] [3.6] Could not connect to: /22.23.24.25:5702. Reason: SocketException[Connection refused to address /22.23.24.25:570
2]
20160401-16:41:02.816 [cached1] TcpIpJoiner INFO - [45.46.47.48]:5701 [dev] [3.6] Address[22.23.24.25]:5702 is added to the blacklist.
20160401-16:41:02.817 [cached3] InitConnectionTask INFO - [45.46.47.48]:5701 [dev] [3.6] Could not connect to: /22.23.24.25:5703. Reason: SocketException[Connection refused to address /22.23.24.25:570
3]
20160401-16:41:02.817 [cached3] TcpIpJoiner INFO - [45.46.47.48]:5701 [dev] [3.6] Address[22.23.24.25]:5703 is added to the blacklist.
20160401-16:41:02.834 [cached2] TcpIpConnectionManager INFO - [45.46.47.48]:5701 [dev] [3.6] Established socket connection between /45.46.47.48:51965 and /22.23.24.25:5701
20160401-16:41:02.849 [hz._hzInstance_1_dev.IO.thread-in-0] TcpIpConnection INFO - [45.46.47.48]:5701 [dev] [3.6] Connection [Address[22.23.24.25]:5701] lost. Reason: java.io.EOFException[Remote socket
closed!]
20160401-16:41:02.851 [hz._hzInstance_1_dev.IO.thread-in-0] NonBlockingSocketReader WARN - [45.46.47.48]:5701 [dev] [3.6] hz._hzInstance_1_dev.IO.thread-in-0 Closing socket to endpoint Address[54.89.161.2
28]:5701, Cause:java.io.EOFException: Remote socket closed!
20160401-16:41:03.692 [cached2] InitConnectionTask INFO - [45.46.47.48]:5701 [dev] [3.6] Connecting to /22.23.24.25:5701, timeout: 0, bind-any: true
20160401-16:41:03.693 [cached2] TcpIpConnectionManager INFO - [45.46.47.48]:5701 [dev] [3.6] Established socket connection between /45.46.47.48:60733 and /22.23.24.25:5701
20160401-16:41:03.696 [hz._hzInstance_1_dev.IO.thread-in-1] TcpIpConnection INFO - [45.46.47.48]:5701 [dev] [3.6] Connection [Address[22.23.24.25]:5701] lost. Reason: java.io.EOFException[Remote socket
closed!]
Part of Hazelcast config
<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.6.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<group>
<name>abc</name>
<password>defg</password>
</group>
<network>
<port auto-increment="true" port-count="100">5701</port>
<outbound-ports>
<ports>0-5900</ports>
</outbound-ports>
<join>
<multicast enabled="false">
<!--<multicast-group>224.2.2.3</multicast-group>
<multicast-port>54327</multicast-port>-->
</multicast>
<tcp-ip enabled="true">
<member>22.23.24.25</member>
</tcp-ip>
</join>
<interfaces enabled="true">
<interface>45.46.47.48</interface>
</interfaces>
<ssl enabled="false" />
<socket-interceptor enabled="false" />
<symmetric-encryption enabled="false">
<algorithm>PBEWithMD5AndDES</algorithm>
<!-- salt value to use when generating the secret key -->
<salt>thesalt</salt>
<!-- pass phrase to use when generating the secret key -->
<password>thepass</password>
<!-- iteration count to use when generating the secret key -->
<iteration-count>19</iteration-count>
</symmetric-encryption>
</network>
<partition-group enabled="false"/>
iperf server and client log statements
Server listening on TCP port 5701
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 22.23.24.25, TCP port 5701
TCP window size: 1.33 MByte (default)
------------------------------------------------------------
[ 5] local 172.31.17.104 port 57398 connected with 22.23.24.25 port 5701
[ 4] local 172.31.17.104 port 5701 connected with 22.23.24.25 port 55589
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 662 MBytes 555 Mbits/sec
[ 4] 0.0-10.0 sec 797 MBytes 666 Mbits/sec
Server listening on TCP port 5701
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local xxx.xx.xxx.xx port 5701 connected with 22.23.24.25 port 57398
------------------------------------------------------------
Client connecting to 22.23.24.25, TCP port 5701
TCP window size: 1.62 MByte (default)
------------------------------------------------------------
[ 6] local 172.31.17.23 port 55589 connected with 22.23.24.25 port 5701
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 797 MBytes 669 Mbits/sec
[ 4] 0.0-10.0 sec 662 MBytes 553 Mbits/sec
Note:
I forgot to mention that I can connect from hazelcast client to server i.e. when I use a hazelcast client to connect to a single hazlecast server node, I am able to connect just fine
An outbound ports range which includes 0 is interpreted by hazelcast as "use ephemeral ports", so the <outbound-ports> element has actually no effect in your configuration. There is an associated test in hazelcast sources: https://github.com/hazelcast/hazelcast/blob/75251c4f01d131a9624fc3d0c4190de5cdf7d93a/hazelcast/src/test/java/com/hazelcast/nio/NodeIOServiceTest.java#L60