Redis Cluster 3.2.0 on EC2 t2.micro (no Elasticache) - amazon-web-services

Question

After much ado, I've come to an impasse. I'm trying to setup a redis cluster of 3 master nodes and 3 slaves on a single t2.micro. My setup on my localhost works great but when I try to run it on EC2 I am encountering a strange problem where my client (on a separate t2.micro using ioredis) can seem to find and connect but then throws many errors repeatedly like "ioredis:connection error: Error: connect ECONNREFUSED" if I have my client in http. If I switch to https I get additional different timeout errors and "manually closed" errors (tried setting the TLS flag in cluster options to no avail).
TL;DR
Thoughts?? Why can I not create the cluster with the publicIP (rather than 127.0.0.1) using redis-trib? This would seem to solve my problems or is there something obvious that I'm missing here like a firewall?...
If you're reading this and struggling with redis, the list of points below could serve as a good summary of almost every proposed redis solution on the top pages of google and stackoverflow. Use them well!
After reading up on several similar topics I've found none of them have addressed the problem. Here's what I tried;
Checked my EC2 security groups to ensure the right ports were open between my redis t2.micro and client t2.micro. Ensured that redis ports+10,000 (for the bus) were also open.
Checked my AWS vpc, internet gateway, subnets and acls to ensure traffic could flow between the two instances
Ran some netstat and it looks like I can connect to the correct ports and that redis is listening on the right ports
Ensured in the redis.conf file for each node that the protected-mode (set to no), bind (commented out) and password fields (commented out) weren't inhibiting communication. At first this was part of the problem. At one point I turning all of them off and still ended up having the same errors.
I removed any old aof, dump.rdb, node.conf files and started fresh instances. I ensured each node had its own folder (no sharing of node.conf files).
I tried connecting the redis cluster using the loopback 127.0.0.1 like so:
./redis-trib.rb create --replicas 1 127.0.0.1:30010 127.0.0.1:30011 127.0.0.1:30012 127.0.0.1:30013 127.0.0.1:30014 127.0.0.1:30015
and still had errors from the client. So then tried the aws public host address of the redis t2.micro, then the public IP, and then the private IP. When I start the nodes (using ps -ef to ensure they are running in daemon mode) and then try ./redis-trib create --replicas 1 publicIP:30010 ..etc using the public IP it looks like it will create the cluster but then hangs at ">>>creating cluster" until it fails and says it cannot connect to the first node. It will not let me create the cluster with the publicIP instead of the 127.0.0.1 (which I suspect is the problem of why my client cannot connect). It seems like other people have had success connecting it but not in this case (I also tried to run redis-trib from my client and it would connect and generate the aof, and node.conf on the redis t2.micro but it would also hang and eventually say it couldn't find the nodes...)
Once I had the cluster up and running under the 127.0.0.1 the nodes would communicate and redis-cli returns PONG to my ping but to set a key it gives "(error) MOVED 16164 127.0.0.1:30012" and the same for 'get'. So I tried to manually set the publicIP by sending a "cluster meet" as in this example: redis-cli redirected to 127.0.0.1
Still no go. While I set the meet some of the 127.0.0.1 remained or the ones that I did set with the publicIP seemed to switch back by the time I'd finished running through all the nodes.
The only thing left to think is if AWS is blocking ports somewhere. I tried opening all ports to both t2.micro instances and opened them wide open to anyone and it still didn't work. I thought about looking into iptables on EC2 instances but they shouldn't be set given that there are security groups (and I haven't messed around with iptables much). I thought this was going to take me an hour and now I'm still sitting here scratching my head.
Some potentially useful code:
Cluster Code:
export var cluster = new Redis.Cluster([{
port: 30010,
host: '52.36.xxx.xxx'
}, {
port: 30011,
host: '52.36.xxx.xxx'
},{
port: 30012,
host: '52.36.xxx.xxx'
}]);
30010 nodes.conf
337e0c0152cc88590d73048a6f97120934d94da8 127.0.0.1:30010 myself,master - 0 0 1 connected 0-5460
8f7cf7a0016c372ebaaffd76b903e26e47f2a513 127.0.0.1:30014 slave 882fed6d144b6dea1531691deb323a3ae0b52936 0 1471601371978 5 connected
2c36b871bbdb6f8b98a2562ff315bf79ca524ec5 127.0.0.1:30013 slave 337e0c0152cc88590d73048a6f97120934d94da8 1471601372982 1471601368969 4 connected
265b166b7231a7c0a8017f4f7fad90261d59fb96 127.0.0.1:30015 slave 42e5b9b8ab9e1d2eefe1832e118085b4e44ae65d 0 1471601367966 6 connected
882fed6d144b6dea1531691deb323a3ae0b52936 127.0.0.1:30011 master - 0 1471601369972 2 connected 5461-10922
42e5b9b8ab9e1d2eefe1832e118085b4e44ae65d 127.0.0.1:30012 master - 0 1471601370977 3 connected 10923-16383
vars currentEpoch 6 lastVoteEpoch 0
127.0.0.1:30010> cluster nodes
337e0c0152cc88590d73048a6f97120934d94da8 127.0.0.1:30010 myself,master - 0 0 1 connected 0-5460
8f7cf7a0016c372ebaaffd76b903e26e47f2a513 127.0.0.1:30014 slave 882fed6d144b6dea1531691deb323a3ae0b52936 0 1471601610630 5 connected
2c36b871bbdb6f8b98a2562ff315bf79ca524ec5 127.0.0.1:30013 slave 337e0c0152cc88590d73048a6f97120934d94da8 0 1471601611632 4 connected
265b166b7231a7c0a8017f4f7fad90261d59fb96 127.0.0.1:30015 slave 42e5b9b8ab9e1d2eefe1832e118085b4e44ae65d 0 1471601609627 6 connected
882fed6d144b6dea1531691deb323a3ae0b52936 127.0.0.1:30011 master - 0 1471601612634 2 connected 5461-10922
42e5b9b8ab9e1d2eefe1832e118085b4e44ae65d 127.0.0.1:30012 master - 0 1471601607622 3 connected 10923-16383
Client errors : sudo DEBUG=ioredis:* node app.js
ioredis:redis status[127.0.0.1:30010]: close -> end +1ms
ioredis:redis status[127.0.0.1:30012]: wait -> connecting +0ms
ioredis:connection error: Error: connect ECONNREFUSED 127.0.0.1:30012 +0ms
ioredis:redis status[127.0.0.1:30012]: connecting -> close +0ms
ioredis:connection skip reconnecting because `retryStrategy` is not a function +0ms
ioredis:redis status[127.0.0.1:30012]: close -> end +0ms
ioredis:cluster status: connect -> close +0ms
ioredis:cluster status: close -> reconnecting +0ms
ioredis:delayqueue send 1 commands in failover queue +94ms
REDIS222 CONNECT error Error: Failed to refresh slots cache.
node error Error: timeout
at Object.exports.timeout (/home/ubuntu/main2/node_modules/ioredis/lib/utils/index.js:153:36)
at Cluster.getInfoFromNode (/home/ubuntu/main2/node_modules/ioredis/lib/cluster/index.js:552:32)
at tryNode (/home/ubuntu/main2/node_modules/ioredis/lib/cluster/index.js:347:11)
at Cluster.refreshSlotsCache (/home/ubuntu/main2/node_modules/ioredis/lib/cluster/index.js:362:3)
SSH in to redis t2.micro and netstat. Seems to be listening on correct ports (30010-30015
ubuntu#ip-xxx-xx-xx-xxx:~$ sudo netstat -ntlp | grep LISTEN
tcp 0 0 0.0.0.0:40013 0.0.0.0:* LISTEN 1328/redis-server *
tcp 0 0 0.0.0.0:40014 0.0.0.0:* LISTEN 1334/redis-server *
tcp 0 0 0.0.0.0:40015 0.0.0.0:* LISTEN 1336/redis-server *
tcp 0 0 0.0.0.0:30010 0.0.0.0:* LISTEN 1318/redis-server *
tcp 0 0 0.0.0.0:30011 0.0.0.0:* LISTEN 1322/redis-server *
tcp 0 0 0.0.0.0:30012 0.0.0.0:* LISTEN 1324/redis-server *
tcp 0 0 0.0.0.0:30013 0.0.0.0:* LISTEN 1328/redis-server *
tcp 0 0 0.0.0.0:30014 0.0.0.0:* LISTEN 1334/redis-server *
tcp 0 0 0.0.0.0:30015 0.0.0.0:* LISTEN 1336/redis-server *
tcp 0 0 0.0.0.0:40010 0.0.0.0:* LISTEN 1318/redis-server *
tcp 0 0 0.0.0.0:40011 0.0.0.0:* LISTEN
1322/redis-server *
tcp 0 0 0.0.0.0:40012 0.0.0.0:* LISTEN
SSH into the client t2.micro and remotely call cluster nodes from the redis remote server and it returns the correct loopback setup:
ubuntu#ip-xxx-xx-xx-x:~/redis-3.2.2/src$ ./redis-cli -h 52.36.237.185 -p 30010 cluster nodes
337e0c0152cc88590d73048a6f97120934d94da8 127.0.0.1:30010 myself,master - 0 0 1 connected 0-5460
8f7cf7a0016c372ebaaffd76b903e26e47f2a513 127.0.0.1:30014 slave 882fed6d144b6dea1531691deb323a3ae0b52936 0 1471629274223 5 connected
2c36b871bbdb6f8b98a2562ff315bf79ca524ec5 127.0.0.1:30013 slave 337e0c0152cc88590d73048a6f97120934d94da8 0 1471629275225 4 connected
265b166b7231a7c0a8017f4f7fad90261d59fb96 127.0.0.1:30015 slave 42e5b9b8ab9e1d2eefe1832e118085b4e44ae65d 0 1471629272217 6 connected
882fed6d144b6dea1531691deb323a3ae0b52936 127.0.0.1:30011 master - 0 1471629276228 2 connected 5461-10922
42e5b9b8ab9e1d2eefe1832e118085b4e44ae65d 127.0.0.1:30012 master - 0 1471629277231 3 connected 10923-16383
-------------------------------------------------------
Thoughts?? Why can I not create the cluster with the publicIP (rather than 127.0.0.1) using redis-trib? This would seem to solve my problems or is there something obvious that I'm missing here like a firewall...
................................
UPDATE
I ran redis-trib.rb check locally on the redis server and it showed everything is dandy:
ubuntu#ip-172-xx-xx-xxx:~/redis-3.2.2/src$ ./redis-trib.rb check 127.0.0.1:30010
>>> Performing Cluster Check (using node 127.0.0.1:30010)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
But when I run it from my client on a different instance using the redis publicIP I get:
ubuntu#ip-172-xx-xx-x:~/redis-3.2.2/src$ ./redis-trib.rb check redispublicIP:30010
[ERR] Sorry, can't connect to node 127.0.0.1:30014
[ERR] Sorry, can't connect to node 127.0.0.1:30013
[ERR] Sorry, can't connect to node 127.0.0.1:30015
[ERR] Sorry, can't connect to node 127.0.0.1:30011
[ERR] Sorry, can't connect to node 127.0.0.1:30012
>>> Performing Cluster Check (using node redispublicIP:30010)
M: 337e0c0152cc88590d73048a6f97120934d94da8 redispublicIP:30010
slots:0-5460 (5461 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
So it does look like I need to switch that 127.0.0.1. It allows me to connect to a single node from the client if I use the publicIP:port but when it tries to find the other nodes it must be thinking they are local
Update2:
Seems like this is my problem but I've double checked and no passwords are set in any of the 6 redis.conf files:
Getting a connection error when using redis-trib.rb to create a cluster?
Update3: This article is very close but I do not understand his solution:
src/redis-trib.rb create 127.0.0.1:6379 127.0.0.1:6380 h2:p1 h2:p2 h3:p1 h3:p2
Specifically why hes declaring the host and ports after h2:p1 h2:p2 h3:p1 h3:p2
Update4:
It appears that this may be an issue with AWS t2.micro instances. I've sent a request to AWS Support:
https://forums.aws.amazon.com/thread.jspa?messageID=647509
SOLVED:
It was using the private IP address in both the client and the redis-trib create command. I had tried the private IP in the client configuration but mistakenly thought I had tried the redis-trib with it.
For anyone else: Lesson: use the private IP of the redis EC2 instance. Thanks to this video for helping me catch on:
https://www.youtube.com/watch?v=s4YpCA2Y_-Q

Answer 1

SOLVED:
It was using the private IP address in both the client and the redis-trib create command that solved the issue. I had tried the private IP in the client configuration but mistakenly thought I had tried the redis-trib with it.
For anyone else=> Lesson: use the private IP of the redis EC2 instance and not the public when joining the cluster with redis-trib. Thanks to this video for helping me catch on:
https://www.youtube.com/watch?v=s4YpCA2Y_-Q