Can't talk to Kafka broker on AWS instance - amazon-web-services

Question

I realize this question comes up a lot. I've read many threads and blog posts, but here I am. Long story short, I have a docker container running wurstmeister/zookeeper & wurstmeister/kafka, and then some services running in their own containers. I'll just mention the NoddJS one for now. Everything works fine at home, using IP addresses (not localhost) so I'm baffled at what the difference is here. On AWS, it simply "doesn't work" even though it seems to at least connect to the broker in the beginning. I'm explicitly using internal IPs in the config as I don't want this exposed to anything externally.
Reading around, I've tried 2 setups. 1 works at home (KAFKA_ADVERTISED_HOST_NAME). 1 doesn't (KAFKA_ADVERTISED_LISTENERS). Neither work on my EC2 Linux box:
Kafka docker-compose.yml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
- my-network
kafka:
image: wurstmeister/kafka
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: <internal-ip>
KAFKA_ADVERTISED_PORT: "9092"
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- my-network
networks:
my-network:
NodeJS docker-compose.yml
version: '2'
services:
nodejs:
build:
context: ./
dockerfile: Dockerfile
networks:
- kafka_my-network
restart: unless-stopped
ports:
- "1337:3000"
volumes:
- "/tmp:/tmp"
- "/var/log:/var/log"
networks:
kafka_my-network:
external: true
Then in NodeJS
const kafkaHost = '<internal-ip>:9092';
const client = new kafka.KafkaClient({kafkaHost});
const producer = new kafka.Producer(client)
const kakfaTopic = 'test';
producer.on('ready', function() {
console.log(`kafka producer is ready`); // I see this, so I'm assuming all is well
ready = true;
});
producer.on('error', function(err) {
console.error(err);
});
const payload = [
{
topic: kafkaTopic,
messages: JSON.stringify(myMessages);
}
]
producer.send(payload, function(err, data) {
if (err) {
console.error(`Send error ${JSON.stringify(err}`);
}
console.log(`Sent data ${JSON.stringify(data)}`);
});
When I start my NodeJS server, I see that I've connected to a Kafka broker. I can confirm as well that :9092 is open after checking w/ telnet and/or nc. Then, when it sends a request, the callback gets an empty error.
I realize KAKFA_ADVERTISED_HOST_NAME is deprecated, so in the name of completion, here is my attempt using ADVERTISED_LISTENERS which failed. With this configuration I seemed to get the same results at home as I did on EC2.
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
- my-network
kafka:
image: wurstmeister/kafka
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://<internal-ip>:9092
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- my-network
networks:
my-network:
EDIT: I will not provide this as a solution, but the bitnami image w/ the following config works. The main difference is it had a pretty straight forward README that I went through. I can't be certain if I tried the equivalent config w/ wurstmeister (I tried many - and again, at least one of which in docker containers on my own machine but not on a single EC2 instance).
Note that I did list 'kafka' with an internal IP (not loopback) in /etc/hosts for this. This should be tantamount to using the internal IP explicitly which I had done above.
version: '2'
services:
zookeeper:
image: 'bitnami/zookeeper:3'
ports:
- '2181:2181'
volumes:
- 'zookeeper_data:/bitnami'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
networks:
- my-network
kafka:
image: 'bitnami/kafka:2'
ports:
- '9092:9092'
- '29092:29092'
volumes:
- 'kafka_data:/bitnami'
- /var/run/docker.sock:/var/run/docker.sock
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
depends_on:
- zookeeper
networks:
- my-network
volumes:
zookeeper_data:
driver: local
kafka_data:
driver: local
networks:
my-network:
driver: bridge