Reconnecting to Kafka with node-rdkafka is slow & inconsistent

I have kafka and zookeeper running in some local docker containers.

I have a node.js codebase which uses node-rdkafka to connect to kafka as a consumer. We'll call this codebase "the consumer"

The consumer uses the code shown here to connect to kafka: https://github.com/Blizzard/node-rdkafka/blob/master/examples/consumer-flow.md

When attempting to connect to an existing Kafka instance, I get this output:

consumer ready.{"name":"rdkafka#consumer-1"}

But then my code which receives and processes messages never fires, or takes several mins (apparently 5 to 10 mins sometimes) before it begins firing.

To ensure that messages are being produced, I use the script kafka-console-consumer.sh which came with kafka to observe the message stream. Sure enough, PLENTY of data is coming.

It appears that connecting sometimes fails, and sometimes takes a very long time to connect.

Ideas?

For what it's worth, I used the kakfa-node module for a while too, and that appeared to have similar issues, though I can't be sure.

via lance.dolan