Skip to the content.

kafka-docker

This project is based on kafka docker modification. For more details, please refer to the original project wurstmeister/kafka-docker


Pre-Requisites

NOTE: There are several ‘gotchas’ with configuring networking. If you are not sure about what the requirements are, please check out the Connectivity Guide

Usage

Start a cluster:

Destroy a cluster:

test kafka:

cd kafka_2.12-2.3.0/bin/
./kafka-topics.sh --create --bootstrap-server kafka-com:9094 --replication-factor 1 --partitions 1  --topic test
./kafka-topics.sh --list --bootstrap-server kafka-com:9094

./kafka-console-producer.sh --broker-list kafka-com:9094 --topic test

./kafka-console-consumer.sh --bootstrap-server kafka-com:9094 --topic test --from-beginning

Broker IDs

You can configure the broker id in different ways

  1. explicitly, using KAFKA_BROKER_ID
  2. via a command, using BROKER_ID_COMMAND, e.g. BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $$2}'"

If you don’t specify a broker id in your docker-compose file, it will automatically be generated (see https://issues.apache.org/jira/browse/KAFKA-1070. This allows scaling up and down. In this case it is recommended to use the --no-recreate option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.

Automatically create topics

If you want to have kafka-docker automatically create topics in Kafka during creation, a KAFKA_CREATE_TOPICS environment variable can be added in docker-compose.yml.

Here is an example snippet from docker-compose.yml:

    environment:
      KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"

Topic 1 will have 1 partition and 3 replicas, Topic 2 will have 1 partition, 1 replica and a cleanup.policy set to compact. Also, see FAQ: Topic compaction does not work

If you wish to use multi-line YAML or some other delimiter between your topic definitions, override the default , separator by specifying the KAFKA_CREATE_TOPICS_SEPARATOR environment variable.

For example, KAFKA_CREATE_TOPICS_SEPARATOR: "$$'\n'" would use a newline to split the topic definitions. Syntax has to follow docker-compose escaping rules, and ANSI-C quoting.

Advertised hostname

You can configure the advertised hostname in different ways

  1. explicitly, using KAFKA_ADVERTISED_HOST_NAME
  2. via a command, using HOSTNAME_COMMAND, e.g. HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"

When using commands, make sure you review the “Variable Substitution” section in https://docs.docker.com/compose/compose-file/

If KAFKA_ADVERTISED_HOST_NAME is specified, it takes precedence over HOSTNAME_COMMAND

For AWS deployment, you can use the Metadata service to get the container host’s IP:

HOSTNAME_COMMAND=wget -t3 -T2 -qO-  http://169.254.169.254/latest/meta-data/local-ipv4

Reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Injecting HOSTNAME_COMMAND into configuration

If you require the value of HOSTNAME_COMMAND in any of your other KAFKA_XXX variables, use the _{HOSTNAME_COMMAND} string in your variable value, i.e.

KAFKA_ADVERTISED_LISTENERS=SSL://_{HOSTNAME_COMMAND}:9093,PLAINTEXT://9092

Advertised port

If the required advertised port is not static, it may be necessary to determine this programatically. This can be done with the PORT_COMMAND environment variable.

PORT_COMMAND: "docker port $$(hostname) 9092/tcp | cut -d: -f2"

This can be then interpolated in any other KAFKA_XXX config using the _{PORT_COMMAND} string, i.e.

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://1.2.3.4:_{PORT_COMMAND}

Listener Configuration

It may be useful to have the Kafka Documentation open, to understand the various broker listener configuration options.

Since 0.9.0, Kafka has supported multiple listener configurations for brokers to help support different protocols and discriminate between internal and external traffic. Later versions of Kafka have deprecated advertised.host.name and advertised.port.

NOTE: advertised.host.name and advertised.port still work as expected, but should not be used if configuring the listeners.

Example

The example environment below:

HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

Will result in the following broker config:

advertised.listeners = OUTSIDE://ec2-xx-xx-xxx-xx.us-west-2.compute.amazonaws.com:9094,INSIDE://:9092
listeners = OUTSIDE://:9094,INSIDE://:9092
inter.broker.listener.name = INSIDE

Rules

Broker Rack

You can configure the broker rack affinity in different ways

  1. explicitly, using KAFKA_BROKER_RACK
  2. via a command, using RACK_COMMAND, e.g. RACK_COMMAND: "curl http://169.254.169.254/latest/meta-data/placement/availability-zone"

In the above example the AWS metadata service is used to put the instance’s availability zone in the broker.rack property.

JMX

For monitoring purposes you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect

For example, to connect to a kafka running locally (assumes exposing port 1099)

  KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
  JMX_PORT: 1099

Jconsole can now connect at jconsole 192.168.99.100:1099