Skip to Content
Getting StartedDocker Quick Start

Docker Quick Start

Run a multi-broker Kafka cluster with Pilot using Docker Compose. This setup mirrors a environment with two brokers, rack awareness, and replication.

Prerequisites

  • Docker and Docker Compose installed
  • Ports 8080 (Pilot), 9092-9093 (Kafka) available

1. Create docker-compose.yml

docker-compose.yml
services:
  controller:
    image: apache/kafka:latest
    container_name: controller
    environment:
      KAFKA_NODE_ID: 0
      KAFKA_PROCESS_ROLES: controller
      KAFKA_LISTENERS: CONTROLLER://:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 0@controller:9093
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
 
  broker-1:
    image: apache/kafka:latest
    container_name: broker-1
    depends_on: [controller]
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-1:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 0@controller:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_DEFAULT_REPLICATION_FACTOR: 2
      KAFKA_MIN_INSYNC_REPLICAS: 1
      KAFKA_BROKER_RACK: rack-a
    healthcheck:
      test: /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
      start_period: 15s
      interval: 10s
      timeout: 10s
      retries: 5
 
  broker-2:
    image: apache/kafka:latest
    container_name: broker-2
    depends_on: [controller]
    ports:
      - "9093:9092"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-2:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 0@controller:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_DEFAULT_REPLICATION_FACTOR: 2
      KAFKA_MIN_INSYNC_REPLICAS: 1
      KAFKA_BROKER_RACK: rack-b
    healthcheck:
      test: /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
      start_period: 15s
      interval: 10s
      timeout: 10s
      retries: 5
 
  pilot:
    image: calinora/pilot:latest
    container_name: pilot
    ports:
      - "8080:8080"
    environment:
      KAFKA_BOOTSTRAP_SERVERS: broker-1:9092,broker-2:9092
      LOG_LEVEL: INFO
      PILOT_EXCLUDE_TOPICS: __consumer_offsets,__transaction_state
    depends_on:
      broker-1:
        condition: service_healthy
      broker-2:
        condition: service_healthy

2. Start the Stack

docker compose up -d

Pilot will wait for both brokers to be healthy before starting.

3. Access the Dashboard

Open http://localhost:8080  in your browser. You should see both brokers with their rack assignments.

4. Verify Connectivity

# Health check
curl http://localhost:8080/api/v1/health
 
# Cluster info - should show 2 brokers
curl http://localhost:8080/api/v1/cluster

5. Create Test Data

Create topics with replication and produce messages to generate activity:

# Create topics with RF=2 so partitions are spread across both brokers
docker exec broker-1 /opt/kafka/bin/kafka-topics.sh --create \
  --topic orders --partitions 6 --replication-factor 2 \
  --bootstrap-server localhost:9092
 
docker exec broker-1 /opt/kafka/bin/kafka-topics.sh --create \
  --topic events --partitions 4 --replication-factor 2 \
  --bootstrap-server localhost:9092
 
# Produce messages to generate activity
docker exec broker-1 /opt/kafka/bin/kafka-producer-perf-test.sh \
  --topic orders --num-records 10000 \
  --record-size 256 --throughput 1000 \
  --command-property bootstrap.servers=localhost:9092
 
docker exec broker-1 /opt/kafka/bin/kafka-producer-perf-test.sh \
  --topic events --num-records 5000 \
  --record-size 128 --throughput 500 \
  --command-property bootstrap.servers=localhost:9092

Refresh the dashboard to see topics with activity scores and partition distribution across both brokers.

6. View the Proposal

Click Proposals in the navigation. With two brokers and multiple topics, Pilot will show whether the partition distribution is balanced and suggest moves if needed.

Cleanup

docker compose down -v

Next Steps

Last updated on
Quick StartOverview