Docker Compose

Docker Compose is useful for running Pilot alongside a multi-broker Kafka cluster locally or in test environments.

Minimal Setup

A simple setup with a single broker:


services:
  broker:
    image: apache/kafka:latest
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
 
  pilot:
    image: calinora/pilot:latest
    ports:
      - "8080:8080"
    environment:
      KAFKA_BOOTSTRAP_SERVERS: broker:9092
    depends_on:
      - broker

Security Variants

The repository includes pre-configured compose files for different security setups:

File	Security
`docker-compose.yml`	SASL/PLAIN
`docker-compose-sasl.yml`	SASL/PLAIN on a dedicated SASL listener
`docker-compose-tls.yml`	SSL/TLS with certificates
`docker-compose-oauth.yml`	OAUTHBEARER

SASL Example


pilot:
  image: calinora/pilot:latest
  environment:
    KAFKA_BOOTSTRAP_SERVERS: broker1:9092,broker2:9092,broker3:9092
    KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
    KAFKA_SASL_MECHANISM: PLAIN
    KAFKA_SASL_USERNAME: pilot
    KAFKA_SASL_PASSWORD: pilot-secret

TLS Example


pilot:
  image: calinora/pilot:latest
  volumes:
    - ./certs:/certs:ro
  environment:
    KAFKA_BOOTSTRAP_SERVERS: broker1:9093,broker2:9093,broker3:9093
    KAFKA_SECURITY_PROTOCOL: SSL
    KAFKA_SSL_CA_CERT_FILE: /certs/ca.pem

For brokers that require a client certificate, add KAFKA_SSL_CERT_FILE and KAFKA_SSL_KEY_FILE. See Mutual TLS.

Pilot Configuration

Configure Pilot via environment variables in the compose file:


pilot:
  image: calinora/pilot:latest
  ports:
    - "8080:8080"
  environment:
    KAFKA_BOOTSTRAP_SERVERS: broker1:9092,broker2:9092,broker3:9092
    LOG_LEVEL: INFO
    PILOT_BALANCE_THRESHOLD: "5.0"
    PILOT_THROTTLE_RATE_MB: "50"
    PILOT_EXCLUDE_TOPICS: "__consumer_offsets,__transaction_state"
    LICENSE_STRING: "${LICENSE_STRING:-}"

Health Checks


pilot:
  image: calinora/pilot:latest
  healthcheck:
    test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/api/v1/health"]
    interval: 10s
    timeout: 5s
    retries: 3