Kubernetes

Deploy Pilot to Kubernetes using standard manifests. Pilot is a single stateless binary that requires no persistent volumes or external databases.

Deployment


apiVersion: apps/v1
kind: Deployment
metadata:
  name: pilot
  labels:
    app: pilot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pilot
  template:
    metadata:
      labels:
        app: pilot
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: pilot
          image: calinora/pilot:0.27.0
          ports:
            - containerPort: 8080
              name: http
          envFrom:
            - configMapRef:
                name: pilot-config
            - secretRef:
                name: pilot-secrets
          livenessProbe:
            httpGet:
              path: /api/v1/health
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /api/v1/ready
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi

ConfigMap

Non-sensitive configuration:


apiVersion: v1
kind: ConfigMap
metadata:
  name: pilot-config
data:
  KAFKA_BOOTSTRAP_SERVERS: "broker-0.kafka:9092,broker-1.kafka:9092,broker-2.kafka:9092"
  LOG_LEVEL: "INFO"
  PILOT_BALANCE_THRESHOLD: "5.0"
  PILOT_THROTTLE_RATE_MB: "50"
  PILOT_EXCLUDE_TOPICS: "__consumer_offsets,__transaction_state"

Secret

Sensitive values (SASL credentials, license, API keys):


apiVersion: v1
kind: Secret
metadata:
  name: pilot-secrets
type: Opaque
stringData:
  KAFKA_SECURITY_PROTOCOL: "SASL_SSL"
  KAFKA_SASL_MECHANISM: "SCRAM-SHA-256"
  KAFKA_SASL_USERNAME: "pilot"
  KAFKA_SASL_PASSWORD: "your-password"
  LICENSE_STRING: "eyJ..."

Service


apiVersion: v1
kind: Service
metadata:
  name: pilot
spec:
  selector:
    app: pilot
  ports:
    - port: 8080
      targetPort: http
      name: http
  type: ClusterIP

Ingress


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pilot
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - pilot.example.com
      secretName: pilot-tls
  rules:
    - host: pilot.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pilot
                port:
                  number: 8080

Note: Set long proxy timeouts for SSE endpoints (chat streaming uses Server-Sent Events).

TLS Certificates

If your Kafka cluster uses TLS, mount the CA certificate from a Secret:


spec:
  containers:
    - name: pilot
      volumeMounts:
        - name: kafka-certs
          mountPath: /certs
          readOnly: true
      env:
        - name: KAFKA_SECURITY_PROTOCOL
          value: SSL
        - name: KAFKA_SSL_CA_CERT_FILE
          value: /certs/ca.pem
  volumes:
    - name: kafka-certs
      secret:
        secretName: kafka-client-certs

For brokers that require a client certificate, add KAFKA_SSL_CERT_FILE and KAFKA_SSL_KEY_FILE from the same Secret. See Mutual TLS.

Resource Recommendations

Cluster Size	CPU Request	Memory Request	CPU Limit	Memory Limit
Small (< 50 partitions)	100m	128Mi	500m	256Mi
Medium (50-500 partitions)	200m	256Mi	1000m	512Mi
Large (500+ partitions)	500m	512Mi	2000m	1Gi

Health Probes

Probe	Endpoint	Purpose
Liveness	`/api/v1/health`	Pilot process is running
Readiness	`/api/v1/ready`	Kafka brokers are reachable

The readiness probe ensures traffic is only routed to Pilot when it has established connectivity with the Kafka cluster.

Single Replica

Pilot is designed to run as a single replica. Running multiple replicas would cause duplicate proposal generation and conflicting reassignment execution. Use Kubernetes liveness/readiness probes for automatic restart on failure.