TLS / SSL

Pilot supports TLS encryption in two contexts:

1. Kafka client TLS - encrypted connections to Kafka brokers
2. Server TLS - native HTTPS for the Pilot HTTP server (without a reverse proxy)

Kafka Client TLS

To connect to Kafka brokers over SSL/TLS, set the security protocol and provide certificates.

Basic SSL (Server Verification Only)

KAFKA_SECURITY_PROTOCOL=SSL
KAFKA_SSL_CA_CERT_FILE=/certs/ca.pem

Mutual TLS (mTLS)

When the brokers require client authentication (ssl.client.auth=required), give Pilot a client certificate and its private key, both PEM:

KAFKA_SECURITY_PROTOCOL=SSL
KAFKA_SSL_CA_CERT_FILE=/certs/ca.pem
KAFKA_SSL_CERT_FILE=/certs/pilot-client.pem
KAFKA_SSL_KEY_FILE=/certs/pilot-client-key.pem

The certificate file may hold the chain, leaf first. KAFKA_SSL_CERT_PEM and KAFKA_SSL_KEY_PEM take the PEM content inline instead of a path (useful for Kubernetes secrets); file and inline forms can be mixed. When both forms of the same item are set, the inline PEM is used and a warning is logged.

The key may be unencrypted (PKCS#1, PKCS#8 or EC) or PKCS#8 encrypted (BEGIN ENCRYPTED PRIVATE KEY with PBES2, as openssl pkcs8 -topk8 produces and Kafka’s own PEM key stores accept). An encrypted key needs its password:

openssl pkcs8 -topk8 -in pilot-client-key.pem -out pilot-client-key.enc.pem   # encrypt, or convert a legacy-encrypted key
KAFKA_SSL_KEY_FILE=/certs/pilot-client-key.enc.pem
KAFKA_SSL_KEY_PASSWORD=secret

PEM only: PKCS#12 and JKS keystores are not read, so export the certificate and key to PEM first. OpenSSL’s legacy PEM encryption (a Proc-Type: 4,ENCRYPTED header) is not supported; convert the key with the openssl pkcs8 -topk8 command above.

Pilot stops at startup, naming the variable, when a certificate or key cannot be read, parsed or decrypted, when only one of the two is set, or when the key does not match the certificate. A password set for an unencrypted key is ignored with a warning. The pair is used under SSL and SASL_SSL only. The certificate and key are read when Pilot starts and whenever it opens a new Kafka client, so restart Pilot after rotating them.

Inline PEM Certificates

Instead of a file path, you can provide the CA certificate content directly (useful for Kubernetes secrets):

KAFKA_SSL_CA_CERT_PEM="-----BEGIN CERTIFICATE-----\n..."

KAFKA_SSL_CA_CERT_FILE and KAFKA_SSL_CA_CERT_PEM may be set together; the broker connection trusts the union of both. When KAFKA_SASL_MECHANISM=OAUTHBEARER, the same CA is also trusted for the HTTPS token endpoint, in addition to the system roots there. See Token Endpoint TLS.

SSL Options

KAFKA_SSL_VERIFY_HOSTNAME=false (case-insensitive; any other value keeps verification enabled) skips only the hostname match: the broker certificate must still chain to KAFKA_SSL_CA_CERT_FILE / KAFKA_SSL_CA_CERT_PEM (or the system roots when neither is set). Use it when broker certificates are CA-valid but do not contain the addresses Pilot connects to; it does not fix unknown authority errors. It applies only under SSL and SASL_SSL, does not affect the OAUTHBEARER token endpoint, and is overridden by KAFKA_SSL_INSECURE_SKIP_VERIFY=true, which disables chain verification as well.

Warning: Setting KAFKA_SSL_INSECURE_SKIP_VERIFY=true disables all certificate validation, for the brokers and for the OAUTHBEARER token endpoint. Use only for testing.

Docker Volume Mounts

Mount the CA certificate into the container:

services:
  pilot:
    image: calinora/pilot:latest
    volumes:
      - ./certs/ca.pem:/certs/ca.pem:ro
    environment:
      KAFKA_SECURITY_PROTOCOL: SSL
      KAFKA_SSL_CA_CERT_FILE: /certs/ca.pem

Combined with SASL

For SASL + SSL, use SASL_SSL as the security protocol:

KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=SCRAM-SHA-256
KAFKA_SASL_USERNAME=pilot
KAFKA_SASL_PASSWORD=secret
KAFKA_SSL_CA_CERT_FILE=/certs/ca.pem

See SASL Authentication for SASL-specific configuration.

Server TLS

Pilot can serve HTTPS directly without a reverse proxy.

SERVER_TLS_ENABLED=true
SERVER_TLS_CERT_FILE=/certs/server.pem
SERVER_TLS_KEY_FILE=/certs/server-key.pem
SERVER_TLS_MIN_VERSION=1.2               # "1.2" or "1.3"

Both SERVER_TLS_CERT_FILE and SERVER_TLS_KEY_FILE are required when TLS is enabled. The minimum TLS version defaults to 1.2.

Docker Example

services:
  pilot:
    image: calinora/pilot:latest
    ports:
      - "8443:8080"
    volumes:
      - ./certs/server.pem:/certs/server.pem:ro
      - ./certs/server-key.pem:/certs/server-key.pem:ro
    environment:
      SERVER_TLS_ENABLED: "true"
      SERVER_TLS_CERT_FILE: /certs/server.pem
      SERVER_TLS_KEY_FILE: /certs/server-key.pem

Access via https://localhost:8443.

For TLS termination at an ingress or reverse proxy instead, see Ingress.