OAuth for Kafka

Pilot supports the OAUTHBEARER SASL mechanism for token-based authentication with Kafka brokers. Any identity provider that exposes an OAuth2 client-credentials token endpoint works; Azure AD / Entra ID, Keycloak, AD FS, and Okta are common choices.

Note: This page covers OAuth authentication for Kafka broker connections. For OAuth authentication of the Pilot UI and API, see UI & API Authentication.

Configuration


KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=OAUTHBEARER
KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL=https://auth.example.com/oauth/token
KAFKA_SASL_OAUTH_CLIENT_ID=pilot-client
KAFKA_SASL_OAUTH_CLIENT_SECRET=client-secret
KAFKA_SASL_OAUTH_SCOPE=kafka

Pilot uses the OIDC client credentials grant to obtain and refresh tokens automatically.

Environment Variables

Variable	Default	Description
`KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL`	`""`	Token endpoint URL (required)
`KAFKA_SASL_OAUTH_CLIENT_ID`	`""`	OAuth client ID
`KAFKA_SASL_OAUTH_CLIENT_SECRET`	`""`	OAuth client secret
`KAFKA_SASL_OAUTH_SCOPE`	`""`	OAuth scope (e.g. `kafka`, `api://kafka/.default`)
`KAFKA_SASL_OAUTH_EXTENSIONS`	`""`	SASL extensions sent to the brokers (comma-separated `key=value`)

Examples

Azure AD / Entra ID


KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=OAUTHBEARER
KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL=https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token
KAFKA_SASL_OAUTH_CLIENT_ID=<app-registration-client-id>
KAFKA_SASL_OAUTH_CLIENT_SECRET=<client-secret>
KAFKA_SASL_OAUTH_SCOPE=api://<kafka-cluster-id>/.default
KAFKA_SSL_CA_CERT_FILE=/certs/ca.pem

Keycloak


KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=OAUTHBEARER
KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL=https://keycloak.example.com/realms/kafka/protocol/openid-connect/token
KAFKA_SASL_OAUTH_CLIENT_ID=pilot
KAFKA_SASL_OAUTH_CLIENT_SECRET=client-secret
KAFKA_SASL_OAUTH_SCOPE=kafka
KAFKA_SSL_CA_CERT_FILE=/certs/ca.pem

AD FS


KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=OAUTHBEARER
KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL=https://<adfs-host>/adfs/oauth2/token
KAFKA_SASL_OAUTH_CLIENT_ID=<client-id>
KAFKA_SASL_OAUTH_CLIENT_SECRET=<client-secret>
KAFKA_SASL_OAUTH_SCOPE=kafka
KAFKA_SSL_CA_CERT_FILE=/certs/ca.pem

OAuth Extensions

Some Kafka deployments require additional SASL extensions from OAuth clients:


KAFKA_SASL_OAUTH_EXTENSIONS="audience=kafka-cluster,issuer=example.com"

Extensions are comma-separated key=value pairs; whitespace around keys and values is trimmed, and entries without = are dropped. They are sent to the brokers with the SASL OAUTHBEARER initial client response on every connection, not to the token endpoint (Kafka 2.1.0+; brokers ignore unexpected extensions). The key auth is reserved by the SASL mechanism itself and is rejected at startup.

Token Endpoint TLS

The token endpoint is called over HTTPS by a client separate from the Kafka connection. Its trust store is the system roots plus the CA configured for the brokers via KAFKA_SSL_CA_CERT_FILE or KAFKA_SSL_CA_CERT_PEM. A token endpoint signed by a private or corporate CA therefore works as soon as that CA is supplied for the brokers, and a publicly signed endpoint keeps working alongside a private broker CA. Under SASL_PLAINTEXT the same two variables (and KAFKA_SSL_INSECURE_SKIP_VERIFY) are read for the token endpoint alone, since the brokers themselves use no TLS.


KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL=https://sso.corp.example.com/oauth2/token   # signed by the corporate CA
KAFKA_SSL_CA_CERT_FILE=/certs/corp-ca.pem                                        # trusted for brokers and the token endpoint

KAFKA_SSL_INSECURE_SKIP_VERIFY=true disables certificate verification for the token endpoint as well as for the brokers; it is not safe for production. If the token endpoint is signed by a CA different from the broker CA, put both certificates in the same PEM bundle.

Docker Compose Example


services:
  pilot:
    image: calinora/pilot:latest
    environment:
      KAFKA_BOOTSTRAP_SERVERS: broker:9092
      KAFKA_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_SASL_MECHANISM: OAUTHBEARER
      KAFKA_SASL_OAUTH_TOKEN_ENDPOINT_URL: https://auth.example.com/oauth/token
      KAFKA_SASL_OAUTH_CLIENT_ID: pilot-client
      KAFKA_SASL_OAUTH_CLIENT_SECRET: "${OAUTH_CLIENT_SECRET}"
      KAFKA_SASL_OAUTH_SCOPE: kafka
      KAFKA_SSL_CA_CERT_FILE: /certs/ca.pem
    volumes:
      - ./certs/ca.pem:/certs/ca.pem:ro