Pilot Agent

The Pilot Agent is a lightweight daemon deployed alongside each Kafka broker. It streams OS-level metrics, monitors Kafka log directories, and enables broker lifecycle management - all from the Pilot UI and API.

Agents are optional. Agentless monitoring via Kafka admin APIs continues to work unchanged. Agents add visibility that Kafka’s protocol cannot provide.

Quick Start

1. Enable agents on the Pilot server

PILOT_AGENT_ENABLED=true
PILOT_AGENT_GRPC_PORT=9190
PILOT_AGENT_BOOTSTRAP_TOKEN=$(openssl rand -hex 32)   # save this value

2. Deploy agents

Choose the method that fits your environment:

3. Verify

Open Agents to inspect connected agents, their broker identity, version, status, and last-seen time. Inspect host opens the corresponding broker’s host telemetry.

What Agents Provide

System Metrics

Each agent reports the following every 5 seconds:

Auto-Discovery

On connect, each agent scans the local Kafka installation and reports:

• Runtime - OS, kernel, architecture, container runtime (bare metal, Docker, Kubernetes)
• Kafka - vendor (Apache, Confluent, Strimzi, Cloudera), version, KRaft mode and role
• Process - PID, Java version, heap max, JMX port
• Paths - install directory, config file, log/data directories
• Service - systemd unit name, restart/stop/status commands
• Log files - known Kafka log files with paths and format

Log directories are auto-detected from the running broker’s server.properties. No manual path configuration needed.

Broker ID 0 is a valid explicit identity. Setting --broker-id=0 or AGENT_BROKER_ID=0 retains that identity. Omit both settings to enable automatic broker-ID detection.

Log Forwarding

For regular log files, the agent starts at the end of the file and forwards new lines. It does not replay the existing file on startup. After rotation or truncation, it reads the replacement file from the beginning so lines written before the file is reopened are included.

Broker Lifecycle Management

Agents can restart, stop, and start the Kafka broker on their host - triggered from the Pilot UI or API.

The agent uses whatever service manager it detected during discovery:

Safety controls:

• Set AGENT_ALLOW_RESTART=false to disable lifecycle commands per agent
• Each command has a configurable timeout (default: 5 minutes)
• Graceful shutdown by default (pass "graceful": false for hard kill)
• All lifecycle endpoints require a valid Pilot license

Override auto-detection if needed:

The variables above are read by the agent process at runtime. Two additional variables are read by the installer (and by the RPM/DEB postinstall hook) to pre-seed non-root mode before the agent ever starts:

These two are install-time hints only; they are not read by the running agent.

Prometheus Metrics

Every heartbeat field listed under System Metrics is also re-exported on Pilot’s central /metrics endpoint as pilot_agent_* gauges, labelled with broker_id and node_id (hostname). Operators with an existing Prometheus stack can alert on broker-host CPU, memory, disk, log-dir, network, or disk-io without adding a per-broker scrape target. When an agent disconnects, every pilot_agent_* series for that broker is removed so a decommissioned host does not leave stale gauges.

See the agent host metrics reference for the full list and example PromQL queries.

Privilege Model

The agent runs as a dedicated pilot-agent system user (no login shell, no home directory). All install paths (scripts/install-agent.sh, the UI deploy dialog, and the RPM/DEB package) set this up automatically:

• A narrow sudoers drop-in at /etc/sudoers.d/99-pilot-agent (mode 0440, validated with visudo -c) lets the pilot-agent user run only systemctl restart|stop|start|is-active|show -p MainPID|show -p ActiveState against the detected Kafka unit. Nothing else. NOPASSWD, no shell access.
• The pilot-agent user is added to the Kafka log-files group so log tailing works unprivileged.
• The systemd unit keeps ProtectSystem=strict, ProtectHome=yes, ProtectControlGroups, ProtectKernelModules, ProtectKernelTunables, and PrivateTmp=true. NoNewPrivileges=yes is intentionally omitted because it is incompatible with sudo.
• Log forwarding, metrics collection, and discovery run unprivileged. Only broker lifecycle calls escalate through the narrow sudoers allow-list.

If Kafka is not running on the host at install time, sudoers is deferred: the agent installs cleanly, log forwarding works, and broker control is enabled later with sudo pilot-agent configure-sudoers (detection re-runs automatically).

Install paths

scripts/install-agent.sh

curl -fsSL http://pilot:8080/api/v1/agents/install-script | sudo bash -s -- \
  --server pilot.internal:9190 \
  --pilot-url http://pilot:8080

Optional overrides for air-gapped hosts or custom Kafka packaging:

sudo bash install-agent.sh \
  --server pilot.internal:9190 \
  --kafka-unit confluent-kafka.service \
  --kafka-group cp-kafka

Pilot UI

Open the Deploy Agent dialog on the Brokers page and fill in the SSH target. The test-connection step detects the running Kafka unit and group on the target host and shows them as placeholders in Advanced Settings, so you can confirm what will be written to sudoers before pressing Deploy. If Kafka is not running yet on the target, the dialog flags this and the agent installs without sudoers; finish the setup later with sudo pilot-agent configure-sudoers once Kafka is up.

RPM/DEB

sudo apt-get install calinora-pilot-agent
sudo systemctl enable --now pilot-agent

If Kafka is not running yet on the host, pre-seed the unit name so the postinstall hook renders sudoers anyway:

sudo PILOT_AGENT_KAFKA_UNIT=confluent-kafka.service apt-get install calinora-pilot-agent

Custom Kafka unit names

Detection is process-based, not string-based. The agent locates the running Kafka JVM by its universal main class kafka.Kafka (the Scala class name used by Apache Kafka, Confluent Platform, Aiven, MSK self-managed, and vendor forks) and resolves the owning systemd unit via systemctl status <pid>. Unit names like kafka.service, confluent-kafka.service, cp-kafka.service, or our-platform-kafka.service all work without configuration.

For air-gapped hosts where detection cannot run, or where the unit name is known ahead of time, pass --kafka-unit <name> to install-agent.sh or set PILOT_AGENT_KAFKA_UNIT before package install.

Subcommands

The agent binary ships two operator subcommands for non-root mode:

Typical uses:

• Verify what the installer will detect: pilot-agent detect --format=env.
• Recover from a Kafka unit rename: sudo pilot-agent configure-sudoers --unit new-name.service.
• Complete a deferred install once Kafka is up: sudo pilot-agent configure-sudoers.

Troubleshooting

Deployment

Docker Compose

Deploy one agent per broker as a sidecar container:

services:
  kafka1:
    image: apache/kafka:latest
    container_name: pilot-kafka1
    volumes:
      - kafka1-data:/var/lib/kafka/data
 
  pilot:
    image: calinora/pilot:latest
    ports:
      - "8080:8080"
      - "9190:9190"
    environment:
      KAFKA_BOOTSTRAP_SERVERS: kafka1:9092
      PILOT_AGENT_ENABLED: "true"
      PILOT_AGENT_GRPC_PORT: "9190"
      PILOT_AGENT_BOOTSTRAP_TOKEN: "your-shared-secret"
 
  pilot-agent-1:
    image: calinora/pilot-agent:latest
    depends_on:
      - kafka1
      - pilot
    pid: "host"
    volumes_from:
      - kafka1:ro
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      AGENT_SERVER: "pilot:9190"
      AGENT_BROKER_ID: "1"
      AGENT_NODE_ID: "kafka1"
      AGENT_BOOTSTRAP_URL: "http://pilot:8080/api/v1/agents/bootstrap-cert"
      AGENT_BOOTSTRAP_TOKEN: "your-shared-secret"
      AGENT_DOCKER_CONTAINER: "pilot-kafka1"

Kubernetes

Deploy the agent as a sidecar container in the broker pod:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  template:
    spec:
      shareProcessNamespace: true
      containers:
        - name: kafka
          image: apache/kafka:latest
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
 
        - name: pilot-agent
          image: calinora/pilot-agent:latest
          env:
            - name: AGENT_SERVER
              value: "pilot.monitoring.svc:9190"
            - name: AGENT_BROKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['broker-id']
            - name: AGENT_BOOTSTRAP_URL
              value: "http://pilot.monitoring.svc:8080/api/v1/agents/bootstrap-cert"
            - name: AGENT_BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: pilot-agent-bootstrap
                  key: token
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
              readOnly: true
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 64Mi

Set shareProcessNamespace: true so the agent can observe the broker’s JVM process.

VM / Bare Metal

Deploy via Pilot UI

1. Go to the Brokers page and click Deploy Agent
2. Enter SSH connection details (host, username, key or password)
3. Optionally add a jumphost if the broker is behind a bastion
4. Set the broker ID and click Deploy

Pilot uploads the binary, generates mTLS certificates, installs a systemd service, and starts the agent. You don’t need to configure a bootstrap token manually - Pilot pushes the certificates and the master token over SSH so the agent can re-bootstrap automatically if the server’s CA is ever regenerated.

For multiple hosts, enable Deploy to multiple hosts in the dialog.

SSH host key verification

Agent deploys require host-key verification by default. Pilot will refuse to open an SSH connection unless one of the following is true for each hop:

1. Captured from Test Connection - running Test Connection from the dialog fetches the host public key and pins it on the form before you click Deploy. This is the standard flow.

2. Known fingerprints - paste a comma-separated list of OpenSSH SHA256 fingerprints into Known SSH Host Fingerprints under Advanced Settings. Collect them with:

   ssh-keyscan -t ed25519 broker-01 | ssh-keygen -lf -

   Fleet-wide fingerprints can be set once in PILOT_AGENT_SSH_KNOWN_FINGERPRINTS; Pilot merges them into any deploy that does not supply its own list.

3. Insecure opt-in - enable Allow insecure connection (skip SSH host key verification) for this deploy only. A red banner appears in the dialog and the audit log records insecureHostKeyBypass=true. Use only in throwaway environments.

Rule 1 wins over 2 wins over 3. Bulk deploys rely on 2 or 3, because a single TOFU-captured key cannot match every host.

Deploy via install script

curl -fsSL http://pilot:8080/api/v1/agents/install-script | sudo bash -s -- \
  --server pilot.internal:9190 \
  --pilot-url http://pilot:8080

After installation, configure /etc/pilot-agent/env and start:

sudo systemctl enable --now pilot-agent

Deploy via package manager

RPM (RHEL, Fedora, Amazon Linux):

sudo rpm --import https://downloads.calinora.io/calinora.gpg.key
 
sudo tee /etc/yum.repos.d/calinora.repo << 'EOF'
[calinora]
name=Calinora
baseurl=https://downloads.calinora.io/rpm/
enabled=1
gpgcheck=1
gpgkey=https://downloads.calinora.io/calinora.gpg.key
EOF
 
sudo dnf install calinora-pilot-agent

DEB (Ubuntu, Debian):

curl -fsSL https://downloads.calinora.io/calinora.gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/calinora-archive-keyring.gpg
 
echo "deb [signed-by=/usr/share/keyrings/calinora-archive-keyring.gpg] https://downloads.calinora.io/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/calinora.list
 
sudo apt update && sudo apt install calinora-pilot-agent

After installing via package manager, create the config file:

sudo tee /etc/pilot-agent/env << 'EOF'
AGENT_SERVER=pilot.internal:9190
AGENT_BOOTSTRAP_URL=https://pilot.internal:8080/api/v1/agents/bootstrap-cert
AGENT_BOOTSTRAP_TOKEN=your-shared-secret
# AGENT_BROKER_ID=1   # optional, auto-detected
EOF
 
sudo chmod 600 /etc/pilot-agent/env
sudo systemctl enable --now pilot-agent

The package installs:

• /usr/local/bin/pilot-agent - agent binary
• /usr/lib/systemd/system/pilot-agent.service - systemd unit
• /etc/pilot-agent/ - configuration directory

Direct download

All release artifacts are published to downloads.calinora.io under a versioned path. Substitute {version} with the Pilot version (matching the server), {arch} with amd64 or arm64, and {rpm-arch} with x86_64 or aarch64.

The RPM and DEB packages install both files under /usr/share/doc/calinora-pilot-agent/, and the Docker image carries them at /LICENSE and /THIRD_PARTY_NOTICES.

Raw binary, verify, and install manually:

VERSION=0.27.0
ARCH=amd64   # or arm64
 
curl -fsSLO "https://downloads.calinora.io/agent/v${VERSION}/pilot-agent-linux-${ARCH}"
curl -fsSLO "https://downloads.calinora.io/agent/v${VERSION}/pilot-agent-linux-${ARCH}.sha256"
sha256sum -c "pilot-agent-linux-${ARCH}.sha256"
 
sudo install -m 0755 "pilot-agent-linux-${ARCH}" /usr/local/bin/pilot-agent

Standalone RPM (no repo configured):

VERSION=0.27.0
RPM_ARCH=x86_64   # or aarch64
sudo rpm --import https://downloads.calinora.io/calinora.gpg.key
sudo dnf install "https://downloads.calinora.io/agent/v${VERSION}/calinora-pilot-agent-${VERSION}-1.${RPM_ARCH}.rpm"

Standalone DEB (no repo configured):

VERSION=0.27.0
ARCH=amd64   # or arm64
curl -fsSLO "https://downloads.calinora.io/agent/v${VERSION}/calinora-pilot-agent_${VERSION}_${ARCH}.deb"
sudo dpkg -i "calinora-pilot-agent_${VERSION}_${ARCH}.deb"

The install script also accepts a direct URL, useful when Pilot itself is unreachable from the target host:

sudo bash install-agent.sh \
  --server pilot.internal:9190 \
  --download-url "https://downloads.calinora.io/agent/v{version}/pilot-agent-linux-{arch}" \
  --version 0.27.0

The {version} and {arch} placeholders are substituted at download time.

For dev builds, replace the host with dev.downloads.calinora.io (internal use only).

Manual install (Ansible, Chef, Puppet, air-gapped)

For configuration-management systems, the path of least resistance is the DEB/RPM package: the postinstall hook creates the pilot-agent user, installs the systemd unit, writes the sudoers drop-in, and adds the agent to the Kafka log group. An Ansible playbook can reuse it directly:

- name: Install Calinora GPG key
  ansible.builtin.get_url:
    url: https://downloads.calinora.io/calinora.gpg.key
    dest: /usr/share/keyrings/calinora-archive-keyring.gpg
 
- name: Add Calinora APT repo
  ansible.builtin.apt_repository:
    repo: "deb [signed-by=/usr/share/keyrings/calinora-archive-keyring.gpg] https://downloads.calinora.io/deb stable main"
    filename: calinora
 
- name: Install pilot-agent
  ansible.builtin.apt:
    name: calinora-pilot-agent
    update_cache: true
  environment:
    PILOT_AGENT_KAFKA_UNIT: "{{ kafka_unit | default(omit) }}"
    PILOT_AGENT_KAFKA_GROUP: "{{ kafka_group | default(omit) }}"
 
- name: Configure agent
  ansible.builtin.copy:
    dest: /etc/pilot-agent/env
    owner: pilot-agent
    group: pilot-agent
    mode: "0600"
    content: |
      AGENT_SERVER={{ pilot_server }}:9190
      AGENT_BOOTSTRAP_URL=https://{{ pilot_server }}:8080/api/v1/agents/bootstrap-cert
      AGENT_BOOTSTRAP_TOKEN={{ pilot_bootstrap_token }}
 
- name: Enable pilot-agent
  ansible.builtin.systemd:
    name: pilot-agent
    enabled: true
    state: started

If the package path is unavailable (air-gapped, unsupported distro, or you want everything expressed as code), these are the exact steps install-agent.sh performs. Each is idempotent, so they map cleanly to Ansible modules or shell tasks with creates: guards:

1. Create the system user:

   sudo groupadd --system pilot-agent
   sudo useradd --system --gid pilot-agent --no-create-home \
     --shell /usr/sbin/nologin --home-dir /var/lib/pilot-agent pilot-agent
   sudo install -d -m 0750 -o pilot-agent -g pilot-agent /var/lib/pilot-agent

2. Install the binary (see Direct download for the URL pattern):

   sudo install -m 0755 pilot-agent-linux-amd64 /usr/local/bin/pilot-agent

3. Create the config directory and env file:

   sudo install -d -m 0750 -o pilot-agent -g pilot-agent /etc/pilot-agent
   sudo tee /etc/pilot-agent/env > /dev/null << 'EOF'
   AGENT_SERVER=pilot.internal:9190
   AGENT_BOOTSTRAP_URL=https://pilot.internal:8080/api/v1/agents/bootstrap-cert
   AGENT_BOOTSTRAP_TOKEN=your-shared-secret
   EOF
   sudo chown pilot-agent:pilot-agent /etc/pilot-agent/env
   sudo chmod 0600 /etc/pilot-agent/env

4. Install the systemd unit:

   sudo tee /usr/lib/systemd/system/pilot-agent.service > /dev/null << 'EOF'
   [Unit]
   Description=Calinora Pilot Agent
   After=network-online.target kafka.service
   Wants=network-online.target
    
   [Service]
   Type=simple
   ExecStart=/usr/local/bin/pilot-agent
   EnvironmentFile=-/etc/pilot-agent/env
   SyslogIdentifier=pilot-agent
   Restart=always
   RestartSec=5
   StartLimitInterval=0
    
   User=pilot-agent
   Group=pilot-agent
    
   ProtectSystem=strict
   ProtectHome=yes
   ProtectControlGroups=true
   ProtectKernelModules=true
   ProtectKernelTunables=yes
   PrivateTmp=true
   ReadWritePaths=/etc/pilot-agent /var/lib/pilot-agent
    
   [Install]
   WantedBy=multi-user.target
   EOF
   sudo systemctl daemon-reload

5. Wire broker lifecycle control (only needed if you want restart/stop/start from the UI). The agent writes its own sudoers drop-in and validates it with visudo -c, so there is no hand-rolled sudoers file to ship:

   sudo pilot-agent configure-sudoers             # auto-detects the running Kafka unit
   sudo pilot-agent configure-sudoers --unit kafka.service   # or pin the unit explicitly

   If Kafka is not running yet at install time, skip this step and run it after Kafka comes up.

6. Grant log-read access. Add the agent to the Kafka log-files group so it can tail broker logs:

   sudo usermod -aG kafka pilot-agent    # replace 'kafka' with your broker's log group

7. Start the service:

   sudo systemctl enable --now pilot-agent

All of these steps are idempotent and safe to re-run, which is the property Ansible/Chef/Puppet expect.

Security

Certificate Authentication (mTLS)

All agent-server communication uses mutual TLS:

1. Agents verify the server via the CA certificate received during enrollment
2. The server verifies agents by checking their client certificate against its CA
3. On first heartbeat, the server validates the certificate’s Common Name matches the agent’s claimed identity

Certificate Enrollment

When using auto-TLS (the default), agents generate a key pair locally and send only a Certificate Signing Request (CSR) to the server. The private key never leaves the agent.

Two token types are supported:

Note: For VM deployments via the Pilot UI, you don’t need to configure bootstrap tokens manually - Pilot pushes both certificates and the master token over SSH.

Auto-TLS

When no explicit certificate paths are set, Pilot auto-generates an ECDSA P-256 CA and server certificate stored in PILOT_AGENT_DATA_DIR (default: /data/agent-certs).

You have two options for the CA storage:

• Persistent volume - mount a volume at PILOT_AGENT_DATA_DIR so the CA and certificates survive container restarts. Agents keep their existing certificates and never need to re-enroll.
• Ephemeral storage - if Pilot restarts without persistent storage, it generates a new CA. Agents with a master token (AGENT_BOOTSTRAP_TOKEN) detect the TLS mismatch and re-bootstrap automatically. Agents enrolled with single-use tokens need manual re-enrollment.

Bring Your Own CA

If your organization has an existing CA or intermediate CA, you can place it in the auto-TLS data directory and Pilot will use it to sign agent certificates instead of generating its own:

# Place your CA key and cert in the data directory
cp corporate-ca.key /data/agent-certs/ca.key
cp corporate-ca.crt /data/agent-certs/ca.crt
chmod 600 /data/agent-certs/ca.key

Pilot loads these on startup and uses them for all CSR signing, bootstrap enrollment, and SSH deploy. Leave PILOT_AGENT_TLS_CERT, PILOT_AGENT_TLS_KEY, and PILOT_AGENT_TLS_CA unset - this is auto-TLS mode. Setting those variables disables the cert manager and Pilot will not sign any certificates.

Limitation: The CA key must be ECDSA P-256. RSA or other curve types are not supported for auto-TLS signing. If your corporate CA uses RSA, issue a P-256 intermediate CA signed by your root and place that in the data directory instead.

Fully Self-Managed Certificates

If you want to manage all certificates yourself (no auto-TLS, no bootstrap enrollment, no CSR signing by Pilot):

Pilot server:

PILOT_AGENT_TLS_CERT=/certs/server.crt
PILOT_AGENT_TLS_KEY=/certs/server.key
PILOT_AGENT_TLS_CA=/certs/ca.crt

Each agent:

AGENT_CERT=/certs/agent.crt
AGENT_KEY=/certs/agent.key
AGENT_CA=/certs/ca.crt

In this mode, you are responsible for generating and distributing all certificates. The bootstrap endpoint and SSH deploy certificate generation are disabled. Requirements: ECDSA P-256 or RSA 2048+ keys. Server cert must include the hostname agents connect to. All certs must chain to the same CA.

Development Mode

For local development without TLS, set PILOT_AGENT_INSECURE=true on the server and AGENT_INSECURE=true on agents.

Warning: Insecure mode disables all authentication and encryption. Never use in production.

Binary Distribution

All binaries are built for linux/amd64 and linux/arm64.

Corporate Proxy / Air-Gapped

For environments behind a corporate proxy, point Pilot to your mirror:

PILOT_AGENT_DOWNLOAD_URL="https://artifactory.corp.internal/calinora-remote/agent/v{version}/pilot-agent-linux-{arch}"

The {version} and {arch} placeholders are optional.

For air-gapped environments with no internet access:

PILOT_AGENT_BINARY_DIR="/opt/pilot-binaries"

Pre-place binaries named pilot-agent-linux-amd64 and pilot-agent-linux-arm64 in that directory.

Upgrading

Four upgrade methods:

1. Package manager - dnf update calinora-pilot-agent or apt upgrade calinora-pilot-agent, then restart
2. Via Pilot UI - agents with an available upgrade show an “Upgrade” button on the Brokers page
3. Via SSH - use the Deploy Agent dialog or POST /api/v1/agents/{brokerId}/upgrade
4. Via install script - re-run the install script

Pilot only serves binaries matching its own version. After upgrading Pilot, outdated agents are flagged as “Upgrade Available”.

Uninstalling

# DEB
sudo apt remove calinora-pilot-agent       # keeps config
sudo apt purge calinora-pilot-agent        # removes everything
 
# RPM
sudo dnf remove calinora-pilot-agent

Configuration Reference

Pilot Server

Agent

REST API

Agents

Lifecycle endpoints accept an optional body:

{
  "graceful": true,
  "timeoutSeconds": 300
}

Enrollment Tokens

SSH Deploy

Certificate Bootstrap

This endpoint uses token-based auth (not OAuth/OIDC) for machine-to-machine communication.

Troubleshooting

Agent not connecting

1. Verify PILOT_AGENT_ENABLED=true on the server and port 9190 is accessible
2. Check agent logs for certificate bootstrap errors
3. Ensure the agent can reach both HTTP (port 8080, for bootstrapping) and gRPC (port 9190)
4. For insecure mode, both PILOT_AGENT_INSECURE=true and AGENT_INSECURE=true must be set

Certificate re-bootstrap failing

1. Verify the agent has a valid AGENT_BOOTSTRAP_TOKEN
2. Single-use enrollment tokens cannot re-bootstrap - they are consumed on first use
3. Check server logs for rate-limiting messages (5 attempts/minute per IP)

No log directory metrics

1. The agent needs filesystem access to the broker’s data directories
2. In Docker, use volumes_from or explicit volume mounts
3. In Kubernetes, share the data volume with the sidecar container

Metrics showing zeroes

System metrics require Linux with /proc. On other platforms, the agent connects but OS metrics are zero.