---
title: "Pilot Agent - Pilot Docs"
description: "Deploy per-broker agents for OS-level metrics, log monitoring, and broker lifecycle management"
url: "https://docs.calinora.io/features/agents/"
---

# Pilot Agent

The Pilot Agent is a lightweight daemon deployed alongside each Kafka broker. It streams OS-level metrics, monitors Kafka log directories, and enables broker lifecycle management - all from the Pilot UI and API.

Agents are **optional**. Agentless monitoring via Kafka admin APIs continues to work unchanged. Agents add visibility that Kafka’s protocol cannot provide.

## Quick Start

### 1. Enable agents on the Pilot server

```bash
PILOT_AGENT_ENABLED=true
PILOT_AGENT_GRPC_PORT=9190
PILOT_AGENT_BOOTSTRAP_TOKEN=$(openssl rand -hex 32)   # save this value
```

### 2. Deploy agents

Choose the method that fits your environment:

| Environment | Method |
| - | - |
| **Docker Compose** | Add agent sidecar containers ([example below](https://docs.calinora.io/features/agents/#docker-compose)) |
| **Kubernetes** | Add agent sidecar to broker pods ([example below](https://docs.calinora.io/features/agents/#kubernetes)) |
| **VM / Bare metal** | Deploy via the Pilot UI, install script, or package manager ([details below](https://docs.calinora.io/features/agents/#vm--bare-metal)) |

### 3. Verify

Open **Agents** to inspect connected agents, their broker identity, version, status, and last-seen time. **Inspect host** opens the corresponding broker’s host telemetry.

***

## What Agents Provide

### System Metrics

Each agent reports the following every 5 seconds:

| Category | Metrics |
| - | - |
| **CPU** | Load average as percentage of available cores |
| **Memory** | Used and total bytes |
| **Disk usage** | Per-mount capacity, used, available, inode counts |
| **Kafka log dirs** | Per-directory capacity, used, available, usage percent |
| **Network I/O** | Bytes/packets sent and received, errors, drops |
| **Disk I/O** | Read/write bytes and ops, I/O time |
| **File descriptors** | Open and max FD count |

### Auto-Discovery

On connect, each agent scans the local Kafka installation and reports:

- **Runtime** - OS, kernel, architecture, container runtime (bare metal, Docker, Kubernetes)
- **Kafka** - vendor (Apache, Confluent, Strimzi, Cloudera), version, KRaft mode and role
- **Process** - PID, Java version, heap max, JMX port
- **Paths** - install directory, config file, log/data directories
- **Service** - systemd unit name, restart/stop/status commands
- **Log files** - known Kafka log files with paths and format

Log directories are auto-detected from the running broker’s `server.properties`. No manual path configuration needed.

Broker ID `0` is a valid explicit identity. Setting `--broker-id=0` or `AGENT_BROKER_ID=0` retains that identity. Omit both settings to enable automatic broker-ID detection.

### Log Forwarding

For regular log files, the agent starts at the end of the file and forwards new lines. It does not replay the existing file on startup. After rotation or truncation, it reads the replacement file from the beginning so lines written before the file is reopened are included.

### Broker Lifecycle Management

Agents can restart, stop, and start the Kafka broker on their host - triggered from the Pilot UI or API.

The agent uses whatever service manager it detected during discovery:

| Manager | How commands run |
| - | - |
| **systemd** | `systemctl restart/stop/start {unit}` |
| **exec** (fallback) | `SIGTERM` for stop, configured command for start |

**Safety controls:**

- Set `AGENT_ALLOW_RESTART=false` to disable lifecycle commands per agent
- Each command has a configurable timeout (default: 5 minutes)
- Graceful shutdown by default (pass `"graceful": false` for hard kill)
- All lifecycle endpoints require a valid Pilot license

**Override auto-detection** if needed:

| Variable | Description |
| - | - |
| `AGENT_KAFKA_SERVICE_UNIT` | systemd unit name (e.g., `my-kafka.service`) |
| `AGENT_KAFKA_START_CMD` | Custom start command |
| `AGENT_KAFKA_STOP_CMD` | Custom stop command |
| `AGENT_DOCKER_CONTAINER` | Docker container name for `docker stop/start/restart` |

The variables above are read by the agent process at runtime. Two additional variables are read by the installer (and by the RPM/DEB postinstall hook) to pre-seed non-root mode before the agent ever starts:

| Variable | Description |
| - | - |
| `PILOT_AGENT_KAFKA_UNIT` | Pre-seeds the Kafka unit name for `scripts/install-agent.sh` and the RPM/DEB postinstall hook. Used to render the sudoers drop-in. |
| `PILOT_AGENT_KAFKA_GROUP` | Pre-seeds the Kafka log-files group. The `pilot-agent` user is added to this group so it can tail logs. |

These two are install-time hints only; they are not read by the running agent.

### Prometheus Metrics

Every heartbeat field listed under [System Metrics](https://docs.calinora.io/features/agents/#system-metrics) is also re-exported on Pilot’s central `/metrics` endpoint as `pilot_agent_*` gauges, labelled with `broker_id` and `node_id` (hostname). Operators with an existing Prometheus stack can alert on broker-host CPU, memory, disk, log-dir, network, or disk-io without adding a per-broker scrape target. When an agent disconnects, every `pilot_agent_*` series for that broker is removed so a decommissioned host does not leave stale gauges.

See the [agent host metrics reference](https://docs.calinora.io/reference/metrics/#agent-host-metrics) for the full list and example PromQL queries.

## Privilege Model

The agent runs as a dedicated `pilot-agent` system user (no login shell, no home directory). All install paths (`scripts/install-agent.sh`, the UI deploy dialog, and the RPM/DEB package) set this up automatically:

- A narrow sudoers drop-in at `/etc/sudoers.d/99-pilot-agent` (mode `0440`, validated with `visudo -c`) lets the `pilot-agent` user run only `systemctl restart|stop|start|is-active|show -p MainPID|show -p ActiveState` against the detected Kafka unit. Nothing else. `NOPASSWD`, no shell access.
- The `pilot-agent` user is added to the Kafka log-files group so log tailing works unprivileged.
- The systemd unit keeps `ProtectSystem=strict`, `ProtectHome=yes`, `ProtectControlGroups`, `ProtectKernelModules`, `ProtectKernelTunables`, and `PrivateTmp=true`. `NoNewPrivileges=yes` is intentionally omitted because it is incompatible with `sudo`.
- Log forwarding, metrics collection, and discovery run unprivileged. Only broker lifecycle calls escalate through the narrow sudoers allow-list.

If Kafka is not running on the host at install time, sudoers is deferred: the agent installs cleanly, log forwarding works, and broker control is enabled later with `sudo pilot-agent configure-sudoers` (detection re-runs automatically).

### Install paths

#### scripts/install-agent.sh

```bash
curl -fsSL http://pilot:8080/api/v1/agents/install-script | sudo bash -s -- \
  --server pilot.internal:9190 \
  --pilot-url http://pilot:8080
```

Optional overrides for air-gapped hosts or custom Kafka packaging:

```bash
sudo bash install-agent.sh \
  --server pilot.internal:9190 \
  --kafka-unit confluent-kafka.service \
  --kafka-group cp-kafka
```

#### Pilot UI

Open the **Deploy Agent** dialog on the Brokers page and fill in the SSH target. The test-connection step detects the running Kafka unit and group on the target host and shows them as placeholders in Advanced Settings, so you can confirm what will be written to sudoers before pressing Deploy. If Kafka is not running yet on the target, the dialog flags this and the agent installs without sudoers; finish the setup later with `sudo pilot-agent configure-sudoers` once Kafka is up.

#### RPM/DEB

```bash
sudo apt-get install calinora-pilot-agent
sudo systemctl enable --now pilot-agent
```

If Kafka is not running yet on the host, pre-seed the unit name so the postinstall hook renders sudoers anyway:

```bash
sudo PILOT_AGENT_KAFKA_UNIT=confluent-kafka.service apt-get install calinora-pilot-agent
```

### Custom Kafka unit names

Detection is process-based, not string-based. The agent locates the running Kafka JVM by its universal main class `kafka.Kafka` (the Scala class name used by Apache Kafka, Confluent Platform, Aiven, MSK self-managed, and vendor forks) and resolves the owning systemd unit via `systemctl status <pid>`. Unit names like `kafka.service`, `confluent-kafka.service`, `cp-kafka.service`, or `our-platform-kafka.service` all work without configuration.

For air-gapped hosts where detection cannot run, or where the unit name is known ahead of time, pass `--kafka-unit <name>` to `install-agent.sh` or set `PILOT_AGENT_KAFKA_UNIT` before package install.

### Subcommands

The agent binary ships two operator subcommands for non-root mode:

| Command | Purpose | Flags |
| - | - | - |
| `pilot-agent detect` | Prints detected Kafka unit, group, log-dir, PID, and running user. Exits 0 even when nothing is found. | `--format=env\|json` |
| `pilot-agent configure-sudoers` | Writes `/etc/sudoers.d/99-pilot-agent` atomically, validated with `visudo -c`. Requires `EUID=0`. Without `--unit`, re-runs detection. | `--unit <name>`, `--user pilot-agent` |

Typical uses:

- Verify what the installer will detect: `pilot-agent detect --format=env`.
- Recover from a Kafka unit rename: `sudo pilot-agent configure-sudoers --unit new-name.service`.
- Complete a deferred install once Kafka is up: `sudo pilot-agent configure-sudoers`.

### Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| `sudo: a password is required` in agent logs when restarting a broker | Sudoers drop-in is missing or references a stale unit name | `sudo pilot-agent configure-sudoers` |
| `Permission denied` tailing Kafka logs | `pilot-agent` user is not a member of the Kafka log-files group | `sudo usermod -aG <kafka-group> pilot-agent && sudo systemctl restart pilot-agent` |
| Broker lifecycle fails after renaming the Kafka unit | Sudoers drop-in still points at the old unit name | `sudo pilot-agent configure-sudoers --unit <new-name>` |

***

## Deployment

### Docker Compose

Deploy one agent per broker as a sidecar container:

```yaml
services:
  kafka1:
    image: apache/kafka:latest
    container_name: pilot-kafka1
    volumes:
      - kafka1-data:/var/lib/kafka/data

  pilot:
    image: calinora/pilot:latest
    ports:
      - "8080:8080"
      - "9190:9190"
    environment:
      KAFKA_BOOTSTRAP_SERVERS: kafka1:9092
      PILOT_AGENT_ENABLED: "true"
      PILOT_AGENT_GRPC_PORT: "9190"
      PILOT_AGENT_BOOTSTRAP_TOKEN: "your-shared-secret"

  pilot-agent-1:
    image: calinora/pilot-agent:latest
    depends_on:
      - kafka1
      - pilot
    pid: "host"
    volumes_from:
      - kafka1:ro
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      AGENT_SERVER: "pilot:9190"
      AGENT_BROKER_ID: "1"
      AGENT_NODE_ID: "kafka1"
      AGENT_BOOTSTRAP_URL: "http://pilot:8080/api/v1/agents/bootstrap-cert"
      AGENT_BOOTSTRAP_TOKEN: "your-shared-secret"
      AGENT_DOCKER_CONTAINER: "pilot-kafka1"
```

| Option | Purpose |
| - | - |
| `pid: "host"` | Agent sees broker processes and survives broker container restarts. Required for lifecycle management. Set `AGENT_DOCKER_CONTAINER` so the agent knows which container to control |
| `pid: "service:kafka1"` | More isolated but the agent stops when the broker stops. Use for monitoring-only deployments |
| `volumes_from: - kafka1:ro` | Agent sees log directories at the same paths as the broker |
| Docker socket mount | Enables `docker stop/start/restart` for container lifecycle management |

### Kubernetes

Deploy the agent as a sidecar container in the broker pod:

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  template:
    spec:
      shareProcessNamespace: true
      containers:
        - name: kafka
          image: apache/kafka:latest
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data

        - name: pilot-agent
          image: calinora/pilot-agent:latest
          env:
            - name: AGENT_SERVER
              value: "pilot.monitoring.svc:9190"
            - name: AGENT_BROKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['broker-id']
            - name: AGENT_BOOTSTRAP_URL
              value: "http://pilot.monitoring.svc:8080/api/v1/agents/bootstrap-cert"
            - name: AGENT_BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: pilot-agent-bootstrap
                  key: token
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
              readOnly: true
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 64Mi
```

Set `shareProcessNamespace: true` so the agent can observe the broker’s JVM process.

### VM / Bare Metal

#### Deploy via Pilot UI

1. Go to the **Brokers** page and click **Deploy Agent**
2. Enter SSH connection details (host, username, key or password)
3. Optionally add a jumphost if the broker is behind a bastion
4. Set the broker ID and click **Deploy**

Pilot uploads the binary, generates mTLS certificates, installs a systemd service, and starts the agent. You don’t need to configure a bootstrap token manually - Pilot pushes the certificates and the master token over SSH so the agent can re-bootstrap automatically if the server’s CA is ever regenerated.

For multiple hosts, enable **Deploy to multiple hosts** in the dialog.

#### SSH host key verification

Agent deploys require host-key verification by default. Pilot will refuse to open an SSH connection unless one of the following is true for each hop:

1. **Captured from Test Connection** - running **Test Connection** from the dialog fetches the host public key and pins it on the form before you click Deploy. This is the standard flow.

2. **Known fingerprints** - paste a comma-separated list of OpenSSH SHA256 fingerprints into **Known SSH Host Fingerprints** under **Advanced Settings**. Collect them with:

   ```bash
   ssh-keyscan -t ed25519 broker-01 | ssh-keygen -lf -
   ```

   Fleet-wide fingerprints can be set once in [PILOT_AGENT_SSH_KNOWN_FINGERPRINTS](https://docs.calinora.io/reference/environment-variables/#pilot-agent); Pilot merges them into any deploy that does not supply its own list.

3. **Insecure opt-in** - enable **Allow insecure connection (skip SSH host key verification)** for this deploy only. A red banner appears in the dialog and the audit log records `insecureHostKeyBypass=true`. Use only in throwaway environments.

Rule 1 wins over 2 wins over 3. Bulk deploys rely on 2 or 3, because a single TOFU-captured key cannot match every host.

#### Deploy via install script

```bash
curl -fsSL http://pilot:8080/api/v1/agents/install-script | sudo bash -s -- \
  --server pilot.internal:9190 \
  --pilot-url http://pilot:8080
```

After installation, configure `/etc/pilot-agent/env` and start:

```bash
sudo systemctl enable --now pilot-agent
```

#### Deploy via package manager

**RPM (RHEL, Fedora, Amazon Linux):**

```bash
sudo rpm --import https://downloads.calinora.io/calinora.gpg.key

sudo tee /etc/yum.repos.d/calinora.repo << 'EOF'
[calinora]
name=Calinora
baseurl=https://downloads.calinora.io/rpm/
enabled=1
gpgcheck=1
gpgkey=https://downloads.calinora.io/calinora.gpg.key
EOF

sudo dnf install calinora-pilot-agent
```

**DEB (Ubuntu, Debian):**

```bash
curl -fsSL https://downloads.calinora.io/calinora.gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/calinora-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/calinora-archive-keyring.gpg] https://downloads.calinora.io/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/calinora.list

sudo apt update && sudo apt install calinora-pilot-agent
```

After installing via package manager, create the config file:

```bash
sudo tee /etc/pilot-agent/env << 'EOF'
AGENT_SERVER=pilot.internal:9190
AGENT_BOOTSTRAP_URL=https://pilot.internal:8080/api/v1/agents/bootstrap-cert
AGENT_BOOTSTRAP_TOKEN=your-shared-secret
# AGENT_BROKER_ID=1   # optional, auto-detected
EOF

sudo chmod 600 /etc/pilot-agent/env
sudo systemctl enable --now pilot-agent
```

The package installs:

- `/usr/local/bin/pilot-agent` - agent binary
- `/usr/lib/systemd/system/pilot-agent.service` - systemd unit
- `/etc/pilot-agent/` - configuration directory

#### Direct download

All release artifacts are published to `downloads.calinora.io` under a versioned path. Substitute `{version}` with the Pilot version (matching the server), `{arch}` with `amd64` or `arm64`, and `{rpm-arch}` with `x86_64` or `aarch64`.

| Artifact | URL |
| - | - |
| Raw binary | `https://downloads.calinora.io/agent/v{version}/pilot-agent-linux-{arch}` |
| SHA256 checksum | `https://downloads.calinora.io/agent/v{version}/pilot-agent-linux-{arch}.sha256` |
| RPM package | `https://downloads.calinora.io/agent/v{version}/calinora-pilot-agent-{version}-1.{rpm-arch}.rpm` |
| DEB package | `https://downloads.calinora.io/agent/v{version}/calinora-pilot-agent_{version}_{arch}.deb` |
| GPG signing key | `https://downloads.calinora.io/calinora.gpg.key` |
| License and third-party notices | `https://downloads.calinora.io/agent/v{version}/LICENSE`, `https://downloads.calinora.io/agent/v{version}/THIRD_PARTY_NOTICES` |

The RPM and DEB packages install both files under `/usr/share/doc/calinora-pilot-agent/`, and the Docker image carries them at `/LICENSE` and `/THIRD_PARTY_NOTICES`.

Raw binary, verify, and install manually:

```bash
VERSION=0.27.0
ARCH=amd64   # or arm64

curl -fsSLO "https://downloads.calinora.io/agent/v${VERSION}/pilot-agent-linux-${ARCH}"
curl -fsSLO "https://downloads.calinora.io/agent/v${VERSION}/pilot-agent-linux-${ARCH}.sha256"
sha256sum -c "pilot-agent-linux-${ARCH}.sha256"

sudo install -m 0755 "pilot-agent-linux-${ARCH}" /usr/local/bin/pilot-agent
```

Standalone RPM (no repo configured):

```bash
VERSION=0.27.0
RPM_ARCH=x86_64   # or aarch64
sudo rpm --import https://downloads.calinora.io/calinora.gpg.key
sudo dnf install "https://downloads.calinora.io/agent/v${VERSION}/calinora-pilot-agent-${VERSION}-1.${RPM_ARCH}.rpm"
```

Standalone DEB (no repo configured):

```bash
VERSION=0.27.0
ARCH=amd64   # or arm64
curl -fsSLO "https://downloads.calinora.io/agent/v${VERSION}/calinora-pilot-agent_${VERSION}_${ARCH}.deb"
sudo dpkg -i "calinora-pilot-agent_${VERSION}_${ARCH}.deb"
```

The install script also accepts a direct URL, useful when Pilot itself is unreachable from the target host:

```bash
sudo bash install-agent.sh \
  --server pilot.internal:9190 \
  --download-url "https://downloads.calinora.io/agent/v{version}/pilot-agent-linux-{arch}" \
  --version 0.27.0
```

The `{version}` and `{arch}` placeholders are substituted at download time.

For dev builds, replace the host with `dev.downloads.calinora.io` (internal use only).

#### Manual install (Ansible, Chef, Puppet, air-gapped)

For configuration-management systems, the path of least resistance is the DEB/RPM package: the postinstall hook creates the `pilot-agent` user, installs the systemd unit, writes the sudoers drop-in, and adds the agent to the Kafka log group. An Ansible playbook can reuse it directly:

```yaml
- name: Install Calinora GPG key
  ansible.builtin.get_url:
    url: https://downloads.calinora.io/calinora.gpg.key
    dest: /usr/share/keyrings/calinora-archive-keyring.gpg

- name: Add Calinora APT repo
  ansible.builtin.apt_repository:
    repo: "deb [signed-by=/usr/share/keyrings/calinora-archive-keyring.gpg] https://downloads.calinora.io/deb stable main"
    filename: calinora

- name: Install pilot-agent
  ansible.builtin.apt:
    name: calinora-pilot-agent
    update_cache: true
  environment:
    PILOT_AGENT_KAFKA_UNIT: "{{ kafka_unit | default(omit) }}"
    PILOT_AGENT_KAFKA_GROUP: "{{ kafka_group | default(omit) }}"

- name: Configure agent
  ansible.builtin.copy:
    dest: /etc/pilot-agent/env
    owner: pilot-agent
    group: pilot-agent
    mode: "0600"
    content: |
      AGENT_SERVER={{ pilot_server }}:9190
      AGENT_BOOTSTRAP_URL=https://{{ pilot_server }}:8080/api/v1/agents/bootstrap-cert
      AGENT_BOOTSTRAP_TOKEN={{ pilot_bootstrap_token }}

- name: Enable pilot-agent
  ansible.builtin.systemd:
    name: pilot-agent
    enabled: true
    state: started
```

If the package path is unavailable (air-gapped, unsupported distro, or you want everything expressed as code), these are the exact steps `install-agent.sh` performs. Each is idempotent, so they map cleanly to Ansible modules or shell tasks with `creates:` guards:

1. **Create the system user:**

   ```bash
   sudo groupadd --system pilot-agent
   sudo useradd --system --gid pilot-agent --no-create-home \
     --shell /usr/sbin/nologin --home-dir /var/lib/pilot-agent pilot-agent
   sudo install -d -m 0750 -o pilot-agent -g pilot-agent /var/lib/pilot-agent
   ```

2. **Install the binary** (see [Direct download](https://docs.calinora.io/features/agents/#direct-download) for the URL pattern):

   ```bash
   sudo install -m 0755 pilot-agent-linux-amd64 /usr/local/bin/pilot-agent
   ```

3. **Create the config directory and env file:**

   ```bash
   sudo install -d -m 0750 -o pilot-agent -g pilot-agent /etc/pilot-agent
   sudo tee /etc/pilot-agent/env > /dev/null << 'EOF'
   AGENT_SERVER=pilot.internal:9190
   AGENT_BOOTSTRAP_URL=https://pilot.internal:8080/api/v1/agents/bootstrap-cert
   AGENT_BOOTSTRAP_TOKEN=your-shared-secret
   EOF
   sudo chown pilot-agent:pilot-agent /etc/pilot-agent/env
   sudo chmod 0600 /etc/pilot-agent/env
   ```

4. **Install the systemd unit:**

   ```bash
   sudo tee /usr/lib/systemd/system/pilot-agent.service > /dev/null << 'EOF'
   [Unit]
   Description=Calinora Pilot Agent
   After=network-online.target kafka.service
   Wants=network-online.target

   [Service]
   Type=simple
   ExecStart=/usr/local/bin/pilot-agent
   EnvironmentFile=-/etc/pilot-agent/env
   SyslogIdentifier=pilot-agent
   Restart=always
   RestartSec=5
   StartLimitInterval=0

   User=pilot-agent
   Group=pilot-agent

   ProtectSystem=strict
   ProtectHome=yes
   ProtectControlGroups=true
   ProtectKernelModules=true
   ProtectKernelTunables=yes
   PrivateTmp=true
   ReadWritePaths=/etc/pilot-agent /var/lib/pilot-agent

   [Install]
   WantedBy=multi-user.target
   EOF
   sudo systemctl daemon-reload
   ```

5. **Wire broker lifecycle control** (only needed if you want restart/stop/start from the UI). The agent writes its own sudoers drop-in and validates it with `visudo -c`, so there is no hand-rolled sudoers file to ship:

   ```bash
   sudo pilot-agent configure-sudoers             # auto-detects the running Kafka unit
   sudo pilot-agent configure-sudoers --unit kafka.service   # or pin the unit explicitly
   ```

   If Kafka is not running yet at install time, skip this step and run it after Kafka comes up.

6. **Grant log-read access.** Add the agent to the Kafka log-files group so it can tail broker logs:

   ```bash
   sudo usermod -aG kafka pilot-agent    # replace 'kafka' with your broker's log group
   ```

7. **Start the service:**

   ```bash
   sudo systemctl enable --now pilot-agent
   ```

All of these steps are idempotent and safe to re-run, which is the property Ansible/Chef/Puppet expect.

***

## Security

### Certificate Authentication (mTLS)

All agent-server communication uses mutual TLS:

1. Agents verify the server via the CA certificate received during enrollment
2. The server verifies agents by checking their client certificate against its CA
3. On first heartbeat, the server validates the certificate’s Common Name matches the agent’s claimed identity

### Certificate Enrollment

When using auto-TLS (the default), agents generate a key pair locally and send only a Certificate Signing Request (CSR) to the server. **The private key never leaves the agent.**

Two token types are supported:

| Token Type | Lifetime | Use Case |
| - | - | - |
| **Master token** | Persistent | Automation (Terraform, Ansible, docker-compose). Set via `PILOT_AGENT_BOOTSTRAP_TOKEN` |
| **Enrollment tokens** | Single-use, time-limited (default 1h) | One-off enrollments via UI or API |

> **Note:** For VM deployments via the Pilot UI, you don’t need to configure bootstrap tokens manually - Pilot pushes both certificates and the master token over SSH.

### Auto-TLS

When no explicit certificate paths are set, Pilot auto-generates an ECDSA P-256 CA and server certificate stored in `PILOT_AGENT_DATA_DIR` (default: `/data/agent-certs`).

You have two options for the CA storage:

- **Persistent volume** - mount a volume at `PILOT_AGENT_DATA_DIR` so the CA and certificates survive container restarts. Agents keep their existing certificates and never need to re-enroll.
- **Ephemeral storage** - if Pilot restarts without persistent storage, it generates a new CA. Agents with a master token (`AGENT_BOOTSTRAP_TOKEN`) detect the TLS mismatch and re-bootstrap automatically. Agents enrolled with single-use tokens need manual re-enrollment.

### Bring Your Own CA

If your organization has an existing CA or intermediate CA, you can place it in the auto-TLS data directory and Pilot will use it to sign agent certificates instead of generating its own:

```bash
# Place your CA key and cert in the data directory
cp corporate-ca.key /data/agent-certs/ca.key
cp corporate-ca.crt /data/agent-certs/ca.crt
chmod 600 /data/agent-certs/ca.key
```

Pilot loads these on startup and uses them for all CSR signing, bootstrap enrollment, and SSH deploy. Leave `PILOT_AGENT_TLS_CERT`, `PILOT_AGENT_TLS_KEY`, and `PILOT_AGENT_TLS_CA` unset - this is auto-TLS mode. Setting those variables disables the cert manager and Pilot will not sign any certificates.

> **Limitation:** The CA key must be ECDSA P-256. RSA or other curve types are not supported for auto-TLS signing. If your corporate CA uses RSA, issue a P-256 intermediate CA signed by your root and place that in the data directory instead.

### Fully Self-Managed Certificates

If you want to manage all certificates yourself (no auto-TLS, no bootstrap enrollment, no CSR signing by Pilot):

**Pilot server:**

```bash
PILOT_AGENT_TLS_CERT=/certs/server.crt
PILOT_AGENT_TLS_KEY=/certs/server.key
PILOT_AGENT_TLS_CA=/certs/ca.crt
```

**Each agent:**

```bash
AGENT_CERT=/certs/agent.crt
AGENT_KEY=/certs/agent.key
AGENT_CA=/certs/ca.crt
```

In this mode, you are responsible for generating and distributing all certificates. The bootstrap endpoint and SSH deploy certificate generation are disabled. Requirements: ECDSA P-256 or RSA 2048+ keys. Server cert must include the hostname agents connect to. All certs must chain to the same CA.

### Development Mode

For local development without TLS, set `PILOT_AGENT_INSECURE=true` on the server and `AGENT_INSECURE=true` on agents.

> **Warning:** Insecure mode disables all authentication and encryption. Never use in production.

***

## Binary Distribution

| Channel | Use Case |
| - | - |
| **Docker Hub** | `docker pull calinora/pilot-agent:latest` |
| **RPM** | `dnf install calinora-pilot-agent` (repo at `https://downloads.calinora.io/rpm/`) |
| **DEB** | `apt install calinora-pilot-agent` (repo at `https://downloads.calinora.io/deb`) |
| **Direct download** | `https://downloads.calinora.io/agent/v{version}/pilot-agent-linux-{arch}` (see [Direct download](https://docs.calinora.io/features/agents/#direct-download)) |
| **Pilot UI / install script** | Pulls from the URL above automatically |

All binaries are built for `linux/amd64` and `linux/arm64`.

### Corporate Proxy / Air-Gapped

For environments behind a corporate proxy, point Pilot to your mirror:

```bash
PILOT_AGENT_DOWNLOAD_URL="https://artifactory.corp.internal/calinora-remote/agent/v{version}/pilot-agent-linux-{arch}"
```

The `{version}` and `{arch}` placeholders are optional.

For air-gapped environments with no internet access:

```bash
PILOT_AGENT_BINARY_DIR="/opt/pilot-binaries"
```

Pre-place binaries named `pilot-agent-linux-amd64` and `pilot-agent-linux-arm64` in that directory.

### Upgrading

Four upgrade methods:

1. **Package manager** - `dnf update calinora-pilot-agent` or `apt upgrade calinora-pilot-agent`, then restart
2. **Via Pilot UI** - agents with an available upgrade show an “Upgrade” button on the Brokers page
3. **Via SSH** - use the Deploy Agent dialog or `POST /api/v1/agents/{brokerId}/upgrade`
4. **Via install script** - re-run the install script

Pilot only serves binaries matching its own version. After upgrading Pilot, outdated agents are flagged as “Upgrade Available”.

### Uninstalling

```bash
# DEB
sudo apt remove calinora-pilot-agent       # keeps config
sudo apt purge calinora-pilot-agent        # removes everything

# RPM
sudo dnf remove calinora-pilot-agent
```

***

## Configuration Reference

### Pilot Server

| Variable | Default | Description |
| - | - | - |
| `PILOT_AGENT_ENABLED` | `false` | Enable the agent gRPC server |
| `PILOT_AGENT_GRPC_PORT` | `9190` | gRPC port |
| `PILOT_AGENT_BOOTSTRAP_TOKEN` | **required** | Master token for enrollment. Generate with `openssl rand -hex 32` |
| `PILOT_AGENT_DATA_DIR` | `/data/agent-certs` | Directory for auto-generated certificates |
| `PILOT_AGENT_TLS_SANS` | `""` | Extra DNS/IPs for auto-generated server cert (comma-separated) |
| `PILOT_AGENT_TLS_CERT` | (auto) | Server TLS certificate (disables auto-TLS) |
| `PILOT_AGENT_TLS_KEY` | (auto) | Server TLS private key |
| `PILOT_AGENT_TLS_CA` | (auto) | CA for verifying agents |
| `PILOT_AGENT_INSECURE` | `false` | Allow gRPC without TLS (dev only) |
| `PILOT_DEPLOY_ALLOW_HTTP` | `false` | Allow SSH deploy endpoints over plain HTTP |
| `PILOT_AGENT_DOWNLOAD_URL` | `https://downloads.calinora.io/...` | Upstream binary URL (supports `{version}` and `{arch}` placeholders) |
| `PILOT_AGENT_BINARY_DIR` | `""` | Local binary directory for air-gapped mode |
| `PILOT_AGENT_BINARY_CACHE_DIR` | `{data-dir}/binaries` | Cache directory for downloaded binaries |

### Agent

| Flag | Environment Variable | Description |
| - | - | - |
| `--server` | `AGENT_SERVER` | Pilot gRPC address (required) |
| `--broker-id` | `AGENT_BROKER_ID` | Kafka broker ID (auto-detected if omitted) |
| `--node-id` | `AGENT_NODE_ID` | Node identifier (defaults to hostname) |
| `--cert` | `AGENT_CERT` | mTLS client certificate |
| `--key` | `AGENT_KEY` | mTLS client private key |
| `--ca` | `AGENT_CA` | CA certificate for server verification |
| `--bootstrap-url` | `AGENT_BOOTSTRAP_URL` | Certificate bootstrap URL |
| `--bootstrap-token` | `AGENT_BOOTSTRAP_TOKEN` | Bootstrap token (master or single-use) |
| `--insecure` | `AGENT_INSECURE` | Allow plaintext gRPC (dev only) |
| `--log-dirs` | `AGENT_LOG_DIRS` | Kafka log directories (auto-detected if omitted) |
| | `AGENT_KAFKA_SERVICE_UNIT` | Override systemd unit name |
| | `AGENT_KAFKA_START_CMD` | Override broker start command |
| | `AGENT_KAFKA_STOP_CMD` | Override broker stop command |
| | `AGENT_ALLOW_RESTART` | Permit lifecycle commands (default: `true`) |
| | `AGENT_DOCKER_CONTAINER` | Docker container name for lifecycle commands |

***

## REST API

### Agents

| Method | Endpoint | Description |
| - | - | - |
| `GET` | `/api/v1/agents` | List all agents with status and metrics |
| `GET` | `/api/v1/agents/{brokerId}` | Get single agent details and discovery info |
| `POST` | `/api/v1/agents/{brokerId}/restart` | Restart the Kafka broker (license required) |
| `POST` | `/api/v1/agents/{brokerId}/stop` | Stop the Kafka broker (license required) |
| `POST` | `/api/v1/agents/{brokerId}/start` | Start the Kafka broker (license required) |
| `POST` | `/api/v1/agents/{brokerId}/upgrade` | Upgrade agent binary via SSH (license required) |
| `GET` | `/api/v1/agents/binary?arch=amd64` | Download agent binary |

Lifecycle endpoints accept an optional body:

```json
{
  "graceful": true,
  "timeoutSeconds": 300
}
```

### Enrollment Tokens

| Method | Endpoint | Description |
| - | - | - |
| `POST` | `/api/v1/agents/enrollment-tokens` | Create a single-use enrollment token (license required) |
| `GET` | `/api/v1/agents/enrollment-tokens` | List active tokens |
| `DELETE` | `/api/v1/agents/enrollment-tokens` | Revoke a token (license required) |

### SSH Deploy

| Method | Endpoint | Description |
| - | - | - |
| `POST` | `/api/v1/agents/deploy` | Deploy agent to a VM via SSH (license required) |
| `POST` | `/api/v1/agents/deploy/bulk` | Deploy to multiple VMs sequentially (license required) |
| `POST` | `/api/v1/agents/deploy/test` | Test SSH connectivity and auto-detect broker info |

### Certificate Bootstrap

| Method | Endpoint | Description |
| - | - | - |
| `POST` | `/api/v1/agents/bootstrap-cert` | Exchange CSR + token for signed certificate. Rate limited to 5/min per IP |

This endpoint uses token-based auth (not OAuth/OIDC) for machine-to-machine communication.

***

## Troubleshooting

### Agent not connecting

1. Verify `PILOT_AGENT_ENABLED=true` on the server and port 9190 is accessible
2. Check agent logs for certificate bootstrap errors
3. Ensure the agent can reach both HTTP (port 8080, for bootstrapping) and gRPC (port 9190)
4. For insecure mode, both `PILOT_AGENT_INSECURE=true` and `AGENT_INSECURE=true` must be set

### Certificate re-bootstrap failing

1. Verify the agent has a valid `AGENT_BOOTSTRAP_TOKEN`
2. Single-use enrollment tokens cannot re-bootstrap - they are consumed on first use
3. Check server logs for rate-limiting messages (5 attempts/minute per IP)

### No log directory metrics

1. The agent needs filesystem access to the broker’s data directories
2. In Docker, use `volumes_from` or explicit volume mounts
3. In Kubernetes, share the data volume with the sidecar container

### Metrics showing zeroes

System metrics require Linux with `/proc`. On other platforms, the agent connects but OS metrics are zero.