Consumer Groups

Pilot monitors Kafka consumer groups in the background and provides visibility into group state, membership, and lag.

Background Collection

Consumer group data is collected automatically:

PILOT_CONSUMER_GROUP_COLLECTION_ENABLED=true   # Enable (default: true)
PILOT_CONSUMER_GROUP_COLLECTION_INTERVAL=15s   # Collection interval

Features

Group Overview

View all consumer groups with their state (Stable, Rebalancing, Empty, Dead), member count, and assigned topics.

Lag Monitoring

Per-partition lag metrics for each consumer group, showing the difference between the latest produced offset and the committed consumer offset.

Consumption Rates and Source Times

Consumption rates measure committed-offset advancement, not current fetch or processing activity. Pilot smooths recent progress with an exponential average whose time constant is 300 seconds. A positive estimate can remain after all members stop: once the raw input becomes zero, about 37% remains after five minutes and 5% after fifteen minutes. Commit intervals also affect the observed rate.

The Topics page uses the maximum group rate for each partition, smooths those values, and sums them across the topic. It does not sum all consumer groups. Consumer byte rates use estimated record sizes, not measured network traffic.

Lag and consumption summaries carry observationStartedAt and observationCompletedAt for the last successful Kafka collection. Failed collection does not advance them. Use those source times when comparing against another exporter rather than assuming the HTTP response time is the observation time. The same bounds are exported as Prometheus metrics.

Offset Reset

Reset consumer group offsets with the earliest retained offset, current log end, a date and time, or exact offsets. A mutation license is required. Stop the group first and wait for Empty or Dead state with no active members.

The dialog provides a review before applying:

1. Select all partitions or a subset. Filters do not remove hidden selected partitions from the operation.
2. Preview exact targets and inspect the Replay, Skip and Unchanged counts. Targets must lie within each partition’s retained range.
3. Acknowledge the processing effect. Before submission, Pilot rechecks group eligibility, retained ranges and current offsets. Changed offsets require a new review.
4. Compare the result with freshly observed committed offsets. Request acceptance and observed target matches are shown separately. If observation fails, keep the group stopped and refresh the result before another request.

Earlier offsets can repeat processing; later offsets can skip records for this group. Moving an offset cannot undo application side effects or restore records deleted by retention.

Reversing an Offset Reset

The Audit Log does not offer automatic reversal of group offset resets. The event records previous offsets, requested targets and the request outcome. The dialog’s later offset observations are not persisted as verified completion in that event. An audit warning about reversal therefore does not mean the original reset failed.

Kafka can move offsets backwards. To restore previous positions, stop the group and review a new exact-offset reset against its current positions and retained ranges. A previous position below the retained log start is no longer recoverable. See Audit Reversal Limits.

Group Deletion

Delete consumer groups that are no longer needed (requires license). The group must be in an Empty or Dead state.

API Endpoints