Skip to Content
GuidesYour First Rebalance

Your First Rebalance

This guide walks through generating a rebalancing proposal, reviewing it, and applying it to your cluster.

Prerequisites

  • Pilot is running and connected to your Kafka cluster
  • You have a valid license for applying proposals
  • Your cluster has some partition imbalance (uneven leader or replica distribution)

Step 1: Check Current Balance

Open the Pilot dashboard at http://localhost:8080. The overview page shows Pilot’s balance status and, per broker, stored data, leader and replica counts and the message rate. Proposal review shows the Balance Summary with the broker farthest from its fair share of each load:

  • Leaders and followers - partition counts per broker
  • Disk - stored bytes per broker
  • Producer and consumer rates - messages and bytes per second per broker

You can also check via the API:

curl -s http://localhost:8080/api/v1/cluster/comprehensive?summary=true | jq '.data'

Step 2: Generate a Proposal

Open Proposal review in the UI to see the latest background proposal. To request a fresh calculation through the API:

curl -X POST http://localhost:8080/api/v1/proposals/generate \
  -H "Content-Type: application/json" \
  -d '{"immediate": true}' | jq '.'

Pilot runs the multi-phase proposal pipeline and returns a proposal with:

  • Proposal ID - unique identifier
  • Expected effect - per load, the broker farthest from its fair share now and after the plan (fairShare)
  • Reassignments - list of partition moves with source and destination brokers
  • Cost - total movement cost

Step 3: Review the Proposal

Examine the proposed moves carefully:

# Get the full proposal
curl -s http://localhost:8080/api/v1/proposals/{proposalId} | jq '.'

Things to check:

  • Number of moves - is it reasonable for your cluster size?
  • Cross-rack moves - more expensive than same-rack moves
  • Balance improvement - how close to its fair share does the plan bring each load’s farthest broker?
  • Affected topics - are critical topics being moved?

Also review generation blockers and loads at their limit. No proposed moves does not by itself mean the cluster is balanced. Some imbalances require changing partition counts, replication factors, or capacity. Others are too small to justify more movement. When correctness repairs are needed, Pilot proposes those first and reassesses optional balancing afterward.

The generate response includes metadata.applicationAllowed and a message explaining any application blocker. Ordinary balancing requires ready measurements, a broker that stays more than 10% from its fair share and a plan that stays worth applying across three fresh measurements and was checked within the last two minutes. Pilot evaluates the same final plan against those measurements; separate calculations do not need to propose matching moves. If the plan changes during review, the review says The plan changed. Review the new plan. See Apply-Time Safety Gating for the full rules.

Monitoring means Pilot is keeping the current placement while checking for sustained benefit. The UI shows current metrics and any observation progress, without a move table or Apply action. It does not mean that every metric is balanced. Settling after moves shows what Pilot is waiting for after recent moves and, when it can be estimated, the remaining time before optional balancing can be considered again. See Settling After Moves.

Step 4: (Optional) Simulate

Before applying, you can run a what-if simulation to understand the impact:

curl -X POST http://localhost:8080/api/v1/what-if/simulate \
  -H "Content-Type: application/json" \
  -d '{}'

Step 5: Apply the Proposal

When the proposal is ready to apply and you have reviewed it, submit it:

curl -X POST http://localhost:8080/api/v1/proposals/{proposalId}/apply

Pilot will:

  1. Set replication throttles on the affected brokers
  2. Submit partition reassignments as broker capacity becomes available
  3. Monitor progress and clear the throttles when the moves finish

Step 6: Monitor Progress

Open Execution in Pilot. The reassignment monitor shows:

  • Active moves - partitions currently being transferred
  • Pending moves - queued for execution
  • Completed moves - successfully finished
  • Failed and cancelled moves - outcomes that require review

The progress percentage counts completed assignments, not bytes copied. Transfer ETA is unavailable, and pending moves still count as unfinished work. If you cancel, inspect the final placement: completed assignments remain changed, and cancellation of active work may remain unconfirmed.

You can also check via the API:

curl -s http://localhost:8080/api/v1/reassignments | jq '.'

Step 7: Verify Balance

After all reassignments complete, check the balance again:

curl -s http://localhost:8080/api/v1/cluster/comprehensive?summary=true | jq '.data'

Compare the result with the proposal’s projected values and the Balance Summary. Allow sampled metrics to settle after the moves. A load at its limit can leave a broker more than 10% from its fair share; review the remaining advisories rather than treating every residual difference as a failed reassignment.

Tips

  • Start with a higher threshold - set PILOT_BALANCE_THRESHOLD=10 initially to limit the number of moves
  • Use conservative throttles - keep PILOT_THROTTLE_RATE_MB at 50 MB/s for your first rebalance
  • Exclude internal topics - set PILOT_EXCLUDE_TOPICS="__consumer_offsets,__transaction_state"
  • Rebalance during low-traffic periods - less impact on producers and consumers
Last updated on
OverviewMonitoring Setup