Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Health and circuits

poise-health provides executor-neutral health state. It does not send network probes, sleep, spawn tasks, or decide which application result is a failure. Those choices remain with adapters and protocol code.

Compose independent signals

HealthChecked<Backend, Health> wraps an existing candidate and adds a HealthSignal. Candidate identity, configured weight, topology metadata, and load delegate to the inner backend. Eligibility requires both the inner candidate and health signal to permit selection.

This composition preserves administrative intent:

  • a draining backend does not become selectable because its probe is healthy;
  • an unavailable backend does not become ready because a circuit closes;
  • nested health wrappers can express multiple independent gates.

Passive circuit states

Closed ── failure threshold ──▶ Open
  ▲                              │
  │                              │ cooldown elapsed
  │                              ▼
  └── success threshold ───── Half-open
                                 │
                                 └── probe failure ──▶ Open

CircuitConfig controls:

  • consecutive failures required to open;
  • open cooldown duration;
  • successful half-open probes required to close;
  • maximum concurrent half-open permits.

Configuration rejects a zero open duration. Large durations remain valid without requiring eager future-instant arithmetic.

Permit semantics

PassiveHealth::try_acquire returns a CircuitPermit or a typed rejection. A permit reserves half-open capacity when applicable. Exactly one terminal action should follow:

  • complete with a classified outcome;
  • cancel explicitly;
  • drop, which cancels.

Every permit carries the circuit epoch from which it was issued. Late completion from an old epoch is ignored, preventing reordered responses from corrupting a newer forced or reopened state.

Active health

ActiveHealth owns classification and scheduling state, while the caller owns I/O and time. Its configuration defines:

  • healthy-result threshold;
  • unhealthy-result threshold;
  • probe interval;
  • initial unknown policy.

Only one caller can reserve a due probe. Completion with Healthy or Unhealthy advances consecutive thresholds. Cancellation preserves the current classification and schedules another attempt.

Use the clock-aware methods when the adapter supplies a simulated or runtime-specific Instant. Do not reserve using one time domain and complete using another.

Tokio adapter

poise-tokio::ActiveHealthRunner supplies timing around a caller-defined TokioProbe. Timeout policy is explicit:

  • classify timeout as unhealthy; or
  • cancel without changing classification.

Dropping the runner future finalizes its reservation as cancellation. Repeated probe scheduling follows Tokio’s clock, including paused test time.

Outcome windows versus circuits

These mechanisms answer different questions:

MechanismQuestion
Consecutive-failure circuitHas this endpoint failed repeatedly right now?
Rolling outcome windowWhat fraction and weighted kind of recent attempts failed?
Outlier detectorIs this endpoint materially worse than sufficiently sampled peers?
Active healthDoes an independent probe currently classify it healthy?

Combining them is reasonable, but define precedence. A typical order is administrative state → active health → circuit permission → outlier ejection.

Choosing thresholds

Thresholds are workload parameters, not library defaults to copy blindly. Derive them from:

  • ordinary request rate per endpoint;
  • expected transient failure bursts;
  • probe interval and timeout;
  • acceptable time to detect and time to recover;
  • minimum healthy capacity;
  • retry amplification.

A failure threshold of three means something very different at one request per minute and ten thousand requests per second.

Operational invariants

  • Force operations take effect immediately and invalidate stale probes.
  • Cancellation never counts as backend failure.
  • Half-open concurrency is an atomic reservation, not a statistical target.
  • Group-relative ejection never exceeds both configured availability caps.
  • Health wrappers never revive administrative draining or opt-out.
  • Control-plane actions should record the signal and epoch that caused them.