Operations runbook

No API replica passes the traffic readiness probe for two minutes.

1. 1.Confirm both replicas independently through the private readiness path; liveness alone is not sufficient.
2. 2.Publish degraded status and isolate only a failing replica when another remains ready.
3. 3.Inspect database, cache, index freshness and ETH/USD dependencies; roll back only when the signed application digest caused the failure.

No indexer candidate holds leadership and reports ready.

1. 1.Identify the advisory-lock holder and confirm both candidates are live.
2. 2.Restart only a dead candidate; for leader failover, prove lock release before the passive candidate acquires it.
3. 3.Verify checkpoint continuity, provider agreement and reconciliation before clearing degraded status.

The complete API readiness metric series is absent for two minutes.

1. 1.Treat a missing series as monitoring failure, not healthy service.
2. 2.Check both scrape targets, credentials, private routing and collector health without exposing the scrape token.
3. 3.Confirm metrics and independent readiness probes from both replicas before resolving.

The complete indexer readiness metric series is absent for two minutes.

1. 1.Check both active and passive candidates and the private monitoring route.
2. 2.Preserve the advisory-lock session while repairing scraping; do not start a second projector.
3. 3.Confirm leader identity, checkpoint continuity and fresh metrics before resolving.

Independent providers disagree at the same L2 height or one cannot participate.

1. 1.Keep projection, USD metrics, graduation and King transitions paused.
2. 2.Capture provider heights, same-height hashes and response times without credentials, then locate the last common block.
3. 3.Resume only after consecutive agreement checks, deterministic replay and onchain reconciliation.

The canonical chain projection enters an unexpected paused state.

1. 1.Preserve raw logs and the current checkpoint; do not toggle state merely to hide the alert.
2. 2.Identify whether provider agreement, reconciliation, an invariant or an operator action caused the pause.
3. 3.Record the common ancestor, replay range and resulting state fingerprint before recovery.

The confirmed chain tip is more than ten blocks ahead of the checkpoint.

1. 1.Compare confirmed tip, checkpoint, database write latency and both RPC providers.
2. 2.Keep a single projection leader and do not reduce confirmation depth during the incident.
3. 3.Escalate if lag grows or public freshness fails, then restore any temporary reviewed tuning after recovery.

The active indexing loop has not completed successfully within its freshness budget.

1. 1.Check process liveness, event-loop saturation, database connectivity, provider timeouts and the lock session.
2. 2.Capture a bounded diagnostic snapshot, then restart the leader only if it is not progressing.
3. 3.Prove no concurrent projector existed and reconcile before closing the incident.

Onchain supply, pool, position and fee-route reconciliation is older than ten minutes.

1. 1.Stop graduation and King transitions and retry through the secondary archive provider.
2. 2.Never waive a supply, canonical-pool, position-owner, locker or fee-recipient mismatch.
3. 3.Pause new launches through the disclosed governance path and escalate based on fund impact.

Five-minute server-error ratio exceeds two percent.

1. 1.Break failures down by registered route and signed deployment digest; keep identifiers and secrets out of metric labels.
2. 2.Remove one faulty replica when possible and preserve its redacted evidence.
3. 3.Roll back only to a previously verified digest and record before/after error rates.

Five-minute cached-read p95 exceeds 300 milliseconds.

1. 1.Compare route latency with database, cache, RPC and instance saturation.
2. 2.Preserve correctness and public rate limits while scaling stateless replicas or restoring a verified digest.
3. 3.Require two consecutive five-minute windows below budget before resolution.