Operating Dwarf

For operators running Dwarf in production. This is the orientation page: what the engine repairs on its own and on what schedule, what genuinely needs a human, and where the rest of the operational documentation lives. If you are writing workflow code rather than running it, start at Driving Flows.

Dwarf is a library, not a service. It runs inside your application’s process, and everything durable about it lives in the SQL databases you point it at — there is no broker, no coordinator, and no state on disk beside the database. That shapes what operating it means: you operate a fleet of application replicas and their databases, and the engine’s own moving parts are background loops inside those replicas.

What the Engine Repairs Without You

Most of what looks alarming in a distributed system is already handled here, on a fixed schedule. Knowing the schedule is what tells you whether to intervene or wait — and the answer is usually wait.

Two consequences worth internalising:

• Execution is at-least-once. A recovered step re-runs its task. That is the contract, not a failure — tasks must be idempotent. What the engine guarantees is that the flow’s persisted state reflects exactly one execution, even when two workers overlap.
• A slow database is absorbed, not escalated. An outage shorter than a step’s lease is invisible: workers wait it out and continue. You will see latency, not errors.

What Actually Needs You

Four things, and everything else in this section of the docs is one of them.

1. An alarm fired. Five counters are documented as zero in a healthy engine, and a non-zero reading means a latent bug, a flow that needs manual recovery, or closed flows deleted without an archive copy. → Runbook
2. You are deploying. Redeploying your own application is an ordinary rolling restart; moving to a new version of Dwarf is not — while Dwarf is v0.x it makes no compatibility promise, and schema migrations run at startup. → Upgrading
3. You are changing the shard set. Shard membership is baked into every flow key, so this is a planned maintenance operation, not a live resize. → Resharding
4. You need to answer for the data. What the engine stores, what free-text search reaches, and how retention is driven. → Data handling

What to Watch

The full instrument catalogue is in Observability. For a dashboard, the split that matters is between counters that should never move and counters that are merely informative. Counters are named here as you query them in Prometheus, with the _total suffix its exporter appends; the instrument itself carries no suffix (dwarf_steps_unwedged).

Alarms — page on any non-zero rate:

Health — alert on sustained levels, not on any movement:

One dashboard trap worth stating up front: the backlog gauges describe a shard, and more than one replica can report the same shard, so summing them across replicas overstates the backlog and makes an age reading meaningless. Take the max per shard first. The per-replica/per-shard split is tabulated in Observability.

Alerting on Lag

Three rules cover dispatch falling behind. The thresholds are starting points; set them from what your workflows can tolerate.

# Dispatch is falling behind on a shard. A shard that keeps up reads under a second.
max by (shard) (dwarf_steps_dispatch_lag_seconds) > 30            # for: 5m

# A shard has nobody dispatching it, so nothing on it runs and it reports no lag at all.
sum by (shard) (dwarf_shard_dispatching) == 0                       # for: 2m

# A priority band that must not starve has waited too long. Add one per band you care about.
max by (priority) (dwarf_steps_oldest_pending_age_seconds{priority="3"}) > 600   # for: 15m

• The lag counts from when a step became runnable, not from when it was created. A step that sleeps, backs off, waits on a subgraph or waits for a Resume does not raise it, so a long-running workflow does not need a looser threshold.
• It covers only the bands being served, leaving out a band that better-priority work leaves no capacity for. Strict priority holding back worse bands is by design. Alert on those bands through dwarf_steps_oldest_pending_age_seconds, and only for bands that carry a deadline.
• Do not alert on the per-band series being absent. dwarf_steps_pending and dwarf_steps_oldest_pending_age_seconds are refreshed every 30 seconds by one replica per shard. They are missing for up to 30 seconds after startup or while that duty passes between replicas, and they go absent if the refresh keeps failing for 90 seconds. The lag and dwarf_shard_dispatching are the liveness signals.
• For trend, not paging: sum by (priority) (increase(dwarf_flows_started_total[15m])) - sum by (priority) (increase(dwarf_flows_closed_total[15m])) staying positive means flows are arriving faster than they finish, per band, before any lag shows.

When a rule fires, Backlog is growing walks through finding the cause.

Sizing and Capacity

Sizing is measured rather than guessed, and the numbers live with the measurements:

• Deployment — choosing a database, declaring shards, connection pools, workers, drain windows, the disk-throughput requirement, and sizing a deployment from the measurements.
• Cloud Benchmarks — the measurements: throughput, cost and failover against managed PostgreSQL across a real network hop.

The rules to carry into a capacity plan: plan a 32-vCPU database at ~18,000 steps/s of average load for steps with little state (less with payloads), size its checkpoints, check what your peaks cost in waiting (planning for peaks), and give each shard two engine replicas at half its vCPU, either of which can carry the shard alone. Both assume your tasks are cheap; a host dispatching tasks over a network transport pays more engine CPU per step and should measure its own.

Where to Look Next

Next: Production Checklist.