Observability

For developers and operators. Developers inject the three providers; operators read what comes out. This is the full instrument catalogue and how to interpret each one. For which of them to alert on, see Operating Dwarf; for what to do when one fires, the Runbook.

Dwarf emits structured logs, OpenTelemetry metrics, and distributed traces. All three go through standard providers you inject — the engine owns no exporter and adds no network connections. With nothing injected, everything degrades to a no-op, so unconfigured and test use pays nothing.

Logging

Inject a standard library *slog.Logger:

eng.SetLogger(slog.Default())

It defaults to a discard logger — the engine (and its sequel DB layer) stay silent until you inject one, rather than writing to the application-owned slog.Default(); a nil logger resets to that silent default. The engine logs through the …Context variants (InfoContext, DebugContext, …), so a context-aware handler can correlate each record with the active step span. To route logs to OpenTelemetry, pass a logger whose handler bridges there — e.g. the otelslog bridge — optionally fanned out to a stdout handler for container logs. The bridge stamps each record with the active trace and span IDs, giving you trace↔log correlation for free.

For development and test runs, SetDebugLogger() is a one-liner that wires a human-readable text logger to stderr at debug level — so the engine’s (and sequel’s) internal logging is visible without standing up an OTEL pipeline. It routes through SetLogger, so it counts as an explicitly-set logger:

eng.SetDebugLogger()   // shorthand; stderr, text, debug level

Metrics

Inject an OpenTelemetry metric.MeterProvider:

eng.SetMeterProvider(otel.GetMeterProvider())

It defaults to the global otel.GetMeterProvider() — the no-op provider unless your process configures the OpenTelemetry SDK. The engine builds its instruments under the scope github.com/microbus-io/dwarf; service identity comes from the provider’s Resource, not from per-metric attributes.

Terms in Metric Names

Some metric names use the engine’s internal vocabulary. In plain terms:

• Claim and lease. A worker claims a step before running it. The claim is time-limited, expiring after the task’s time budget plus 30 seconds; the metrics call it a lease. A recovered step is one whose lease expired because its worker crashed. See Crash Recovery.
• Dispatcher. A replica chosen to run a given shard’s steps. See Dispatchers.
• Refill. How a replica finds due steps: it periodically queries each shard for them and keeps a small local queue of candidates for its workers.
• Doorbell and offered. When a replica creates a step it can run itself, it hands the step straight to its own workers instead of waiting for its next query; this is local to the replica, and nothing is sent between replicas. An offered step is one that arrived this way.
• Stolen. A step picked up by a replica other than the one that would normally run it, which happens when a peer is alive but not keeping up.
• Turnstile. The engine’s queue for database access. A worker waits for a turn before each database call, and the number of turns is a fixed multiple of the connection pool. When turns run out, database connections are the bottleneck.
• Peer registry. The small dwarf_peers table in each shard where every replica records itself periodically. It is how replicas know about each other.
• Park and wedged. A flow is parked while it waits on an Interrupt or a subgraph, holding no worker. A wedged park is a step waiting on a subgraph that never reported back; unwedged counts the ones a sweep repaired.

Instruments

The engine emits 49 instruments — 24 counters, 19 gauges, and 6 histograms. Counter instrument names carry no _total suffix; a Prometheus exporter appends it at the scrape boundary, so the names below are what you query in PromQL with _total (e.g. the dwarf_flows_started instrument is queried as dwarf_flows_started_total):

The counters increment inline at their event sites; the gauges are observable (async) and read engine state at collection time. No gauge queries the database when it is collected, so a short scrape interval costs the database nothing.

Is dispatch keeping up? dwarf_steps_dispatch_lag_seconds is the live answer: how long the most overdue step in the priority bands being served has been due, per shard. A step that slept, backed off, or waited on a subgraph or an interrupt is measured from when it became runnable, not from when it was created. The lag covers only the bands being served: a band that better-priority work leaves no capacity for is left out, and a shard holding only such bands reads zero. When better-priority work leaves capacity over, the served bands reach further, and if a band that has waited a long time comes into reach the lag jumps to its age. It falls back when the better bands fill the capacity again. A jump of that kind is priority at work, not a stalled dispatcher: read it beside the per-band ages. Strict priority holds those bands back by design, and that shows in dwarf_steps_oldest_pending_age_seconds, one reading per band, refreshed every 30 seconds along with dwarf_steps_pending. While the lead for a shard passes from one replica to another, those two may be missing for up to 30 seconds.

Open Flows

dwarf_flows_started and dwarf_flows_closed carry the same shard and priority, so the difference of their increases over a window is how many open flows each band gained or shed:

sum by (priority) (increase(dwarf_flows_started_total[5m]))
  - sum by (priority) (increase(dwarf_flows_closed_total[5m]))

Read it as a trend, not as a level. Each replica’s counters start from zero when the replica does, so a running total of started − closed drifts across restarts. It also counts flows rather than steps: a flow waiting on a sleep, an interrupt or a long task is open without being in any backlog.

Read the two in-flight state gauges as a quotient, not separately. bytes / steps is the mean state a task carries, and that is what distinguishes the two ways a replica ends up holding a lot: a few tasks each carrying a large document (a big mean) versus a wide fan-out of tasks each carrying a little (a small mean against a large count). They call for different responses — the first for narrowing what the workflow carries, the second for the fan-out width itself — and neither number alone tells them apart.

It measures the JSON a task’s state was built from, not the in-memory form, which is several times larger. That is deliberate: read against your runtime’s heap it gives the expansion factor, and it stays comparable across engine versions that represent state differently. It is also the only reading that covers a cost nothing else reports — a task holds no database connection while it runs, so the worker pool grows freely for long tasks, and held state times pool size is a memory ceiling dwarf_workers_resident cannot show on its own.

The Archive

Every closed flow waits in the live tables for about a minute and then moves to the archive (dwarf_flows_archived), unless it was scheduled for deletion or the retention is 0 (dwarf_flows_reaped). Two readings say whether that is keeping up. dwarf_archive_lag_seconds is how far behind the move is: rising means flows close faster than they are archived, and the live tables, which every dispatch reads, grow with it. Archiving pauses for the last minute of each UTC day, so the reading rises to about 75 seconds then and falls back after midnight: set an alert above that. dwarf_reap_pass_seconds approaching its 15-second interval says the same from the other side.

Alert on dwarf_flows_reaped{reason="archive_failed"}. While archiving fails, closed flows wait; after a minute of failure they are deleted unarchived, so that a fault in the archive cannot fill the live tables. Each one counted there is a record the archive does not have. dwarf_archive_failing_seconds rises from the first failure, a minute before any loss. See Archiving is failing.

dwarf_archive_days and dwarf_archive_oldest_day_age_seconds show the retention being kept: both settle once a full retention has accumulated, and both climbing past it means expired days are not being dropped (dwarf_archive_maintain_failed).

None of these costs a query: each is counted from work the engine does anyway. The sizes of the tables, and on PostgreSQL their dead rows, are the database’s to report. Watch them with your database’s own exporter: the dwarf_flows and dwarf_steps sizes against the open flows, and the dwarf_*_archive_* tables against the retention.

Aggregating the gauges across replicas — two kinds, and mixing them up inflates your dashboard by the replica count:

The per-shard gauges describe a shard’s database. Each replica dispatching a shard reports its dispatch lag and the two reaper readings, one of them, the shard’s leader, reports the per-band pair, and every replica reports the two archive day readings. More than one replica can report the same shard: two dispatchers, or an outgoing and an incoming leader for a moment while the lead passes. So never sum across replicas. Take the max per shard first:

sum by (priority) (max by (shard, priority) (dwarf_steps_pending))

dwarf_db_phase_workers carries a role label that aggregates across replicas but not across roles: enter and exit are separate populations. For the pressure a phase actually reached rather than whatever the scrape happened to catch, take max_over_time of it.

Labels are deliberately bounded: there are no per-fairness_key labels (that would be unbounded cardinality), so fairness metrics are aggregate-only. priority and workflow are labels on the assumption that a deployment uses a small, fixed set of each: every value is a separate series, multiplied by the other labels. A deployment that mints priorities or workflow URLs per request should drop those labels with a metric View in its OpenTelemetry pipeline.

Reading the two refill histograms. They answer “is candidate selection costing me throughput, and if so where.” Each shard’s selection runs on its own cycle, against its own database, so:

• dwarf_refill_query_duration_seconds split by shard shows whether one database is slower than its peers. A single shard’s tail dragging while the others stay flat is the shard, not the engine. It slows that shard’s own dispatch; the other shards keep selecting on their own cycles, planning from the slow shard’s last report (dwarf_refill_tally_age_seconds shows how old that report is).
• The same metric split by phase separates the two queries a pass runs. A tail that appears only on band_keys and only intermittently, on a backlog that has not changed, points at the database’s query planner rather than at load (on PostgreSQL, stale table statistics can flip this query between an index scan and a sequential scan — measured on one rig at 0.3 ms versus 100 ms on identical data). ANALYZE on dwarf_steps, or more aggressive autovacuum settings for it, is the fix.
• Do not sum the phases. Selection runs as two independently paced loops — one scanning the priority band, one turning the resulting plan into cached candidates — so band_keys and the other three phases are measured on different clocks and their sum describes nothing. Read each phase on its own; the rate at which candidates reach the workers is set by the second loop, so a slow band_keys does not hold it back.

dwarf_refill_candidates_discarded over dwarf_refill_candidates_selected is the selection waste ratio. Each pass replaces the cache wholesale, so anything the workers had not yet picked up is dropped and re-selected later — always safe, never lost work, but a high ratio means selection is running far ahead of what the workers can consume and most of its database round-trips are being thrown away. A ratio near 0 means the opposite: selection is the slower half.

A sustained high ratio is a cadence question, not a fault: selection paces itself from the shard’s connection budget, assuming each connection drains steps at a steady rate, so a workload whose steps drain more slowly than that will be over-supplied. It costs database work rather than correctness — no step is lost. A large budget on a database that is already saturated is the usual cause, since extra connections there add little throughput.

Tracing

Inject an OpenTelemetry trace.TracerProvider:

eng.SetTracerProvider(otel.GetTracerProvider())

It defaults to the global otel.GetTracerProvider(). The host injects only the provider — there is no span code to write and no trace context to thread by hand.

Install the OpenTelemetry SDK before Startup. The engine resolves its tracer once, at Startup. If no SDK is installed by then, it traces through a no-op provider, so an engine running without tracing spends nothing building spans. A provider installed after Startup is not picked up until the next Startup.

The engine creates two kinds of span, under the github.com/microbus-io/dwarf scope:

• A root “workflow” span at Create. It’s detached (its own trace, not nested under the request that created the flow — a flow is a long-lived, async thing), and its W3C context is persisted on the flow so it survives across replicas and time.
• A per-step span in each dispatch, named by the task, parented to the flow’s root span and placed on the context handed to your host’s ExecuteTask — so any spans your task creates (the downstream call it makes) nest under it automatically.

Subgraphs nest naturally: a subgraph gets its own “workflow” span parented to the caller step’s span, so a trace reads workflow → caller-step → workflow(subgraph) → subgraph-steps, mirroring the call structure. A step that yields and re-dispatches (after an interrupt or subgraph) produces one span per execution attempt.

SQL Layer

The same three providers are handed to the engine’s sequel database layer, so the SQL underneath your workflows shows up in the same pipeline: per-operation spans (nested under the active step span), migration logs, and nine sequel_* instruments — all under the scope github.com/microbus-io/sequel. The logger is forwarded only when you explicitly set one, so an unconfigured engine stays silent here too.

These are not Dwarf’s instruments and are not part of the 49 above, but you will see them on the same scrape, and two of them answer questions no dwarf_* metric can:

The pool pair is the counterpart to the turnstile metrics, and it is where the two views meet. dwarf_turnstile_* shows the queue Dwarf orders; sequel_pool_wait_duration_seconds shows the wait that remains at the pool itself, which the turnstile deliberately does not remove — a turn admits a caller to compete for a connection, it does not reserve one. Pool wait at saturation is the expected steady state, and this is the number that says how much of it there is. It also silently rate-limits candidate selection, so it is worth watching whenever throughput moves for no visible reason.

The two wait instruments are cumulative counters - they come straight from Go’s sql.DBStats, which only ever increases, so they are observable counters rather than gauges and a Prometheus exporter appends _total to both. Take rate() or increase() over them; a raw value is a since-process-start total, and averaging it across replicas is meaningless.

The database label is the DSN’s database name, not the shard index — it falls back to the driver name when the DSN has none. In the recommended shard-per-server topology every shard often holds a database of the same name, in which case all of them report under one label value and their series merge. If you shard and want these split per shard, give each server’s database a distinct name; dwarf_* metrics label by shard and are unaffected either way.

Configuration Timing

All three observability knobs — SetLogger, SetMeterProvider, SetTracerProvider — are construction-time: set them before Startup. The engine resolves the providers and wires them into the worker hot path and the shard DBs once at startup, so a call after Startup is rejected with a 409.

Putting It Together

A typical host configures the OpenTelemetry SDK once and injects all three providers:

eng := dwarf.NewEngine()
eng.SetLogger(slog.New(otelslog.NewHandler("myapp")))
eng.SetMeterProvider(otel.GetMeterProvider())
eng.SetTracerProvider(otel.GetTracerProvider())
eng.SetHost(host)

Next: Upgrading.