v0.14.0 — A Daily Archive for Closed Flows, Lists by State, Continue by Key

Upcoming release: these notes may change before it is published on GitHub Releases.

Closed flows now move to an archive about a minute after they close, kept by the day for a retention you set and dropped a day at a time, so the live tables that every step reads hold the open flows and little else. An archived flow reads by its key as before. The lists are named by a flow’s state - ListOpen and ListClosed - and SetFlowExpiration can end flows that would otherwise never close. This release edits migrations that earlier releases shipped, so existing databases cannot be upgraded in place: start v0.14.0 on fresh databases (see Upgrading). Nothing detects an old database: Startup succeeds on one, and ListOpen then fails on every call.

Breaking Changes

• Closed flows are deleted after 30 days by default. Each closed flow moves to the archive about a minute after it closes, and is deleted with its archive day once the retention has passed (SetRetention(days), default 30; see New). In v0.13.0 every flow was kept until Delete or Purge removed it. To keep flows longer, raise the retention before Startup.
• On PostgreSQL, every replica must connect as the role that owns Dwarf’s tables: Startup fails otherwise, naming the owner. The archive creates and drops a table per day, which only the owner may do.
• MySQL 8.0 or later, or MariaDB 10.6 or later, is required: archiving locks with SKIP LOCKED, which earlier versions lack. Both are past their end of life.
• Go 1.27.3 or later is required, as sequel v1.18.0 requires it.
• SQL Server is reached through github.com/microsoft/go-mssqldb, in place of the deprecated github.com/denisenkom/go-mssqldb. A program that imports the old package itself must switch to the new one: both register the mssql driver, so linking both panics at startup.
• Threads are removed; Continue takes the key of the flow to continue. Continue(ctx, flowKey, additionalState) (nextKey string, err error) builds a new flow from the completed root flow flowKey names, carrying its final state, graph and policy as before. Your application keeps the chain: continue the key the last Continue returned.
  • Continuing one flow twice makes two flows, each built on it. Earlier, a second Continue of the same thread was rejected with 409 while the first turn ran.
  • Like Create, Continue is not idempotent: a call retried after a lost response makes a second flow.
  • Continue of a flow that is not completed, or is scheduled for deletion, is a 409. Earlier, Continue searched the thread for its latest completed flow, skipping forks and flows scheduled for deletion.
  • FlowOptions.ThreadKey, Query.ThreadKey and FlowSummary.ThreadKey are removed. To start the next flow with different options, call Create with the previous flow’s final state (from Snapshot).
• List is replaced by ListOpen and ListClosed. A flow is open while it is running or interrupted, and closed once it has completed, failed, been terminated or been cancelled.
  • ListOpen(ctx, query) lists the open flows, the most recently updated first. A status that is not open is a 400.
  • ListClosed(ctx, query) lists the closed flows, the most recently archived first, by the new FlowSummary.ArchivedAt. A flow appears there when it is archived, about a minute after it closes; until then it is in neither list, and is read by its key. A status that is not closed is a 400.
  • Ties go by shard, then the newest flow. Both require Query.Limit (at most 65,536). ListClosed also requires a window with both ends, the new Query.FromTime (inclusive) and Query.ToTime (exclusive); ListOpen takes them optionally, on UpdatedAt. Earlier, List defaulted to 100.
  • ListClosed’s window is cut to the retention and to now, and may then reach at most 31 days.
  • Neither has pagination: List loses its cursor result, and Query.Cursor is removed. The result is the first Limit across all shards, in one order. Earlier, on more than one shard, each shard returned ceil(Limit/shards) of its own flows, grouped by shard, so a page could hold more than Limit.
  • The dwarf_operation_seconds label op="list" is now list_open and list_closed. A new op, purge_before, times PurgeBefore, and Purge records one sample per stage it runs on a shard (reading the archive, and deleting there) rather than one per call.
• FlowsEnded is renamed FlowsClosed, and the metric dwarf_flows_ended is renamed dwarf_flows_closed (dwarf_flows_closed_total in Prometheus): update dashboards and alerts that read the old name.
• Query.OlderThan and Query.NewerThan are removed; give the window as Query.FromTime and Query.ToTime instead. “Older than 30 days” is ToTime: time.Now().Add(-30 * 24 * time.Hour). The bounds now compare with this process’s clock rather than the database’s.
• Delete and Purge remove only closed flows. Neither terminates anything any more.
  • Delete of a running or interrupted flow returns 409; Terminate it first. Earlier, Delete terminated an interrupted flow, subgraphs included, before marking it.
  • Delete can also return 409 when the flow was archived while it was being deleted; call it again.
  • Purge never touches an open flow, which it used to terminate when interrupted. A Purge whose Query.Status is not closed (running, interrupted) returns 400.
• Purge removes what ListClosed lists, returns (removed int, done bool, err error), and requires Query.Limit and both window bounds. It removes at most Limit roots (up to 65,536) in a call, the oldest archived first across all shards, and deletes each at once; earlier it took at most 4,096, ceil(Limit/shards) from each shard. done says that nothing archived and matching is left: call it until done. A query needs no other filter: the window bounds it, and like ListClosed’s may reach at most 31 days of the archive, so a longer range is purged a month at a time. It removes only archived flows: one that closed shortly before the call is purged by a later call, once archived; Delete removes one by its key.
• A subgraph that cannot be started comes back to the calling task from flow.Subgraph, as a failed child does, instead of failing the step. That covers a LoadGraph that returns an error or panics, returns no graph, or returns an invalid one. The task’s next run gets err, with no child flow created and nothing in out. Earlier the step failed at once, skipping the graph’s onError and giving the task no chance to retry.
  • err keeps its status code: your host’s own (return a 503 when the graph’s source is unreachable), 404 for no graph, 400 for an invalid one, 500 for a panic.
  • Returning err takes the task’s onError transition, or fails the step as before; a task that already returns Subgraph’s error needs no change. To wait out a 503, arm flow.Retry: its next Subgraph call starts the child again. See When the child fails, or cannot start.
  • As on any return from a subgraph, the task runs again from the top, so code before its Subgraph call runs a second time when a spawn fails.
• The Flow’s JSON format is versioned, and its durations are whole milliseconds. This matters only to a host that carries the flow as JSON to a task in another process.
  • Every encoding carries "v": 1. Decoding refuses a body without v (status 400) or with a newer one (status 501, naming both versions), so a host and a task server on different releases fail the step loudly instead of misreading each other. Upgrade both sides together.
  • sleepDuration, backoffInitialDelay and backoffMaxDelay, which were integer nanoseconds, are now sleepMillis, backoffInitialDelayMillis and backoffMaxDelayMillis, in whole milliseconds. A task server written in another language must use the new names.
  • A Flow decoded from JSON encodes back in the version it came in, so a task server replies in its request’s version.
• State field names are checked. A name is an ASCII letter or underscore, followed by ASCII letters, digits, underscores or hyphens: line_item and line-item-2 are valid; 2nd, order.items and naïve are not. workflow.IsValidFieldName applies the rule. Earlier only the names a graph declares were checked, and those could not hold a hyphen. See Field names.
  • Create, Run, Continue and Fork refuse caller state holding any other name with a 400.
  • In a task, Set, Overwrite, the typed setters, Del and Clear panic on such a name, which fails the step. Earlier, Set and Overwrite returned an error for a name ending with ! or + and accepted every other name.
  • A task in another process whose result writes such a name fails its step.
  • A subgraph’s in holding such a name comes back from flow.Subgraph as a 400, with no child started.
  • A when expression cannot compare a field whose name holds a hyphen.
• A fairness key is at most 256 ASCII letters, digits, -, _ and ., and is case-sensitive. Create and Run refuse any other key with a 400: one holding a space, /, @, :, a quote, an accent or any other character. The empty key is still the default bucket. Earlier any valid UTF-8 key was accepted, and on MySQL, MariaDB and SQL Server keys differing only in case, accents or trailing spaces were scheduled and matched as one.
• Graph names, task names and workflow URLs must not begin or end with whitespace. Validate refuses such a graph or task name, so Create and Run return a 400 before anything is stored, for a graph built in code or decoded from JSON. Create and Run refuse such a workflow URL with a 400, and flow.Subgraph returns one without starting a child. Names and URLs are compared exactly, case included. Earlier a padded inner fan-in’s cohort was left out of the outer fan-in’s state.
• flow.Retry’s giveUpAfter bounds the total of the backoff delays, not the time since the step was created. Retry returns false once the delays it has asked for, the next one included, would add up to more than giveUpAfter. Time spent queued, running or sleeping no longer counts, so a step held up behind other work keeps its retries; earlier, a loaded database could use them all up before the first retry ran. A task that wants its sleeps or its run time bounded keeps its own budget. A zero initialDelay never reaches the bound, so bound immediate retries by f.Attempt(). See Retry.
• Shard and fairness weights are at most 1,048,576 (2^20). SetShard refuses a larger weight, NaN or an infinity, and Create and Run refuse such a FairnessWeight with a 400. A share is set by the ratio of weights, so a fraction still expresses a small one. Earlier, two very large shard weights summed to infinity and sent every new flow to one shard.
• A graph with a task named "" is invalid. Validate refuses it, so Create and Run return a 400, and a subgraph of it fails its caller with a 400. Earlier the flow ran, skipped that task and completed.
• Flow.SetChanges is removed. Write each output field with flow.Set instead. SetChanges marshalled its struct, so a field tagged omitzero or omitempty and left at zero was never written, and a task could not reset it to false, 0 or nil.
• Migrations were edited in place. dwarf_flows loses thread_id, thread_token and their index, and gains archived_at; dwarf_steps gains parked_child_flow_id. On PostgreSQL every time column is TIMESTAMP(3), milliseconds as on the other databases, where it was microseconds. On MySQL and MariaDB the tables dwarf_flows and dwarf_steps are utf8mb4_bin, and on SQL Server their text columns are Latin1_General_100_BIN2 (see Fixes).
• Times are milliseconds on every database, PostgreSQL included. FlowSummary’s and FlowStep’s times, and the Query.FromTime/ToTime bounds, compare and read back to the millisecond. Requires sequel v1.16.0.

Fixes

• On MySQL, MariaDB and SQL Server, text compares exactly, as on PostgreSQL and SQLite. The tables took the server’s default collation, which ignores case (and on MariaDB accents), so ListOpen, ListClosed and Purge filtered by FairnessKey, WorkflowURL, WorkflowName or StoppedTaskName also matched values differing only in case: a list of tenant acme’s flows included tenant Acme’s, and a Purge of acme deleted Acme’s flows too. Query.Search stays case-insensitive.

• A failover that ends just as a step’s outcome write gives up no longer fails the step. When an outcome cannot be stored, the engine retries the write for about a minute, then fails the step if the database is reachable, taking the outcome itself as unstorable. A database that came back during the last retry’s wait was reachable by then, so a healthy flow failed for good and counted in dwarf_steps_write_failed. The engine now first checks that the database answers and retries the write at once; a write that loses its connection is retried on a fresh one, up to three times. A step fails only when its outcome fails with the database answering just before and after. Requires sequel v1.18.0.

• A stalled worker can no longer complete a flow while its cancellation handler runs. A worker that stalled after its task finished, past its lease, could resume and mark the flow completed after recovery had replayed the step and routed a Cancel to the onError handler, with the handler still running.

• A subgraph’s late result can no longer settle a newer attempt. When a calling task retried with flow.Retry and started a new child, a result from the earlier child arriving late could be delivered to it, and the task could finish on the old result while the new child still ran.

• Two Resume calls cannot both answer one interrupt. Two concurrent calls for the same interrupt both returned nil, and one answer was discarded; one now gets a 409. A Resume that selected an interrupt, then lost the race to another Resume, could also answer the task’s next interrupt with its old data after the task retried; it now gets a 409 as well.

• On MySQL and MariaDB a task can no longer run twice from one claim. A worker delayed past its lease between claiming a step and reading it could read a later claim’s step and run it, so the task ran twice and its next step was created twice.

• On SQL Server, Query.Search treats [ literally. It opened a character set, so a search for [ab] matched every value holding an a or a b, and Purge with that search deleted those flows too.

• Fairness weights hold their shares at any scale. Equal weights split the work evenly whether they are 1e-6 or 1e20; earlier, at very large or very small weights one key took every slot.

• Fork keeps the outcome of a finished subgraph it copies. Snapshot of a kept child’s key returned an empty state for a completed child and no error for a failed one.

• A Retry multiplier below 1 shrinks a delay that starts above maxDelay. Retry(10s, 0.1, time.Second, 0) waited 1s at every attempt rather than 100ms at the third.

• A forEach transition decoded from JSON without as uses item, as AddTransitionForEach does. Its item, itemIndex and itemCount fields were left in the state after the fan-in.

• A step whose worker died before its next step was created is recovered on its own lease. A fan-out branch waited until every sibling had finished, which could be a sibling’s day-long retry backoff, a subgraph or an interrupt, and a step in an interrupted flow was never recovered.

• flow.Subgraph("") returns false with a 400, so the step fails. Earlier the step completed with no child, skipping the code after the call.

• An interrupt or Resume in a deeply nested subgraph works on SQL Server. An interrupt more than about 700 subgraphs deep, or a Resume more than about 2,100 deep, failed every time.

• A replica restarted under the same SetEngineID takes the same place on every shard. If a peer had removed its record on some shards while it was down, it was ranked differently per shard, so the shards’ dispatching was divided unevenly between replicas, and nothing corrected it while the fleet was unchanged.

• A replica whose step reads are failing gives up its share of the work. Its earlier successes kept it counted as dispatching, for longer the longer it had been up, so its share of a shard’s steps waited.

• Await, Poll and Run no longer wait on another shard. A slow or stalled shard held up the result of every flow on the other shards until it answered.

New

• MariaDB works under its own driver name. sequel v1.16.0 reports mariadb for a database opened with a mariadb:// URL or the mariadb driver name; Dwarf treats it as it treats MySQL. A MariaDB DSN in the MySQL driver’s own format (user:pass@tcp(host:3306)/db) still works, as mysql.
• Closed flows move to an archive about a minute after they close, and are kept for SetRetention days (30 by default), then dropped a day at a time. An open flow is never removed, however old, unless it expires (below). A failed, terminated or cancelled DeleteOnCompletion flow is archived like any other. Delete of an archived flow deletes it from the archive at once, with its subgraph trees; a closed flow not archived yet is marked for deletion, as before.
• An archived flow reads as it did before it moved: Snapshot, Fingerprint, Await, Poll, Run, History, Step and Continue find it by its key, Fork clones it and ListClosed lists it, until its day is dropped. Resume, Terminate and Cancel answer it as any closed flow: Resume 409, Terminate and Cancel a no-op.
• SetRetention(days) sets how many days the archive keeps each day’s closed flows (default 30; 0 keeps no archive). Construction-time.
• SetFlowExpiration(d) terminates a flow still running or interrupted d after its creation, with its subgraphs and a reason saying it expired; it is then archived like any other. Off by default; checked every 5 minutes. It bounds what abandoned interrupts and flows polling forever leave in the live tables.
• Metrics for the archive and expiration, none costing a query: dwarf_flows_archived, dwarf_flows_reaped{reason} (archive_failed counts records lost to a failing archive: alert on it), dwarf_flows_removed{op,side}, dwarf_archive_days_dropped, dwarf_archive_maintain_failed, dwarf_reap_pass_seconds, the gauges dwarf_archive_lag_seconds, dwarf_archive_failing_seconds, dwarf_archive_days and dwarf_archive_oldest_day_age_seconds, and dwarf_flows_expired and dwarf_flows_expire_failed.
• workflow.RequestCarrier(f) and workflow.ResultCarrier(f) carry a task across a process boundary one direction at a time: pass them to json.Marshal, json.Unmarshal or an encoder in place of the Flow. The result leaves out the state and the graph’s reducers, so a large state crosses the wire once rather than twice, and reading a result onto the host’s Flow takes only the task’s changes and control signals. The engine reads back nothing else either way, so the plain Flow encoding keeps working. See Tasks in another process.
• workflow.CarrierVersion is the newest version of the Flow’s JSON format this build reads and writes.
• PurgeBefore(t) drops the archive’s records of every day before t’s, on every shard; a t after today is a 400.
• The live tables vacuum in small, unthrottled passes on PostgreSQL, and are never truncated by vacuum.
• SQLite is supported for one replica at a modest flow rate: on a file with _pragma=journal_mode(WAL) for durable flows, or in memory, where flows last only as long as the process. Earlier, the docs ruled it out of production. See Deployment.
• Retention counts from the end of the UTC day a flow was archived in, so a flow is kept between retention and retention + 1 days after it is archived, plus up to an hour for the hourly drop. A flow archived late, after an outage, is kept its full retention from then.
• A new table, dwarf_flow_days, records the archive day of each flow archived more than a day after it was created. It stays small: only flows that ran about a day or longer get a row.

Performance

• Archiving writes each closed flow’s rows again: the move copies them into the archive and deletes them from the live tables, and on PostgreSQL every archived row is frozen once more. With little state per step this costs nothing measurable: one 32 vCPU PostgreSQL 16 shard ran ~24,100 steps/s with the archive on. With payloads, the write-ahead log sets the pace, so size the database’s write throughput in MB/s from what a step logs. See Deployment and the cloud benchmarks.
• Size PostgreSQL’s checkpoints for the write load (new guidance): max_wal_size about 2 × peak WAL MB/s × 60 s and checkpoint_timeout 15 minutes. With 8 KB payloads a 32 vCPU shard ran 42% more steps/s at 20 GB than at the 1.5 GB default many managed services use. See Deployment and Checkpoints.
• ListOpen and ListClosed on more than one shard read up to Limit flows from each table on each shard, rather than Limit in all.
• Continue makes two fewer round trips to the database, and takes no lock.
• Creating a subgraph flow writes one statement fewer on MySQL, SQL Server and SQLite.
• A union fan-in takes time linear in its branches. It was quadratic, inside the fan-in’s transaction: 4,000 one-element branches took 8.2s, now 16ms.
• Validating a graph and routing its fan-ins take time linear in its edges, where a large graph took time proportional to its tasks times its edges.
• flow.Retry on SQLite looks up the step’s earlier subgraph children by index rather than scanning every flow.