Resharding

For operators changing the set of database shards behind a running fleet. Read Deployment → Sharding first for what shards are and how many to run; this page is about changing the set after flows already exist.

The Constraint

A flow’s shard is part of its identity and never changes. Every flow key is {shard}-{hour}-{id}-{token}, so the shard index is encoded in the handle your callers hold, and it is what routes every subsequent operation. Everything derived from a flow stays on its shard for life: subgraph children, continuations, and forks are all shard-pinned.

Three rules follow, and they are the whole of this page:

  1. The shard set is fixed for a process’s lifetime. SetShard is construction-time only and is rejected on a running engine. Changing the set means restarting replicas.
  2. The index-to-connection-string map must be identical on every replica. A flow created on a shard a peer does not know about is unroutable there — that replica answers “not found,” and a caller may legitimately act on that.
  3. Nothing rebalances. A flow stays on the shard it was created on for its whole life. Growth changes where new flows land, never where existing ones live.

Indexes are sparse. They must be unique and at least 1, but need not be contiguous — shards 1, 2 and 99 are a valid set. This exists precisely so a retired shard’s index can be dropped without renumbering the others, which would otherwise invalidate every flow key on them.

A shard’s life, with a rolling restart of the fleet at each arrow:

%%{init: {'themeVariables': {'primaryColor': '#32a7c1', 'primaryTextColor': '#f4f2ef', 'primaryBorderColor': '#32a7c1', 'lineColor': '#32a7c1', 'secondaryColor': '#e5f4f3', 'tertiaryColor': '#f4f2ef'}}}%%
stateDiagram-v2
    direction LR
    [*] --> Cordoned: add with Cordoned true
    Cordoned --> Active: set Cordoned false
    Active --> Retiring: set Cordoned true
    Retiring --> Drained: resume, terminate or purge its flows
    Drained --> [*]: remove SetShard

A cordoned shard is open and reachable but receives no new top-level flows. “Retiring” is the same cordoned state, entered on the way out.

Adding a Shard

Adding capacity is the common case and it can be done without a maintenance window, in two rolling restarts. The trick is to make every replica able to route to the new shard before any replica starts placing flows on it.

Phase 1 — introduce it, cordoned.

eng.SetShard(engine.ShardSpec{
    Index:    4,
    DSN:      "...",
    Cordoned: true,   // opened, migrated, routable — but takes no new flows
})

Roll the fleet. Each replica opens and migrates shard 4 at startup and can route flow keys to it, but placement skips a cordoned shard entirely, so nothing is created there yet. At the end of this phase the fleet is uniform and no flow exists that any replica cannot reach.

Phase 2 — activate it. Flip Cordoned to false and roll again. Shard 4 now takes new flows, under the placement rule below.

Do not collapse this into one step. A single rolling restart leaves a window in which upgraded replicas place flows on shard 4 while replicas still on the old configuration cannot route to them. Work still executes — the replicas that know shard 4 dispatch it — but callers hitting an old replica get “not found” for flows that exist. That is precisely the failure the two-phase order avoids.

How new flows are placed. With every Weight left at its default of 1, a replica places a new flow on one of the non-cordoned shards it dispatches, chosen at random, so the flow’s first step runs at once; a replica that dispatches no shard places across every non-cordoned shard. Where flows land then follows where they are created: a fleet whose creates all come through one replica fills that replica’s shards. Any Weight other than 1, on any non-cordoned shard, switches every replica to weighted placement over all non-cordoned shards, in proportion to their weights - a weight of 1.0001 counts. So declare weights to give a new server larger than the existing ones more than an even share, or to spread creates evenly whatever replica they arrive on (equal weights other than 1 do that). Nothing measures a shard’s capacity for you - the weights you declare are trusted.

Expect the new shard to fill slowly. It only receives new flows. The existing imbalance persists until the old shards’ flows complete and age out under your retention policy.

Retiring a Shard

Phase 1 — cordon it. Set Cordoned: true on the shard and roll the fleet. It stops receiving new flows immediately. Everything resident continues normally: flows keep executing, and the flows made from them - subgraph children, Continue’s and Fork’s - are still created on it, because those are shard-pinned.

Phase 2 — drain. Nothing drains automatically. The shard is empty when it holds no flow you still need, which means both:

// no open flow, running or interrupted
open, _ := eng.ListOpen(ctx, workflow.Query{Shard: 3, Limit: 100})

and no closed flow you still want readable: the shard’s closed flows sit in its archive until the retention drops their day, so the shard is empty once the retention has passed since the end of the UTC day its last flow was archived in.

Query.Shard scopes a list to one shard. Watch interrupted flows especially — they wait indefinitely and are the ones that will still be there when you assumed the shard was empty. Either resume them, terminate them, or wait.

Phase 3 — remove it. Drop the SetShard call and roll the fleet. Leave the index unused; do not renumber the survivors, and do not reuse the retired index for a different database — old flow keys naming it would resolve against unrelated data.

Deciding when it is safe to remove: any flow key on the retired shard becomes permanently unresolvable. If callers hold keys durably (in their own database, in emails, in webhooks), they will get “not found” rather than an error explaining why. Retire the index only once you are content for those keys to be gone.

Moving a Shard to a Different Server

This is not a reshard. If the shard index stays the same, the engine does not care where the database lives: replace the connection string, keep the index, and do an ordinary database migration by whatever means your dialect supports. Restart the fleet to pick up the new connection string.

This is the supported way to grow a shard’s capacity — move it to a bigger server, and adjust its Weight (and MaxConnections, if you set one) — as opposed to adding shards, which grows the number of servers. Vertical growth of one shard needs no coordination beyond the restart; horizontal growth needs the two-phase dance above.

Changing the Shard Count Is a Capacity Decision, Not a Scaling Reflex

Before resharding at all, check that the shard count is what actually binds. Shards exist to add database servers, and every shard carries what one carries alone: 2 shards ×2.03 and 4 shards ×3.83 against a single shard. Scaling is linear in the measured range, so do not stop at two on the assumption that later shards add nothing. → Cloud Benchmarks

Co-locating several shards on one database server buys nothing. The point of a shard is a separate write path; two shards on one server contend for the same one and add overhead for no capacity. If you are not adding a server, you are not adding capacity.

See Deployment → Sharding for the sizing table, and Cloud Benchmarks for the measurements behind it.

When a Shard Is Down

Shards multiply capacity, not availability — each shard’s own availability is its database’s. ListOpen, ListClosed and Purge fail as a whole if any shard errors. ShardInfo is built to answer anyway: it returns a row per shard with that shard’s error recorded on it, so it is the diagnostic to reach for in exactly that situation.

Next: Backup and Restore.

Backward Upgrading Backup and Restore Forward