Skip to content

Account every completion write before it runs and bound deadline closes (#1116) - #1195

Merged
rmcdaniel merged 15 commits into
mainfrom
general/1116-completion-budget-deadline-close
Nov 19, 2026
Merged

rmcdaniel merged 15 commits into
mainfrom
general/1116-completion-budget-deadline-close

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Closes the remaining Phase B work of #1116 (split from #930).

Write accounting

Every write a workflow task completion makes is charged against history_transaction_rows before the statement runs:

  • HistoryTransactionMeter::run() registers a before-executing hook on the storage connection, once per connection. While a meter is current, it charges every insert/update/delete: one row per inserted or upserted tuple, one row for any other write. Reads cost nothing. The hook replaces the explicit history and local-activity row charges, so totals are not double counted.
  • Statements that can change several existing rows go through HistoryTransactionMeter::affecting(). It counts the affected rows first with a read bounded by the remaining rows + 1, charges that count, then runs the statement. The wrapped sites are wait projections, child-projection repairs, ChildSupervision::followRetry, the stale-projection cleanup, the memo and search-attribute deletes, and the continue-as-new message transfer.
  • The child-call projection writes after commit, so it is charged one row up front, inside the transaction.
  • docs/architecture/completion-budget.md lists each write family and how it is charged.

A refusal throws before the statement, the transaction rolls back, and the task keeps its lease.

Deadline close (closure semantics change: needs General + Security review before merge)

  • Selection defect. deadlineExpiredRunQuery() used to exclude every expired run with a Ready/Leased workflow task. As a result, an ordinary run with an unclaimed Ready task could stay Pending forever. Now only a live lease defers the close. For an ordinary run whose task is Ready or whose lease has expired, the watchdog closes the run itself, under the run lock and using that task. Opted-in timeout cleanup and cooperative cancellation are unchanged.
  • Bounded close. One close transaction may cancel at most min(history_transaction_size − 1, history_transaction_rows / 5) open items; the default is 4,000. A larger ordinary close works like this:
    • Each batch cancels activities with their tasks, then timers, then other tasks. It writes the same events, in the same order, as the single close.
    • The run stays open and expired, and its task is released to Ready.
    • The worker gets reason: run_deadline_closing.
    • The watchdog, or the in-process executor, continues until the remainder fits. The existing timeoutRun() then closes the run.
    • Every pass shrinks the open work, so a budget refusal can never block a close. Setting both limits to 0 restores the single-transaction close.

Replay

Recorded history does not change. A batched close writes exactly the events the single close writes, and a test checks this by comparing payloads with identities removed. 2.29.x histories replay unchanged.

New evidence: tests/Fixtures/V2/ReplayRegression/legacy-deadline-close-batched-cancellations.json.

  • The fiber runner cold-replays the recorded close history: WorkflowStarted, TimerScheduled, TimerCancelled, WorkflowTimedOut.
  • The embedded consumer then drives an ordinary run past its deadline, with a deadline task that no worker claims. At base the run stays Waiting (the defect). With this change the watchdog closes it, writing exactly that history. This is the counterfactual the corpus validator requires.

Tests

tests/Unit/V2/CompletionWriteFamilyBudgetTest.php:

  • Ten write families. Each completion is checked at its exact row total (admitted) and at one row less (refused). A refusal must leave every table byte-identical and the lease kept.
  • Unmetered writes. No write statement may run outside the meter.
  • Bulk counting. A bulk delete is counted before it runs.
  • Watchdog closes. The watchdog closes a run whose task is Ready and unclaimed, and a run whose lease has expired. A live lease defers the close to the worker.
  • Batched close. A batched close takes 3 passes for 7 open items with a batch of 2, and its history matches the single close.

Review fixes (second round)

  1. Completion row limit derived from the event limit. (Superseded in round three: this now applies to completion_rows, not history_transaction_rows.) Unless configured, it is 103 × history_transaction_size + 5,000 = 520,000 by default. 103 rows per event is the measured maximum of any first-party family: a 100-entry memo or search-attribute upsert. The other families measure:

    Family Rows per event
    Activity 6
    Timer 7
    Child 6.3
    Child inheriting 100 memos + 100 SAs 73
    Local activity 4
    Accepted update failed at close 5.5
    Signal carried by continue-as-new 4

    The 5,000 headroom covers event-less rows: the completion's own rows and continue-as-new message transfer. A configured value, including 0, is honored. CompletionRowBudgetDerivationTest measures each family and fails if any exceeds 103 rows per event.

  2. Early refusal from the command list. CompletionRowEstimate computes per-family row costs, plus inherited and upserted metadata. They are exact for every command family the family tests cover. An over-limit completion is refused with completion_budget_exceeded before any write statement runs. The meter stays the exact backstop for state-driven rows.

  3. Bounded terminal cascade. The final close transaction takes the remaining batch for accepted updates, then parent-close children. Leftovers continue in watchdog passes from a deadline_close_settlement: pending marker on the closing task. Updates go before children, children in link order, and children with a recorded outcome are skipped. The history is equal to a single close.

  4. Watchdog fairness. No-op candidates are no longer selected: cooperative cancellation pending, or timeout cleanup, while a workflow task is open. Selection orders by deadline, then id.

  5. Under-counts.

    • The child-task namespace bulk update now goes through affecting().
    • SQLite writer admission runs unmetered, so it never inflates totals relative to MySQL or PostgreSQL.
    • A guard test asserts that no engine write uses insert-from-select.

Tests added or changed:

  • Families: exact estimate for 10 families; the meter backstop for the state-driven family.
  • Zero writes: an over-limit completion is refused before any write statement.
  • Derivation: the default-limit derivation and the per-family rows-per-event table.
  • Fan-out: a 2.29.1-shaped fan-out of 1,000 children + 1,000 activities is 25,003 rows by the exact estimate, within the 520,000 default. An executed 100 + 100 slice matches the estimate exactly.
  • Bounded cascade: 200 children + 300 updates settle over batches of 100 with the single-close history.
  • Fairness: no-op candidates cannot starve an ordinary close.
  • Guard: no engine write inserts from a select.

Not executed at 1,000 children: on base too, starting N children in one completion costs O(N²) (600 children ≈ 256 s on SQLite), and terminating 1,000 children costs ≈ 2 min. So the 1,000-scale cases are covered by the exact estimate rather than run.

Re-review fixes (third round)

  1. Published-config compatibility. Real-row accounting moved to a new setting, completion_rows (DW_V2_LIMIT_COMPLETION_ROWS).

    • Default: it ships unset, so a published config doesn't pin a number. Unset, it is derived as 103 × events + 5,000 = 520,000.
    • Old key: history_transaction_rows keeps exactly its 2.29 key, meaning and default of 20,000. It charges two rows per event plus local-activity records. The 2.29 charges are restored, and WorkflowHistoryEvent is identical to main.
    • Tests:
      • testAPublished2291ConfigAdmitsWhat2291Admitted checks that a pinned 20,000 admits the 1,000 + 1,000 fan-out through both early checks. An executed 100 + 100 slice charges exactly 2 × events = 800 history rows.
      • testAnExplicitCompletionRowLimitIsHonored checks a configured value and its exact boundary.
      • testThePublishedConfigLeavesTheCompletionRowLimitUnset.
  2. Estimate. The row estimate now counts:

    • the memos and search attributes a continued run inherits;
    • the memo and search attributes a start_service_operation carries (25 + entries);
    • deletes only of keys the run holds, so deleting an absent key costs nothing.

    New exact-boundary families cover continue-as-new with inherited metadata and a memo delete of an absent key.

  3. Settlement lookup. A pending settlement holds the closing task Leased by deadline-close-settlement, with no expiry.

    • Workers can't claim it, task repair skips it because it has no lease expiry, and a late complete() leaves it held.
    • The watchdog reads held tasks by status and owner, in id order, up to the scan limit, and then reads only those runs.
    • testThePendingSettlementLookupDoesNotGrowWithRetainedClosedRuns: the query count is the same with 0 and 60 retained Failed runs, and the lookup SQL is checked.
    • The doc states that the run reads Failed while the settlement is pending.

Settlement-hold fixes (fourth round)

  1. Hold protection. fail() leaves a held task alone, as complete() does. Beyond that, the WorkflowTask model refuses any update that changes a held task's status, lease or owner, and refuses deleting it, unless the settlement itself makes the change. Retention reports deadline_close_settlement_pending and keeps the run. Test: a stale worker's fail() after run_deadline_closing leaves the hold intact, a direct write is refused, and the settlement finishes.
  2. Scoped selection. The held-task lookup applies RequestedTaskScope::applyToTasks, the instance filter and the run ids before the limit. It selects due tasks only, ordered by available_at then id. Test: 30 held tasks in another namespace and a scan limit of 25; the scoped watchdog still settles its own.
  3. Failure handling. A failed pass rolls back. A separate transaction then records last_error, increments attempt_count and the payload failure count, and sets a backoff on available_at (10 s doubling, capped at 1 h). The settlement is never released.
    • Visibility:
      • OperatorMetrics and queue repair visibility report deadline_close_settlements held and stuck counts. A settlement is stuck after 3 failed passes, and a stuck one sets needs_attention.
      • Run describe (RunDetailView) shows deadline_close_settlement: state, failed attempts, last error and next attempt.
    • Test: a forced failure records the attempt, error and backoff. The task is skipped until due, becomes stuck and visible after 3 failures, and the next due pass settles it.
  4. Same-completion delete. A delete of a key written earlier in the same completion is estimated as one row.

Native parity: durable-workflow/server#766.

🤖 Generated with Claude Code

Charge every row a workflow task completion writes against
history_transaction_rows before the statement runs (#1116). The storage
connection's before-executing hook charges each insert, update or delete
issued under the meter; statements that can change several existing rows
are counted first with a read bounded by the remaining rows plus one.
The after-commit child-call projection is charged in advance.

Bound an ordinary run's deadline close: open work larger than one
completion's budget is cancelled in batches that write the single close's
history, the workflow task is released, and the watchdog finishes the
close. The watchdog now closes an ordinary expired run whose workflow task
is Ready and unclaimed or whose lease expired, instead of leaving it open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel rmcdaniel added the agent:general Work owned by General label Nov 18, 2026
rmcdaniel and others added 3 commits November 18, 2026 14:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new replay evidence now drives the embedded consumer through an
ordinary run whose deadline task is never claimed: at the base revision the
watchdog leaves it open, now it closes with the recorded history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close

# Conflicts:
#	CHANGELOG.md
rmcdaniel and others added 2 commits November 18, 2026 16:46
…lose cascade

Review fixes for #1116:

- The default history_transaction_rows now follows the event limit:
  103 rows per event (a full 100-entry memo or search attribute upsert,
  the most any first-party command writes per event) x 5,000 + 5,000 =
  520,000, so it never refuses a completion the event limit admits. A
  configured value is honored as is.
- Before the first write, a completion is checked against an exact
  per-family row estimate from its command list and refused with
  completion_budget_exceeded without a write statement. The meter stays
  the exact backstop for state-driven rows.
- A bounded deadline close also bounds its terminal cascade: accepted
  updates, then parent-close children, at most one batch per transaction,
  continued by the watchdog from a marker on the closing task, with the
  single close's history.
- The watchdog selects no no-op deadline candidates and takes expired runs
  oldest deadline first.
- The child task namespace bulk update is counted before it runs, SQLite
  writer admission is never metered, and no engine write inserts from a
  select.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close

# Conflicts:
#	CHANGELOG.md
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.72617% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.34%. Comparing base (c2a6ca3) to head (3956369).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
src/V2/TaskWatchdog.php 87.65% 10 Missing ⚠️
src/V2/Support/CompletionRowEstimate.php 91.80% 5 Missing ⚠️
src/V2/Support/WorkflowExecutor.php 96.72% 4 Missing ⚠️
src/V2/Support/DefaultWorkflowTaskBridge.php 84.21% 3 Missing ⚠️
src/V2/Support/DeadlineCloseSettlement.php 97.05% 1 Missing ⚠️
src/V2/Support/HistoryTransactionMeter.php 98.00% 1 Missing ⚠️
src/V2/Support/ParentClosePolicyEnforcer.php 96.42% 1 Missing ⚠️
src/V2/Support/PendingUpdateCloser.php 93.75% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##               main    #1195    +/-   ##
==========================================
  Coverage     94.34%   94.34%            
  Complexity     1595     1595            
==========================================
  Files           722      724     +2     
  Lines         92181    92573   +392     
==========================================
+ Hits          86969    87339   +370     
- Misses         5212     5234    +22     
Flag Coverage Δ
v2 94.34% <94.72%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Re-review fixes for #1116:

- Real-row accounting gets its own setting, completion_rows
  (DW_V2_LIMIT_COMPLETION_ROWS), shipped unset and derived from the event
  limit (103 x events + 5,000 = 520,000 by default).
  history_transaction_rows keeps exactly its 2.29 key, meaning (two rows
  per event plus local activity records) and default, so a config
  published from 2.29 behaves the same after upgrading.
- The completion row estimate counts the memos and search attributes a
  continued run inherits, the metadata a service operation carries, and
  only deletes of keys the run holds.
- A bounded close's pending settlement holds its closing task Leased by
  deadline-close-settlement with no expiry. The watchdog finds held tasks by
  status and owner among open tasks, in id order and bounded by the scan
  limit, instead of scanning closed runs. A late completion leaves the held
  task alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel and others added 3 commits November 18, 2026 18:50
Re-review fixes for #1116:

- Only the settlement may release a held closing task: fail() leaves it
  alone like complete(), the task model refuses any other write that
  changes its status, lease or owner or deletes it, and retention keeps
  the run while it is held.
- The watchdog applies its connection, queue, namespace, instance and run
  scope to held tasks before the scan limit, and takes due ones oldest
  first.
- A failed settlement pass records its error, one more attempt and an
  exponential backoff on the held task in its own transaction, and is
  retried; it is never released. Held and stuck (3+ failed passes)
  settlements appear in OperatorMetrics and queue repair visibility, and
  run describe shows the settlement state and last error.
- The row estimate counts a delete of a key written earlier in the same
  completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close

# Conflicts:
#	CHANGELOG.md
…eadline-close' into general/1116-completion-budget-deadline-close
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 0937e49 into main Nov 19, 2026
38 checks passed
rmcdaniel added a commit that referenced this pull request Nov 19, 2026
Brings in the completion-budget deadline close (#1195), the payment race
fixes (#1212) and the merged shared worker-loss driver (#1211).

Refs #1069

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general Work owned by General

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant