Repository navigation
Account every completion write before it runs and bound deadline closes (#1116) - #1195
Merged
Merged
Conversation
Charge every row a workflow task completion writes against history_transaction_rows before the statement runs (#1116). The storage connection's before-executing hook charges each insert, update or delete issued under the meter; statements that can change several existing rows are counted first with a read bounded by the remaining rows plus one. The after-commit child-call projection is charged in advance. Bound an ordinary run's deadline close: open work larger than one completion's budget is cancelled in batches that write the single close's history, the workflow task is released, and the watchdog finishes the close. The watchdog now closes an ordinary expired run whose workflow task is Ready and unclaimed or whose lease expired, instead of leaving it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new replay evidence now drives the embedded consumer through an ordinary run whose deadline task is never claimed: at the base revision the watchdog leaves it open, now it closes with the recorded history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close # Conflicts: # CHANGELOG.md
…lose cascade Review fixes for #1116: - The default history_transaction_rows now follows the event limit: 103 rows per event (a full 100-entry memo or search attribute upsert, the most any first-party command writes per event) x 5,000 + 5,000 = 520,000, so it never refuses a completion the event limit admits. A configured value is honored as is. - Before the first write, a completion is checked against an exact per-family row estimate from its command list and refused with completion_budget_exceeded without a write statement. The meter stays the exact backstop for state-driven rows. - A bounded deadline close also bounds its terminal cascade: accepted updates, then parent-close children, at most one batch per transaction, continued by the watchdog from a marker on the closing task, with the single close's history. - The watchdog selects no no-op deadline candidates and takes expired runs oldest deadline first. - The child task namespace bulk update is counted before it runs, SQLite writer admission is never metered, and no engine write inserts from a select. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close # Conflicts: # CHANGELOG.md
This was referenced Nov 18, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1195 +/- ##
==========================================
Coverage 94.34% 94.34%
Complexity 1595 1595
==========================================
Files 722 724 +2
Lines 92181 92573 +392
==========================================
+ Hits 86969 87339 +370
- Misses 5212 5234 +22
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Re-review fixes for #1116: - Real-row accounting gets its own setting, completion_rows (DW_V2_LIMIT_COMPLETION_ROWS), shipped unset and derived from the event limit (103 x events + 5,000 = 520,000 by default). history_transaction_rows keeps exactly its 2.29 key, meaning (two rows per event plus local activity records) and default, so a config published from 2.29 behaves the same after upgrading. - The completion row estimate counts the memos and search attributes a continued run inherits, the metadata a service operation carries, and only deletes of keys the run holds. - A bounded close's pending settlement holds its closing task Leased by deadline-close-settlement with no expiry. The watchdog finds held tasks by status and owner among open tasks, in id order and bounded by the scan limit, instead of scanning closed runs. A late completion leaves the held task alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close
Re-review fixes for #1116: - Only the settlement may release a held closing task: fail() leaves it alone like complete(), the task model refuses any other write that changes its status, lease or owner or deletes it, and retention keeps the run while it is held. - The watchdog applies its connection, queue, namespace, instance and run scope to held tasks before the scan limit, and takes due ones oldest first. - A failed settlement pass records its error, one more attempt and an exponential backoff on the held task in its own transaction, and is retried; it is never released. Held and stuck (3+ failed passes) settlements appear in OperatorMetrics and queue repair visibility, and run describe shows the settlement state and last error. - The row estimate counts a delete of a key written earlier in the same completion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-budget-deadline-close # Conflicts: # CHANGELOG.md
…eadline-close' into general/1116-completion-budget-deadline-close
2 of 14 tasks
…on-budget-deadline-close # Conflicts: # CHANGELOG.md
|
This was referenced Nov 19, 2026
rmcdaniel
added a commit
that referenced
this pull request
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes the remaining Phase B work of #1116 (split from #930).
Write accounting
Every write a workflow task completion makes is charged against
history_transaction_rowsbefore the statement runs:HistoryTransactionMeter::run()registers a before-executing hook on the storage connection, once per connection. While a meter is current, it charges everyinsert/update/delete: one row per inserted or upserted tuple, one row for any other write. Reads cost nothing. The hook replaces the explicit history and local-activity row charges, so totals are not double counted.HistoryTransactionMeter::affecting(). It counts the affected rows first with a read bounded by the remaining rows + 1, charges that count, then runs the statement. The wrapped sites are wait projections, child-projection repairs,ChildSupervision::followRetry, the stale-projection cleanup, the memo and search-attribute deletes, and the continue-as-new message transfer.docs/architecture/completion-budget.mdlists each write family and how it is charged.A refusal throws before the statement, the transaction rolls back, and the task keeps its lease.
Deadline close (closure semantics change: needs General + Security review before merge)
deadlineExpiredRunQuery()used to exclude every expired run with a Ready/Leased workflow task. As a result, an ordinary run with an unclaimed Ready task could stay Pending forever. Now only a live lease defers the close. For an ordinary run whose task is Ready or whose lease has expired, the watchdog closes the run itself, under the run lock and using that task. Opted-in timeout cleanup and cooperative cancellation are unchanged.history_transaction_size− 1,history_transaction_rows/ 5) open items; the default is 4,000. A larger ordinary close works like this:reason: run_deadline_closing.timeoutRun()then closes the run.Replay
Recorded history does not change. A batched close writes exactly the events the single close writes, and a test checks this by comparing payloads with identities removed. 2.29.x histories replay unchanged.
New evidence:
tests/Fixtures/V2/ReplayRegression/legacy-deadline-close-batched-cancellations.json.Tests
tests/Unit/V2/CompletionWriteFamilyBudgetTest.php:Review fixes (second round)
Completion row limit derived from the event limit. (Superseded in round three: this now applies to
completion_rows, nothistory_transaction_rows.) Unless configured, it is 103 ×history_transaction_size+ 5,000 = 520,000 by default. 103 rows per event is the measured maximum of any first-party family: a 100-entry memo or search-attribute upsert. The other families measure:The 5,000 headroom covers event-less rows: the completion's own rows and continue-as-new message transfer. A configured value, including 0, is honored.
CompletionRowBudgetDerivationTestmeasures each family and fails if any exceeds 103 rows per event.Early refusal from the command list.
CompletionRowEstimatecomputes per-family row costs, plus inherited and upserted metadata. They are exact for every command family the family tests cover. An over-limit completion is refused withcompletion_budget_exceededbefore any write statement runs. The meter stays the exact backstop for state-driven rows.Bounded terminal cascade. The final close transaction takes the remaining batch for accepted updates, then parent-close children. Leftovers continue in watchdog passes from a
deadline_close_settlement: pendingmarker on the closing task. Updates go before children, children in link order, and children with a recorded outcome are skipped. The history is equal to a single close.Watchdog fairness. No-op candidates are no longer selected: cooperative cancellation pending, or timeout cleanup, while a workflow task is open. Selection orders by deadline, then id.
Under-counts.
affecting().Tests added or changed:
Not executed at 1,000 children: on base too, starting N children in one completion costs O(N²) (600 children ≈ 256 s on SQLite), and terminating 1,000 children costs ≈ 2 min. So the 1,000-scale cases are covered by the exact estimate rather than run.
Re-review fixes (third round)
Published-config compatibility. Real-row accounting moved to a new setting,
completion_rows(DW_V2_LIMIT_COMPLETION_ROWS).history_transaction_rowskeeps exactly its 2.29 key, meaning and default of 20,000. It charges two rows per event plus local-activity records. The 2.29 charges are restored, andWorkflowHistoryEventis identical to main.testAPublished2291ConfigAdmitsWhat2291Admittedchecks that a pinned 20,000 admits the 1,000 + 1,000 fan-out through both early checks. An executed 100 + 100 slice charges exactly 2 × events = 800 history rows.testAnExplicitCompletionRowLimitIsHonoredchecks a configured value and its exact boundary.testThePublishedConfigLeavesTheCompletionRowLimitUnset.Estimate. The row estimate now counts:
start_service_operationcarries (25 + entries);New exact-boundary families cover continue-as-new with inherited metadata and a memo delete of an absent key.
Settlement lookup. A pending settlement holds the closing task Leased by
deadline-close-settlement, with no expiry.complete()leaves it held.testThePendingSettlementLookupDoesNotGrowWithRetainedClosedRuns: the query count is the same with 0 and 60 retained Failed runs, and the lookup SQL is checked.Settlement-hold fixes (fourth round)
fail()leaves a held task alone, ascomplete()does. Beyond that, theWorkflowTaskmodel refuses any update that changes a held task's status, lease or owner, and refuses deleting it, unless the settlement itself makes the change. Retention reportsdeadline_close_settlement_pendingand keeps the run. Test: a stale worker'sfail()afterrun_deadline_closingleaves the hold intact, a direct write is refused, and the settlement finishes.RequestedTaskScope::applyToTasks, the instance filter and the run ids before the limit. It selects due tasks only, ordered byavailable_atthen id. Test: 30 held tasks in another namespace and a scan limit of 25; the scoped watchdog still settles its own.last_error, incrementsattempt_countand the payload failure count, and sets a backoff onavailable_at(10 s doubling, capped at 1 h). The settlement is never released.deadline_close_settlementsheld and stuck counts. A settlement is stuck after 3 failed passes, and a stuck one setsneeds_attention.RunDetailView) showsdeadline_close_settlement: state, failed attempts, last error and next attempt.Native parity: durable-workflow/server#766.
🤖 Generated with Claude Code