[normj]

Summary

Makes Amazon.Lambda.DurableExecution safe to run on Lambda Managed Instances (LMI), where several invocations share one process and Lambda does not stop an invocation that times out. Addresses the source-review findings in #2604.

1. Cancellation lost at teardown (all modes)

TerminationManager completes its task with RunContinuationsAsynchronously, so WorkflowCancellation received the cancel through a threadpool continuation. If the wrapper disposed the cancellation source first, the continuation hit ObjectDisposedException, which was swallowed, and the cancel was lost. The wrapper now cancels synchronously (BeginTeardown) before draining and disposing, and Dispose also cancels before releasing the source.

2. Abandoned work after the invocation ends (all modes)

When a workflow suspends, user work it started but did not await (a step still running, a losing parallel branch) kept running. On LMI the process stays live, so that work runs alongside other invocations.

• The workflow token is cancelled when the invocation ends.
• The token is captured at construction, so stale code that reads it after disposal gets a cancelled token instead of ObjectDisposedException.
• Every IDurableContext operation entry point rejects new operations after teardown with OperationCanceledException.
• The abandoned workflow task's fault is observed, so it can't surface through UnobservedTaskException in another invocation.

3. Invocation deadline (LMI only)

Only when AWS_LAMBDA_MAX_CONCURRENCY is set; on-demand behavior is unchanged.

• The deadline comes from ILambdaContext.RemainingTime. Zero or negative means no deadline, so an unconfigured TestLambdaContext doesn't cancel immediately.
• At the deadline: the workflow token is cancelled, new operations are rejected, and state hydration (GetExecutionStateAsync) is cancelled.
• Checkpoint drain is bounded by the time left until the deadline plus a 2 second cleanup budget. Past that, the flush is cancelled and the batcher worker is abandoned rather than awaited.
• WrapAsync throws TimeoutException. This escapes as an invocation error, the same outcome as an on-demand timeout, and is never recorded as a Failed durable execution, even if user code catches the cancellation and throws its own exception.
• User code gets the full deadline; there's no early-firing margin.

Testing

Unit tests

• New LambdaManagedInstancesTests (48 test methods) drive DurableFunction and its internals concurrently, using an in-memory service client:
  • Concurrent invocations through one shared handler, with every invocation inside a step at the same time: execution ARN, request ID, operation IDs, payloads and checkpoint token chains stay isolated.
  • A peer timing out while a healthy peer keeps running and succeeds.
  • Replay of one execution while a fresh peer is live: completed steps are skipped.
  • Teardown ordering in both modes: cancellation requested before dispose and never lost, token usable after dispose, new operations rejected.
  • The deadline hitting each point in an invocation:
    • during a step, state loading, the final drain, or output serialization;
    • while a START checkpoint is awaited or replay state is restored;
    • with a stalled checkpoint call, or an invocation that is already expired when it starts.
  • Cancellation callbacks registered by user code or the service client that block or throw. Neither runs on SDK threads, and neither can crash the process.
  • Code that converts or swallows the cancellation (step, child context, condition check, map/parallel branches): no FAIL or RETRY is recorded.
  • On-demand: no deadline, and state loading gets a token that can't be cancelled.
• LMI mode and the deadline are driven through internal test seams rather than the process-wide environment variable and real timers:
  • Tests fire the deadline themselves once setup is in place, so setup never races a timer.
  • A timer that isn't armed models a timer callback the thread pool hasn't run yet.
  • Ordering uses signals, not sleeps, and no test asserts on elapsed time.
• Every regression test was checked by reverting its fix and confirming the test fails.
• Amazon.Lambda.DurableExecution.Tests (563) and Amazon.Lambda.DurableExecution.Testing.Tests (154) pass on net8.0 and net10.0, across repeated full runs.

Live testing on Lambda Managed Instances

These tests aren't in CI; provisioning LMI is too slow for it. I ran them by hand in a dev account (us-west-2) against this branch.

Setup

• A capacity provider in the default VPC (x86_64, auto scaling).
• A dotnet10 durable function on it:
  • 15 s timeout, 2 GB memory, JSON logging;
  • PerExecutionEnvironmentMaxConcurrency of 8, ExecutionTimeout of 900 s;
  • published versions, 3 execution environments each.
• The function is an executable (bootstrap) that project-references this branch's Amazon.Lambda.DurableExecution, Amazon.Lambda.RuntimeSupport and Amazon.Lambda.Serialization.SystemTextJson, so the deployed code is exactly the PR.
• It routes to a scenario by name and logs structured markers, including:
  • request ID, process instance and in-flight count;
  • remaining time, thread-pool and heap metrics;
  • every step entry, cancellation, retry attempt and callback.
• The harness starts executions asynchronously (Invoke with --durable-execution-name), polls ListDurableExecutionsByFunction, completes callbacks with SendDurableExecutionCallbackSuccess, then checks two things:
  • CloudWatch Logs and the platform.report records for each invocation;
  • GetDurableExecutionHistory for each execution, to confirm what was actually recorded.

Several invocations always ran in the same process (up to 8 in flight per environment), and resumes often moved to a different environment.

Scenarios
"While young" means while the execution was younger than a set age, measured from when the harness started it. That lets the first invocations hit the 15 s deadline and the retries finish.

• Isolation: dozens of concurrent step → wait → step executions. No log line carried another invocation's request ID, trace ID or executionArn scope.
• Deadline:
  • a cooperative step waiting on its token while young;
  • a step that blocks a thread and ignores the token;
  • a 40-item map with the deadline landing mid-run.
• Cancellation callbacks:
  • a step whose cancellation callback blocks for 20 s, while a parallel sibling suspends;
  • a step whose cancellation callback throws.
• Code that swallows or converts the cancellation:
  • a condition check that swallows the cancellation and returns a state;
  • a step with a retry strategy, and a step inside a child context, each turning the cancellation into its own exception.
• Other paths:
  • parallel suspension with a sibling step still running;
  • durable step retries;
  • callbacks;
  • about 6 MB of checkpoint history, so the resume invocation gets a NextMarker and has to load the rest of its state.
• Warm load: 10 waves of mixed executions over about 100 invocations per process. Between waves, probes forced a full GC and recorded heap size, thread-pool size, OS threads, handles and active timers.

Results

• Every execution succeeded: 227 across the three latest-code runs, and about 80 more in earlier runs during review. No execution failed.
• Timeouts:
  • Cooperative steps were cancelled within ±8 ms of the deadline. Timed-out invocations surfaced as invocation errors, and the service retried them until the execution succeeded.
  • The histories show no StepFailed or retry events caused by a timeout. The swallowing condition check used no attempt, and the converting steps used no retry.
• Map deadline: no item that had already succeeded ran again. Items cut off mid-step re-ran on retry, and each item's StepSucceeded was recorded exactly once.
• Cancellation callbacks:
  • The invocation whose step had a 20 s blocking callback still returned Pending in about 130 ms.
  • The throwing callback was contained and logged at Warning under the right request ID, and the process kept serving work.
• Code that ignores cancellation: the step that blocks a thread kept running after its invocation timed out, while the retry ran the same step in the same process. Its late result was never recorded, but its side effects still happened twice. This is documented under "Not covered".
• Warm load: heap after a forced GC stayed at 1.0 to 1.3 MB per process, and handle and timer counts stayed flat. There were no unobserved task exceptions.

[Copilot]

🟡 Changes recommended

Critical deadline and cancellation races can leave invocations running indefinitely, produce incorrect outcomes, and allow work or checkpoint writes after teardown.

12 open issues

• Recheck cancellation after checkpoint waits before running step code · New
• Prevent queued workflows from starting after the deadline · New
• Check the deadline even when the workflow task wins · New
• Suppress timeout-induced failure and retry checkpoints · New
• Bound hydration when the client ignores cancellation · New
• Recheck the deadline after the final checkpoint drain · New
• Stop dispatching checkpoint batches after cancellation · New
• Derive deadline expiration from the deadline token · New
• Coordinate teardown with in-progress cancellation delivery · New
• Signal deadlines independently of blocking cancellation callbacks · New
• Contain exceptions from cancellation callbacks · New
• Treat an expired invocation deadline as an immediate timeout · New

What changed in this PR

Adds deadline and teardown handling to Amazon.Lambda.DurableExecution for Lambda Managed Instances, addressing part of #2604’s shared-process execution concerns.

Changes:

• Adds invocation-scoped deadlines, timeout reporting, and bounded checkpoint cleanup.
• Cancels abandoned workflow work and rejects new durable operations after teardown.
• Adds concurrency and lifecycle tests, cancellation guidance, and patch release notes.

File	Description
Libraries/​test/​Amazon.Lambda.DurableExecution.Tests/​LambdaManagedInstancesTests.cs	Tests invocation isolation, replay, deadlines, and teardown.
Libraries/​src/​Amazon.Lambda.DurableExecution/​Internal/​WorkflowCancellation.cs	Tracks cancellation, teardown, and LMI deadlines.
Libraries/​src/​Amazon.Lambda.DurableExecution/​Internal/​DurableInvocationOptions.cs	Provides test overrides for LMI detection and cleanup budgets.
Libraries/​src/​Amazon.Lambda.DurableExecution/​Internal/​CheckpointBatcher.cs	Adds timed draining and worker abandonment.
Libraries/​src/​Amazon.Lambda.DurableExecution/​DurableFunction.cs	Integrates deadlines, hydration cancellation, and cleanup.
Libraries/​src/​Amazon.Lambda.DurableExecution/​DurableExecutionHandler.cs	Races workflow execution against deadlines and observes abandoned faults.
Libraries/​src/​Amazon.Lambda.DurableExecution/​DurableContext.cs	Guards operation entry points against teardown.
Libraries/​src/​Amazon.Lambda.DurableExecution/​docs/​core/​cancellation.md	Documents LMI cancellation behavior and cooperative-code limits.
.autover/​changes/​46bb3a38-e1f3-48d1-ab9b-10a36475e901.json	Records the patch release changes.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Cancellation callbacks can indefinitely block teardown or terminate the process, while deadline races can persist incorrect failure or retry checkpoints.

12 open issues

• Cancellation callbacks can exceed the cleanup timeout · New
• Concurrent branches can checkpoint deadline failures as permanent · New
• Workflow-token cancellation can bypass the cancellation filter · New
• Post-body serialization failures can persist after timeout · New
• Condition checks can record outcomes after deadline cancellation · New
• Deadline-token callback exceptions can escape the timer · New
• Recheck cancellation after checkpoint waits before running step code
• Cleanup budget ignores time already spent after the deadline · New
• Output mapping can succeed after the deadline expires · New
• Caught cancellation can leave request faults unobserved · New
• Abandonment leaves pending batch-item completions unresolved · New
• Race test does not trigger or verify the claimed deadline fault · New

11 resolved since last review

• Treat an expired invocation deadline as an immediate timeout
• Contain exceptions from cancellation callbacks
• Signal deadlines independently of blocking cancellation callbacks
• Coordinate teardown with in-progress cancellation delivery
• Derive deadline expiration from the deadline token
• Stop dispatching checkpoint batches after cancellation
• Recheck the deadline after the final checkpoint drain
• Bound hydration when the client ignores cancellation
• Suppress timeout-induced failure and retry checkpoints
• Check the deadline even when the workflow task wins
• Prevent queued workflows from starting after the deadline

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Blocking cancellation callbacks can hang teardown and deadline handling, while deadline races can allow expired workflows to start, succeed, or persist outcomes.

7 open issues

• Recheck deadline after state serialization · New
• Check the monotonic deadline, not timer callback state · New
• Publish deadline state independently of callbacks · New
• Run cancellation callbacks off the teardown thread · New
• Recheck cancellation after checkpoint waits before running step code
• Observe faults after every abandonment cancellation · New
• Register callbacks before triggering the test deadline · New

11 resolved since last review

• Deadline-token callback exceptions can escape the timer
• Condition checks can record outcomes after deadline cancellation
• Post-body serialization failures can persist after timeout
• Workflow-token cancellation can bypass the cancellation filter
• Concurrent branches can checkpoint deadline failures as permanent
• Cancellation callbacks can exceed the cleanup timeout
• Race test does not trigger or verify the claimed deadline fault
• Abandonment leaves pending batch-item completions unresolved
• Caught cancellation can leave request faults unobserved
• Output mapping can succeed after the deadline expires
• Cleanup budget ignores time already spent after the deadline

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Critical deadline-handling races can leave hydration uncancelled and allow step, child, and condition callbacks to start after expiration.

6 open issues

• Check the deadline before starting a child function · New
• Check the deadline before starting an at-most-once step · New
• Check the deadline before starting a replayed condition check · New
• Initialize deadline cancellation before signaling the timer task · New
• Recheck cancellation after checkpoint waits before running step code
• Make the deadline-during-step test deterministic · New

6 resolved since last review

• Run cancellation callbacks off the teardown thread
• Publish deadline state independently of callbacks
• Check the monotonic deadline, not timer callback state
• Recheck deadline after state serialization
• Register callbacks before triggering the test deadline
• Observe faults after every abandonment cancellation

_{Review effort: Balanced}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Copilot]

🟡 Changes recommended

A moderate issue remains in deadline handling because hydration faults after the deadline can escape as SDK errors instead of being consistently translated into the promised timeout exception.

1 open issue

• Post-deadline hydration faults escape timeout translation · New

_{Review effort: Lite}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because the change contains multiple deadline, cancellation, and cleanup race conditions that can allow exceptions or checkpoint work to escape the promised timeout behavior.

No open issues 🎉

1 resolved since last review

• Post-deadline hydration faults escape timeout translation

_{Review effort: Lite}