[Bug]: OTel re-exports already completed STEP spans after invocation retry #892

Expected Behavior

Replaying an already checkpointed terminal STEP must not trigger another SDK-generated operation or attempt span in either OTel view. Span delivery remains best-effort.

Apply this without persisting spans or export acknowledgements, adding checkpoint operations, or changing stored operation results.

Actual Behavior

When Lambda retries an invocation that failed after a step was successfully checkpointed, the Python Execution view can export that completed step again with the same trace/span ID but different timestamps. The step body is correctly skipped. The second span represents reconstruction from checkpoint history, not another step execution.

The requested fix is best-effort suppression of terminal STEP replay telemetry. It does not require persistent span storage or claim exactly-once telemetry delivery.

Steps to Reproduce and Evidence

Verified on AWS Lambda Python 3.13 using Python SDK 3.0.1 and OTel 2.0.0 from commit 4625d01b. A real in-memory OTel exporter captured spans before any external collector could deduplicate them. An outer wrapper recorded allowlisted fields from the actual Lambda invocation input without modifying it.

Each scenario was run in both Execution and Invocation views: six successful durable executions and fourteen Lambda invocations. Every step body ran once; execution history contained one StepStarted and one StepSucceeded per step. All invocation request IDs matched history, and all input and span-completion records were present. Invocation view produced distinct span IDs for its additional segments; it did not exhibit the same span-identity collision.

These are new cloud measurements, not the exact durations of the original report. The probe used an explicit provider, in-memory exporter, and no incoming trace-context extractor, so it verifies the SDK/plugin lifecycle independently of the original collector configuration. FAILED checkpoint replay was not exercised by this cloud run and needs regression coverage.

Why the existing replay check misses this

In the verified revision:

• DurableContext routes an operation in UpdatedOperationIds through the update hook, even when its STEP checkpoint is already terminal on invocation entry.
• The terminal update is converted into OperationEndInfo with is_replayed=False. Checking only that flag therefore does not identify this case.
• Execution-view deduplication is invocation-local, and span timing incorporates the current invocation's placeholder when present. The retry has no original attempt timing. Its recorded checkpoint start/end were equal in these runs, while the first export included the actual attempt interval.

Requested behavior

Distinguish a STEP that was already terminal in the invocation's checkpoint input/history from a STEP that actually executes or reaches a terminal state in this invocation. Do not export an additional logical STEP span merely because the former is encountered through UpdatedOperationIds. Apply that suppression in both bundled OTel views: Invocation view currently avoids a span-ID collision by creating a new segment, but that segment still represents a cached terminal step rather than new work.

The implementation may live at the core/plugin boundary or in OTel handling. Use the operation's provenance/already-terminal-on-entry state, not a blanket check of its current terminal status: a step that just completed in this invocation still needs its span. Do not delete generic plugin terminal-update notifications merely to suppress OTel replay spans.

Acceptance criteria

• A STEP already in SUCCEEDED or FAILED in invocation input/history must not produce another logical STEP operation span, or a synthetic step-attempt span, solely because it is replayed through UpdatedOperationIds. Cover both a listed ID and the ordinary replay path, in both OTel views.
• Apply that decision by operation type STEP, regardless of subtype. Include a non-default/custom STEP subtype regression, rather than filtering only sub_type == "Step".
• Preserve replay behavior: a succeeded step returns its stored value; a failed step reproduces its stored failure; neither executes its body again. The telemetry change must not alter checkpoint contents or operation identity.
• Preserve telemetry for actual work: a new step, a real retry attempt of a nonterminal step, and a terminal checkpoint first produced by that attempt still emit their normal operation/attempt telemetry. Include a retry/resume regression that would fail if all replayed or pre-existing STEP IDs were suppressed.
• Keep invocation spans and invocation outcomes accurate. Test both configured views so a shared-hook change does not silently remove actual invocation or retry-attempt telemetry.
• Do not globally filter terminal operations from UpdatedOperationIds. WAIT, CHAINED_INVOKE, and CALLBACK completion delivery must continue to work, including completions first observed on resume. Include focused non-STEP regression coverage.
• Cover the three-invocation control: A -> wait/PENDING -> B -> invocation failure -> retry. A and B execute once each; terminal replay does not reconstruct another STEP span; the completed wait still reaches its normal completion handling.
• Implement and document this as best-effort replay suppression without persisting spans or export acknowledgements in durable checkpoints.

Related work

• #854 concerns discovering/delivering replay-visible completions. Preserve its handling of externally completed operations. This issue targets already-terminal STEP history, not all replay-visible updates.
• #855 concerns invocation-local export retry. Bounded retry of an export being delivered within the current invocation remains compatible with this change. Suppress reconstruction from a later invocation's terminal STEP history, not retry of a current invocation's live completion export.

Remaining limitation

The original expectation, “export once, or make every copy identical,” is stronger than this fix can guarantee without durable telemetry state. Step checkpointing and span export are separate actions. If an invocation ends after the step checkpoint succeeds but before its span is exported, suppressing terminal replay may leave that step without a span. Re-exporting instead cannot reliably recreate the original local attempt timestamps, attributes, or events from the operation checkpoint alone. Export transports can also deliver duplicate batches.

The bounded objective is therefore to stop avoidable reconstruction of already-terminal STEP spans during replay, while retaining real execution telemetry. It is not a promise of exactly-once export, lossless recovery after interruption, or byte-identical copies across failures.

SDK Version / Python Version

Core 3.0.1 and OTel 2.0.0, source 4625d01bbfa97da67050014851af7eb41c61a842; Lambda Python 3.13. This was still the default branch revision when checked on 2026-10-30.

Is this a regression?

Unknown. No working version was established.

Companion Java implementation: aws/aws-durable-execution-sdk-java#857.