Repository navigation
Closed
Labels
Description
Summary
The OTel plugins intentionally drop references to some recording spans without ending them when a durable execution returns a non-terminal status.
Affected behavior:
ExecutionOtelPlugindrops open operation spans onPENDINGorRETRY.- Both
ExecutionOtelPluginandInvocationOtelPlugindrop the execution-scopedWorkflowspan on non-terminal invocations.
OpenTelemetry requires every created span to be ended.
Evidence
packages/aws-durable-execution-sdk-python-otel/src/aws_durable_execution_sdk_python_otel/execution_plugin.py, invocation-end handling around lines 275-337packages/aws-durable-execution-sdk-python-otel/src/aws_durable_execution_sdk_python_otel/invocation_plugin.py, workflow handling around lines 438-461packages/aws-durable-execution-sdk-python-otel/tests/test_execution_plugin.py::test_open_operation_span_not_exported_at_invocation_endexplicitly asserts that an operation span is never endedpackages/aws-durable-execution-sdk-python-otel/tests/test_invocation_plugin.py::test_workflow_span_not_exported_on_non_terminalcodifies the same workflow behavior
A focused runtime probe retains the workflow span reference, invokes on_invocation_end with PENDING, and observes workflow.is_recording() is True after plugin state has been reset.
OpenTelemetry requirement
The Trace API states that every span must be ended and that failing to end spans may leak memory or other resources:
https://opentelemetry.io/docs/specs/otel/trace/api/#end
Impact
- Span processors can receive
on_startwithout a correspondingon_end. - Processor or exporter state may remain live across warm Lambda invocations.
- Re-created deterministic spans can trigger multiple starts for the same logical span ID.
- The behavior prevents a strict OpenTelemetry compliance claim.
Acceptance criteria
- Every recording span created by either OTel plugin is ended exactly once.
- Non-terminal workflow and suspended-operation behavior does not rely on abandoning a recording span.
- Cross-invocation workflow and operation correlation remains intact.
- Tests assert that retained span references are no longer recording after invocation cleanup.
- Tests cover
PENDING,RETRY, terminal success, and terminal failure for both plugin variants.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
fix(otel): stop reusing operation span IDs on resume