Skip to content

fix: bound telemetry truncation for built-in exporters - #881

Open
pcbeingused333 wants to merge 1 commit into
aws:mainfrom
pcbeingused333:c1842-bounded-truncation
Open

pcbeingused333 wants to merge 1 commit into
aws:mainfrom
pcbeingused333:c1842-bounded-truncation

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

Fixes #877

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

#877

Description

Exporting a long execution re-serialized the whole telemetry record once per operation. truncateRecord calls truncate with bounded = false, and Meter.fits then measures the record before every step. DurableExecutionTelemetry.exportRecord used that path for every exporter except LambdaLogExporter, which already calls truncateMeasured (bounded = true). With CloudWatchLogsExporter's limit (256,000 bytes) and by-name rendering, a record of 2,000 simple steps measured the record thousands of times. The same path is what #877 timed at about 15 s for 4,000 steps.

Built-in exporters now use the bounded meter. The shrink bound covers two copies of each operation, so OperationsFormat.BOTH (the operations array and the by-name map) and the OTel body, which embeds that JSON once as a string, drop the same operations as checking after every step. LambdaLogExporter is unchanged and still measures its line with one copy. A custom render is not in the built-in package and still checks after every drop.

Demo/Screenshots

N/A

Design

Assumptions

None. The size checks measure this module's own JSON (Json.byteSize of render, or LambdaLogExporter.renderedSize). No assumption about CloudWatch, OTel, or any other service.

Decisions not covered by the issue or the design note

  • A built-in exporter is recognized by its package (software.amazon.lambda.durable.telemetry.exporters), not by a new method on TelemetryExporter. Adding that method would change the public API. A custom class in some other package keeps exact checks.
  • Every built-in JSON rendering uses a shrink bound of two copies, rather than a branch per OperationsFormat. Array and by-name renderings only need one copy; the extra copy measures a bit more often and drops the same operations. BOTH needs two: with one copy the pass dropped 27 operations where the exact pass dropped 18.
  • No separate skip was added for operations that have no result. Once a measurement is still over the limit and nothing has been dropped, the bounded meter already skips the later checks in that step. Custom renders still measure every step.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change
  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: the description or commit order shows the regression test failing before the fix and passing after it (rule 5)
  • Every edited test expectation is listed in the description, with the reason (rule 5)

Testing

Unit Tests

Yes. TruncationCostTest fails on the unchanged code: cloudWatchByNameShapeMeasuresAFewTimesAndKeepsTheExactOperations reports 2433 measurements for 2000 operations (the limit in the test is 100). bothFormatAndOtelBodyMatchTheExactPass passes on the unchanged code because the exact pass is the reference; it guards the two-copy bound (see below). After the fix all three tests pass, the kept operations match truncateRecord, and a custom render still measures more than once per operation.

mvn -pl telemetry-exporter -am test -Dtest=TruncationCostTest,LambdaLogLineSizeTest,TruncationTest -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=true in maven:3.9-eclipse-temurin-21: 13 tests, 0 failures. The same command without -Dtest, still in that image, runs the telemetry-exporter module: 251 tests, 0 failures. No existing test expectation was edited. With the shrink bound forced to one copy, bothFormatAndOtelBodyMatchTheExactPass fails (droppedOperations 27 vs 18).

Integration Tests

No. Truncation runs in memory before any exporter sends.

Examples

Not applicable.


This change was prepared with the assistance of an AI coding tool (Claude) and submitted by the account owner.

truncateRecord passed bounded=false, so every size-limited exporter
except LambdaLogExporter re-serialized the record once per operation.
Built-in exporters now use the bounded meter, with a shrink bound that
covers both the operations array and the by-name map. A custom render
still checks after every drop.

Fixes aws#877
@pcbeingused333
pcbeingused333 requested a review from a team November 7, 2026 16:29
@pcbeingused333
pcbeingused333 deployed to ai-pr-review-runtime November 7, 2026 16:29 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 1d64daa7 Deployed Nov 7, 2026 by pcbeingused333 via notify / Generate notification summary #345
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: telemetry truncation time grows with the square of the operation count for size-limited exporters other than LambdaLogExporter

1 participant