Repository navigation
fix: bound telemetry truncation for built-in exporters - #881
Open
pcbeingused333 wants to merge 1 commit into
Open
pcbeingused333 wants to merge 1 commit into
pcbeingused333 wants to merge 1 commit into
Conversation
truncateRecord passed bounded=false, so every size-limited exporter except LambdaLogExporter re-serialized the record once per operation. Built-in exporters now use the bounded meter, with a shrink bound that covers both the operations array and the by-name map. A custom render still checks after every drop. Fixes aws#877
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #877
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue Link, if available
#877
Description
Exporting a long execution re-serialized the whole telemetry record once per operation.
truncateRecordcallstruncatewithbounded = false, andMeter.fitsthen measures the record before every step.DurableExecutionTelemetry.exportRecordused that path for every exporter exceptLambdaLogExporter, which already callstruncateMeasured(bounded = true). WithCloudWatchLogsExporter's limit (256,000 bytes) and by-name rendering, a record of 2,000 simple steps measured the record thousands of times. The same path is what #877 timed at about 15 s for 4,000 steps.Built-in exporters now use the bounded meter. The shrink bound covers two copies of each operation, so
OperationsFormat.BOTH(the operations array and the by-name map) and the OTel body, which embeds that JSON once as a string, drop the same operations as checking after every step.LambdaLogExporteris unchanged and still measures its line with one copy. A customrenderis not in the built-in package and still checks after every drop.Demo/Screenshots
N/A
Design
CloudWatchLogsExporterleaves the other size-limited exporters on the slow path. Turningboundedon insidetruncateRecordwould also bound custom renders, which [Bug]: telemetry truncation time grows with the square of the operation count for size-limited exporters other than LambdaLogExporter #877 says must keep exact checks. A binary search over the drop count, as in the Go SDK, can drop operations the linear pass keeps when removing a small result makes the record larger. The bounded meter already in this module skips only checks that cannot succeed.truncateRecordpassesbounded = false, andexportRecordused it for every exporter other thanLambdaLogExporter.Assumptions
None. The size checks measure this module's own JSON (
Json.byteSizeofrender, orLambdaLogExporter.renderedSize). No assumption about CloudWatch, OTel, or any other service.Decisions not covered by the issue or the design note
software.amazon.lambda.durable.telemetry.exporters), not by a new method onTelemetryExporter. Adding that method would change the public API. A custom class in some other package keeps exact checks.OperationsFormat. Array and by-name renderings only need one copy; the extra copy measures a bit more often and drops the same operations. BOTH needs two: with one copy the pass dropped 27 operations where the exact pass dropped 18.Checklist
Testing
Unit Tests
Yes.
TruncationCostTestfails on the unchanged code:cloudWatchByNameShapeMeasuresAFewTimesAndKeepsTheExactOperationsreports 2433 measurements for 2000 operations (the limit in the test is 100).bothFormatAndOtelBodyMatchTheExactPasspasses on the unchanged code because the exact pass is the reference; it guards the two-copy bound (see below). After the fix all three tests pass, the kept operations matchtruncateRecord, and a custom render still measures more than once per operation.mvn -pl telemetry-exporter -am test -Dtest=TruncationCostTest,LambdaLogLineSizeTest,TruncationTest -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=trueinmaven:3.9-eclipse-temurin-21: 13 tests, 0 failures. The same command without-Dtest, still in that image, runs the telemetry-exporter module: 251 tests, 0 failures. No existing test expectation was edited. With the shrink bound forced to one copy,bothFormatAndOtelBodyMatchTheExactPassfails (droppedOperations27 vs 18).Integration Tests
No. Truncation runs in memory before any exporter sends.
Examples
Not applicable.
This change was prepared with the assistance of an AI coding tool (Claude) and submitted by the account owner.