Skip to content

fix(sdk): register processor queue capacity as observable up-down counter - #5896

Open
hardikkaurani wants to merge 3 commits into
open-telemetry:mainfrom
hardikkaurani:fix/processor-queue-capacity-observable-counter
Open

hardikkaurani wants to merge 3 commits into
open-telemetry:mainfrom
hardikkaurani:fix/processor-queue-capacity-observable-counter

Conversation

@hardikkaurani

@hardikkaurani hardikkaurani commented Nov 7, 2026 •

Copy link
Copy Markdown

Description

This pull request fixes missing and inconsistent reporting of processor queue capacity metrics in the OpenTelemetry Python SDK.

Previously, otel.sdk.processor.span.queue.capacity and otel.sdk.processor.log.queue.capacity were implemented as synchronous UpDownCounter instruments and recorded only once during processor initialization. This caused incorrect metric behavior in two scenarios:

  1. Deferred MeterProvider initialization: When a BatchSpanProcessor or BatchLogRecordProcessor was created before metrics.set_meter_provider() was called, the global meter provider could still be a _ProxyMeterProvider. Its proxy instrument discarded the initial measurement without replaying it when the actual MeterProvider was registered.
  2. Delta aggregation temporality: With AggregationTemporality.DELTA, the one-time .add(capacity) measurement was reported during the initial collection but disappeared from subsequent collection cycles.

This change registers both queue capacity metrics as asynchronous ObservableUpDownCounter instruments, ensuring the configured capacity is observed during metric collection rather than recorded only once during initialization.

Changes

  • opentelemetry/sdk/_internal/metrics.py

    • Replaced synchronous queue capacity UpDownCounter registration with asynchronous ObservableUpDownCounter registration.
    • Added a callback that observes the configured queue capacity using Observation.
    • Preserved the appropriate metric names, descriptions, units, and standard processor attributes for span and log record processors.
  • tests/sdk/_logs/test_export.py

    • Added regression coverage for queue capacity metrics when the processor is initialized before the global MeterProvider.
    • Added coverage for DELTA aggregation temporality across consecutive collection cycles.
  • tests/sdk/trace/export/test_export.py

    • Added equivalent regression coverage for BatchSpanProcessor.
    • Verified that queue capacity is observable after deferred MeterProvider initialization and across repeated metric collections.

Validation

  • Added regression tests covering both span and log record processors.
  • Verified queue capacity values and processor attributes under deferred MeterProvider initialization.
  • Added tests for repeated metric collection with AggregationTemporality.DELTA.

Motivation

Queue capacity is a configuration metric that should remain observable throughout the processor's lifecycle. Using an asynchronous observable instrument avoids losing the initial measurement when the global MeterProvider is initialized later and ensures the metric remains available across collection cycles.

@hardikkaurani
hardikkaurani requested a review from a team as a code owner November 7, 2026 15:35
Copilot AI balanced review requested due to automatic review settings November 7, 2026 15:35
@linux-foundation-easycla

linux-foundation-easycla Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: hardikkaurani / name: Hardik Kaurani (9065598)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Waiting on reviewers · refreshed 2026-11-11 10:06 UTC

Review the latest changes.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@hardikkaurani

hardikkaurani commented Nov 7, 2026 •

Copy link
Copy Markdown
Author

Hi @herin049, could you please take a look at this PR when you have some time?

This resolves #5891 by registering otel.sdk.processor.span.queue.capacity and otel.sdk.processor.log.queue.capacity as asynchronous ObservableUpDownCounter callbacks (following the existing pattern in register_queue_size).

This ensures queue capacity is observed across all collection cycles under AggregationTemporality.DELTA, and correctly transfers to the real meter when processors are instantiated before metrics.set_meter_provider(...). Regression test coverage and changelog entry have been added, and EasyCLA is signed.

Thanks!

@knowprasad

Copy link
Copy Markdown
Contributor

I reviewed the changes and tested the new callback behavior against the proxy meter and temporality flows. Switching queue_capacity from a synchronous UpDownCounter to an asynchronous ObservableUpDownCounter cleanly solves both issues described in #5891.

A few specific observations from tracing the implementation:

  1. Deferred Proxy Binding Parity:
    When a BatchSpanProcessor or BatchLogRecordProcessor is instantiated before metrics.set_meter_provider(), the active meter is a _ProxyMeter. Synchronous .add() calls on proxy instruments drop the measurement, meaning capacity was permanently lost. Because _ProxyMeter.create_observable_up_down_counter() retains callback references and forwards them when the real provider is registered, observing capacity via record_queue_capacity ensures the metric is cleanly reported once the concrete MeterProvider is bound.

  2. Temporality Handling Across Cycles:
    Exporters configuring preferred_temporality={UpDownCounter: AggregationTemporality.DELTA} previously caused the synchronous one-time measurement to be reported on the first collection and dropped thereafter. Because ObservableUpDownCounter defaults to cumulative temporality, it continues reporting the full capacity on every collection cycle. The test coverage in test_queue_capacity_observable_delta_temporality also verifies the explicit case: when a reader explicitly requests DELTA for ObservableUpDownCounter, _SumAggregation computes 1024 - 1024 = 0 on subsequent cycles.

  3. Architectural Symmetry with register_queue_size:
    This brings queue_capacity into direct alignment with how register_queue_size already operates in _processor_metrics.py, avoiding the codegen limitations of create_otel_sdk_processor_*_queue_capacity.

  4. Test Hygiene:
    The cleanups in both test_export.py files (self.addCleanup(reset_metrics_globals), self.addCleanup(processor.shutdown), and self.addCleanup(meter_provider.shutdown)) keep the global metric registry isolated across the test suite.

The change is focused, backward-compatible, and the regression tests cover both traces and logs across all collection phases.

@hardikkaurani

Copy link
Copy Markdown
Author

Thanks so much for taking the time to review and test this out, @knowprasad! Really appreciate the detailed feedback and verification on the proxy meter and temporality flows.

@sangkyoonnam sangkyoonnam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The callback mirrors register_queue_size, and the proxy test covers the #5891 case where the processor is built before set_meter_provider.

Small thing on test_queue_capacity_delta_temporality: it sets DELTA only for UpDownCounter, which doesn't apply to the capacity instrument anymore, so what it really shows is that the new instrument ignores that setting. Worth saying that in the test name or a comment?

…comments

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
@hardikkaurani

hardikkaurani commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

Thanks for the review and suggestion, @sangkyoonnam!

Good point — that test specifically reproduces the reader configuration from #5891 (preferred_temporality={UpDownCounter: AggregationTemporality.DELTA}). With the switch to ObservableUpDownCounter, that setting no longer applies to the capacity instrument, which is why it retains its default cumulative temporality across collection cycles.

I have updated the test in both tests/logs/test_export.py and tests/trace/export/test_export.py:

  • Renamed test_queue_capacity_delta_temporality to test_queue_capacity_ignores_up_down_counter_delta_temporality to make the intent immediately clear.
  • Added explicit docstrings and comments detailing that setting UpDownCounter: DELTA is ignored by the observable instrument, contrasting it directly with test_queue_capacity_observable_delta_temporality (which explicitly targets ObservableUpDownCounter: DELTA).

Pushed in commit f403173.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants