Skip to content

Resampler drops samples before its buffer is sized, so the first window after a start is wrong #1471

Open

Description

@malteschaaf

What happened?

The resampling buffer starts as a deque(maxlen=initial_buffer_len) (16 by default). The input sampling period is only estimated, and the buffer only resized, in _ResamplingHelper.resample(), i.e. at a tick, and only once the buffer is full and at least resampling_period.total_seconds() * max_data_age_in_periods samples were received. Until then, every new sample pushes the oldest one out of the full buffer, and the resized buffer is created from the samples that are left. The first resampled value after a start is therefore computed from the last initial_buffer_len samples only.

Minimal reproduction (one sample per second, 60 s resampling period):

from datetime import datetime, timedelta, timezone

from frequenz.sdk.timeseries import ResamplerConfig2
from frequenz.sdk.timeseries._resampling._resampler import _ResamplingHelper

config = ResamplerConfig2(resampling_period=timedelta(seconds=60), max_data_age_in_periods=1)
helper = _ResamplingHelper("repro", config)
t0 = datetime(2026, 1, 1, tzinfo=timezone.utc)

for window in range(3):
    start = t0 + timedelta(seconds=60 * window)
    for i in range(1, 61):  # values 1..60 per window
        helper.add_sample((start + timedelta(seconds=i), float(i)))
    print(f"window {window}: {helper.resample(start + timedelta(seconds=60)).value} (expected 30.5)")
window 0: 52.500 (expected 30.5)   <- mean of the last 16 samples (45..60)
window 1: 30.500 (expected 30.5)
window 2: 30.500 (expected 30.5)

How much this matters depends on how many input samples a resampling window needs (resampling_period / input_period * max_data_age_in_periods when downsampling):

  • Streaming resampling (1 s period, max_data_age_in_periods=3): samples are only lost if a source sends faster than every 3/16 s ≈ 190 ms, and only for the first tick or two after a start.
  • Long downsampling, e.g. a MovingWindow with resampling: a 15-minute window over 1 s data averaged only its last 16 s. In production this made a peak tracker record 30.63 kW for a 15-minute window whose real average was 32.99 kW, and such a value can persist (e.g. as a stored peak).
  • With ResamplerConfig2 the first tick is at the next aligned boundary, so after a start in the middle of a window both the partial window and the first full window are affected: the rate is not estimated at the partial window's tick yet, and at the next tick the buffer only holds the last 16 samples.

What did you expect instead?

All samples within the relevance window are used from the first resampled value on, without having to know and set initial_buffer_len.

Affected version(s)

v1.0.0-rc2211 up to v1.0.0-rc2214 (the resampler code is unchanged since the tick_delay addition).

Affected part(s)

Data pipeline / resampling (frequenz.sdk.timeseries._resampling), MovingWindow.

Extra information

Proposed fix: size the buffer before the first sample is dropped

Keep the bounded, per-timeseries buffer and the existing sizing formula, but trigger the sizing at the moment it is needed: in add_sample(), when the buffer is full and the sampling period is still unknown, estimate the period and resize the buffer before appending the new sample.

def add_sample(self, sample: tuple[datetime, float]) -> None:
    if (
        self._source_properties.sampling_period is None
        and len(self._buffer) == self._buffer.maxlen
    ):
        # The next append would drop the oldest sample: size the buffer first.
        self._estimate_sampling_period(newest=sample[0])
        self._update_buffer_len()
    self._buffer.append(sample)
    ...
  • The period is estimated from the sample timestamps, (newest - sampling_start) / (received_samples - 1), i.e. from the first initial_buffer_len samples (guarding against a zero interval).
  • The buffer length uses the existing formula, capped at max_buffer_len, plus a small headroom. The headroom absorbs the error of an estimate from few samples and the samples of the next window that arrive during tick_delay; today the computed length has no headroom, so with tick_delay such a sample evicts the oldest still-relevant one from a full buffer.
  • The received_samples >= resampling_period.total_seconds() * max_data_age_in_periods condition is no longer needed. It compares a sample count with a number of seconds, which only matches for 1 s input.
  • resample() no longer needs to resize the buffer (or must at least not shrink it below the length including headroom).

For streaming this keeps today's behaviour: the steady-state buffer length is the same (plus the headroom), sources whose relevance window fits into initial_buffer_len samples only shrink their buffer at the 16th sample instead of at a later tick, and the extra cost per add_sample() is one None check and one length comparison, with a single estimate per timeseries. No configuration or API changes are needed. Tests that assert buffer lengths after the buffer filled up will need updating, since the resize happens earlier.

As today, the sampling period is estimated only once, so a source that later sends faster than estimated can still lose samples.

More robust variant: grow on demand instead of estimating

Instead of sizing the buffer from an estimated rate, grow it whenever the sample that is about to be dropped is still relevant, i.e. newer than newest_timestamp - period * max_data_age_in_periods (plus tick_delay), up to max_buffer_len. Optionally the buffer can be shrunk again at a tick to the number of relevant samples plus headroom.

This needs no estimate for sizing (the sampling period would still be estimated for SourceProperties and the upsampling case), and it also handles sources whose rate increases later on, which the estimate-once approach cannot. It would make #241 (only re-calculate the buffer size on significant rate changes) obsolete, since the size follows the actual samples instead of an estimated rate. The cost is a timestamp comparison on every append when the buffer is full, and a larger change to the helper.

Alternative: an expected input period hint

Resampler.add_timeseries() could take an optional expected sampling period to size the buffer up front. MovingWindow already receives input_sampling_period and could pass it on, which fixes the MovingWindow case with either config, while the resampling actor passes nothing and keeps today's behaviour.

This only fixes callers that pass the hint: anyone using the Resampler directly without it (with ResamplerConfig or ResamplerConfig2) still loses samples, as does a MovingWindow whose declared input_sampling_period is slower than the real input. The later resize at a tick would also need to keep the hinted length (or its headroom).

Workaround

Workaround: set initial_buffer_len (and warn_buffer_len/max_buffer_len accordingly) to the expected number of samples in the relevance window. Note that max_buffer_len (1024 by default) also caps long windows, e.g. 15 minutes of 1 s data with max_data_age_in_periods=3 needs 2700 samples.

Activity

  1. added
    part:❓We need to figure out which part is affected
    priority:❓We need to figure out how soon this should be addressed
    type:bugSomething isn't working
    on Oct 15, 2026
  2. malteschaaf commented on Oct 15, 2026

    @malteschaaf
    ContributorAuthor

    Another thought related to this, in case it's relevant for the fix: the buffer length is capped at max_buffer_len (1024 by default). As far as I understand, the cap was added together with the rate-based sizing to keep buffers from growing out of hand if a source sends much faster than expected, and I think some limit still makes sense, especially if the buffer ends up growing on demand.

    A couple of things I noticed while looking into it:

    • The default might be on the low side for longer windows. If I estimated correctly, a buffered sample takes about 136 bytes, so 1024 samples are roughly 136 KiB. Some windows we use need a bit more than that: 15 minutes of 1 s data with max_data_age_in_periods=3 would need 2700 samples (about 360 KiB), 1 hour with max_data_age_in_periods=1 about 3600 (about 480 KiB).
    • When the cap is reached, the values keep looking normal. The resampler logs an error once when the buffer is sized, but afterwards each value is the average of the most recent 1024 samples rather than the whole relevance window, and consumers can't tell from the value itself.

    I wonder whether reaching the cap could be treated more like an error, e.g. rejecting a configuration whose relevance window can't fit (if the input period is known up front), or emitting None instead of a truncated average. Raising the default or deriving it from the configuration might also be options. Happy to hear if there are reasons for the current behaviour that I'm missing.

    On a related note, warn_buffer_len (128) seems to warn for any window longer than about 2 minutes of 1 s data, even when that's intended. Maybe it could warn only when the buffer has to grow well beyond what the configuration needs?

  3. llucax commented on Oct 22, 2026

    @llucax
    Contributor

    This is a known issue and I never thought it was important because it is just an initialization issue. But I was thinking about the live resampling, not the 15 minutes resampling, I can see how it might be a problem there.

    On the other hand, this report says this is affects versions since tick_delay was introduced, so now I'm not sure this is the problem I had in mind or only an issue when using tick_delay (I'm trying to make sense of it but I find these long reports hard to read and follow).

    If we need to fix this, I would go for the less intrusive variant. As we are moving more code to rust, it is likely that the resampler here finally moves there too eventually (or even the microgrid API itself).

  4. added theissue type on Oct 22, 2026
  5. updated Affected Version v1.0.0-rc2211 and up and Effort Medium on Oct 22, 2026
  6. added this to the Untriaged milestone on Oct 22, 2026
  7. added and removed
    part:❓We need to figure out which part is affected
    on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    part:data-pipelineAffects the data pipelinepriority:❓We need to figure out how soon this should be addressedtype:bugSomething isn't working

    Type

    Fields

    Priority

    None yet

    Effort

    Medium

    Affected Version

    v1.0.0-rc2211 and up

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions