Repository navigation
Description
What happened?
The resampling buffer starts as a deque(maxlen=initial_buffer_len) (16 by default). The input sampling period is only estimated, and the buffer only resized, in _ResamplingHelper.resample(), i.e. at a tick, and only once the buffer is full and at least resampling_period.total_seconds() * max_data_age_in_periods samples were received. Until then, every new sample pushes the oldest one out of the full buffer, and the resized buffer is created from the samples that are left. The first resampled value after a start is therefore computed from the last initial_buffer_len samples only.
Minimal reproduction (one sample per second, 60 s resampling period):
from datetime import datetime, timedelta, timezone
from frequenz.sdk.timeseries import ResamplerConfig2
from frequenz.sdk.timeseries._resampling._resampler import _ResamplingHelper
config = ResamplerConfig2(resampling_period=timedelta(seconds=60), max_data_age_in_periods=1)
helper = _ResamplingHelper("repro", config)
t0 = datetime(2026, 1, 1, tzinfo=timezone.utc)
for window in range(3):
start = t0 + timedelta(seconds=60 * window)
for i in range(1, 61): # values 1..60 per window
helper.add_sample((start + timedelta(seconds=i), float(i)))
print(f"window {window}: {helper.resample(start + timedelta(seconds=60)).value} (expected 30.5)")window 0: 52.500 (expected 30.5) <- mean of the last 16 samples (45..60)
window 1: 30.500 (expected 30.5)
window 2: 30.500 (expected 30.5)
How much this matters depends on how many input samples a resampling window needs (resampling_period / input_period * max_data_age_in_periods when downsampling):
- Streaming resampling (1 s period,
max_data_age_in_periods=3): samples are only lost if a source sends faster than every 3/16 s ≈ 190 ms, and only for the first tick or two after a start. - Long downsampling, e.g. a
MovingWindowwith resampling: a 15-minute window over 1 s data averaged only its last 16 s. In production this made a peak tracker record 30.63 kW for a 15-minute window whose real average was 32.99 kW, and such a value can persist (e.g. as a stored peak). - With
ResamplerConfig2the first tick is at the next aligned boundary, so after a start in the middle of a window both the partial window and the first full window are affected: the rate is not estimated at the partial window's tick yet, and at the next tick the buffer only holds the last 16 samples.
What did you expect instead?
All samples within the relevance window are used from the first resampled value on, without having to know and set initial_buffer_len.
Affected version(s)
v1.0.0-rc2211 up to v1.0.0-rc2214 (the resampler code is unchanged since the tick_delay addition).
Affected part(s)
Data pipeline / resampling (frequenz.sdk.timeseries._resampling), MovingWindow.
Extra information
Proposed fix: size the buffer before the first sample is dropped
Keep the bounded, per-timeseries buffer and the existing sizing formula, but trigger the sizing at the moment it is needed: in add_sample(), when the buffer is full and the sampling period is still unknown, estimate the period and resize the buffer before appending the new sample.
def add_sample(self, sample: tuple[datetime, float]) -> None:
if (
self._source_properties.sampling_period is None
and len(self._buffer) == self._buffer.maxlen
):
# The next append would drop the oldest sample: size the buffer first.
self._estimate_sampling_period(newest=sample[0])
self._update_buffer_len()
self._buffer.append(sample)
...- The period is estimated from the sample timestamps,
(newest - sampling_start) / (received_samples - 1), i.e. from the firstinitial_buffer_lensamples (guarding against a zero interval). - The buffer length uses the existing formula, capped at
max_buffer_len, plus a small headroom. The headroom absorbs the error of an estimate from few samples and the samples of the next window that arrive duringtick_delay; today the computed length has no headroom, so withtick_delaysuch a sample evicts the oldest still-relevant one from a full buffer. - The
received_samples >= resampling_period.total_seconds() * max_data_age_in_periodscondition is no longer needed. It compares a sample count with a number of seconds, which only matches for 1 s input. resample()no longer needs to resize the buffer (or must at least not shrink it below the length including headroom).
For streaming this keeps today's behaviour: the steady-state buffer length is the same (plus the headroom), sources whose relevance window fits into initial_buffer_len samples only shrink their buffer at the 16th sample instead of at a later tick, and the extra cost per add_sample() is one None check and one length comparison, with a single estimate per timeseries. No configuration or API changes are needed. Tests that assert buffer lengths after the buffer filled up will need updating, since the resize happens earlier.
As today, the sampling period is estimated only once, so a source that later sends faster than estimated can still lose samples.
More robust variant: grow on demand instead of estimating
Instead of sizing the buffer from an estimated rate, grow it whenever the sample that is about to be dropped is still relevant, i.e. newer than newest_timestamp - period * max_data_age_in_periods (plus tick_delay), up to max_buffer_len. Optionally the buffer can be shrunk again at a tick to the number of relevant samples plus headroom.
This needs no estimate for sizing (the sampling period would still be estimated for SourceProperties and the upsampling case), and it also handles sources whose rate increases later on, which the estimate-once approach cannot. It would make #241 (only re-calculate the buffer size on significant rate changes) obsolete, since the size follows the actual samples instead of an estimated rate. The cost is a timestamp comparison on every append when the buffer is full, and a larger change to the helper.
Alternative: an expected input period hint
Resampler.add_timeseries() could take an optional expected sampling period to size the buffer up front. MovingWindow already receives input_sampling_period and could pass it on, which fixes the MovingWindow case with either config, while the resampling actor passes nothing and keeps today's behaviour.
This only fixes callers that pass the hint: anyone using the Resampler directly without it (with ResamplerConfig or ResamplerConfig2) still loses samples, as does a MovingWindow whose declared input_sampling_period is slower than the real input. The later resize at a tick would also need to keep the hinted length (or its headroom).
Workaround
Workaround: set initial_buffer_len (and warn_buffer_len/max_buffer_len accordingly) to the expected number of samples in the relevance window. Note that max_buffer_len (1024 by default) also caps long windows, e.g. 15 minutes of 1 s data with max_data_age_in_periods=3 needs 2700 samples.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Affected Version
Projects
- StatusShow more project fieldsTo do
Activity
malteschaaf commentedon Oct 15, 2026
Another thought related to this, in case it's relevant for the fix: the buffer length is capped at
max_buffer_len(1024 by default). As far as I understand, the cap was added together with the rate-based sizing to keep buffers from growing out of hand if a source sends much faster than expected, and I think some limit still makes sense, especially if the buffer ends up growing on demand.A couple of things I noticed while looking into it:
max_data_age_in_periods=3would need 2700 samples (about 360 KiB), 1 hour withmax_data_age_in_periods=1about 3600 (about 480 KiB).I wonder whether reaching the cap could be treated more like an error, e.g. rejecting a configuration whose relevance window can't fit (if the input period is known up front), or emitting
Noneinstead of a truncated average. Raising the default or deriving it from the configuration might also be options. Happy to hear if there are reasons for the current behaviour that I'm missing.On a related note,
warn_buffer_len(128) seems to warn for any window longer than about 2 minutes of 1 s data, even when that's intended. Maybe it could warn only when the buffer has to grow well beyond what the configuration needs?llucax commentedon Oct 22, 2026
This is a known issue and I never thought it was important because it is just an initialization issue. But I was thinking about the live resampling, not the 15 minutes resampling, I can see how it might be a problem there.
On the other hand, this report says this is affects versions since
tick_delaywas introduced, so now I'm not sure this is the problem I had in mind or only an issue when usingtick_delay(I'm trying to make sense of it but I find these long reports hard to read and follow).If we need to fix this, I would go for the less intrusive variant. As we are moving more code to rust, it is likely that the resampler here finally moves there too eventually (or even the microgrid API itself).