Skip to content

fix(library-udf): query-scoped UDF state and inclusive gap-fill endpoint - #19305

Open
NUAAwqy wants to merge 1 commit into
apache:masterfrom
NUAAwqy:fix/udf-state-isolation-and-gapfill-endpoint
Open

NUAAwqy wants to merge 1 commit into
apache:masterfrom
NUAAwqy:fix/udf-state-isolation-and-gapfill-endpoint

Conversation

@NUAAwqy

@NUAAwqy NUAAwqy commented Nov 17, 2026

Copy link
Copy Markdown

Fixes 19304 — six defects, each reproduced against master before fixing and covered by new unit tests.

library-udf dprofile — query-scoped state (4 files)

  • UDTFMvAvg: promote windowSum to an instance field and reset it in beforeStart; it was a transform() local, so the "sliding mean" was actually (new − evicted) / w.
  • UDTFMinMax: skip non-finite rows entirely so the timestamps/values tables stay in lockstep (same approach as UDTFZScore).
  • UDTFSegment: move the two collected lists from static to instance fields — UDTF instances are per-query, the static lists let overlapping queries erase each other.
  • UDAFPercentile: move the 4 exact-count maps from static to instance state; PercentileOperations.initialize/insertExact now take the owning UDAFPercentile (same pattern as UDTFDistinct), so the shared singleton no longer reassigns static state.

UDTFHistogram — overflow-safe bucketing
Keep the original formula for finite gaps (zero behaviour change, covered by per-bucket assertions); when end - start overflows to ±Inf (auto boundaries) fall back to a stable interval-centre based mapping that is monotonic, reaches both endpoint buckets, and degenerates to the old clamp when start == end.

calc-commons — inclusive gap-fill endpoint
hasRemainingGapInPreviousGroup() compared currentTime < endTime while the group-switch path and fillGaps treat endTime as inclusive; the last gap point at endTime was silently dropped when the final real row sat at endTime − step.

Tests: 10 new library-udf tests via a shared lifecycle-fake harness (UdfTestFakes), 1 new datanode gap-fill operator case (testGapFillWoGroupWoMoOperator3) — mvn -pl library-udf test 29/29 green, spotless clean; GapFillWoGroupWoMoOperatorTest 3/3 green. The MvAvg and gap-fill fixes were also regression-checked by reverting the source change and observing the new tests fail, then re-applying.

Six defects, each reproduced against master before fixing:

- UDTFMvAvg kept windowSum as a transform() local, reducing the sliding
  mean to (new - evicted) / w; promote it to an instance field
- UDTFMinMax appended timestamps unconditionally but values only when
  finite, desynchronising both tables after a NaN row; skip such rows
- UDTFSegment stored collected points in static fields, letting
  overlapping queries erase each other's state; use instance fields
- UDAFPercentile kept the exact-count maps in static Maps shared by the
  singleton operations object; move them to instance state and pass the
  owning UDF into the operations methods (UDTFDistinct pattern)
- UDTFHistogram overflowed end-start to +Inf with default boundaries so
  every value landed in bucket 0; use an overflow-safe, monotonic
  mapping that keeps the original formula for finite gaps
- AbstractGapFillOperator used currentTime < endTime in the terminal
  fill guard while the endpoint is inclusive everywhere else, dropping
  the gap row at endTime when the last real row sat at endTime - step

New tests: 10 in library-udf (shared UdfTestFakes lifecycle harness),
1 in datanode (GapFillWoGroupWoMoOperatorTest#testGapFillWoGroupWoMoOperator3).
mvn -pl library-udf test: 29/29; GapFillWoGroupWoMoOperatorTest: 3/3.
spotless:check clean on both modules.

Signed-off-by: NUAAwqy <146524024+NUAAwqy@users.noreply.github.com>
@HTHou
HTHou requested a review from JackieTien97 November 18, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant