Skip to content

[Performance] Reduce aligned memtable write overhead on wide devices - #19310

Open
luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:fix/aligned-write-hotpath
Open

luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:fix/aligned-write-hotpath

Conversation

@luoluoyuyu

@luoluoyuyu luoluoyuyu commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR reduces the write-path cost of aligned (multi-measurement) timeseries on devices with many columns. The previous code made several per-row operations scale with the device-wide column count C, even when an insert touched one measurement.

  • Sparse aligned writes: AlignedWritableMemChunk now passes a compact column-index mapping (destColumns) to AlignedTVList instead of materializing device-wide Object[C] / BitMap[C] reorder arrays. Value copying and bitmap masking iterate only the columns in the insert plan. Columns that already hold a value array in the target block are still explicitly null-marked for the rows this write does not touch.
  • No per-row schema stream: AbstractMemTable.createAlignedMemChunkGroupIfNotExistAndGet builds its filtered schema list only inside the computeIfAbsent lambda, so existing devices no longer pay a stream/lambda allocation per insert.
  • Cached structure RAM cost: AlignedTVList caches calculateContainerRamCost(null) and invalidates it on structural changes (extendColumns, expandValues, clearValue, clearBitMap, deleteColumn, releaseNonQueryColumns, clone/move, sort metadata). getStructureRamSize() is now O(1) on the common path instead of two O(C) estimates per row.
  • Lazy null prefix for new columns: newly added value columns use a NullPrefixedValueList that represents existing blocks as implicit nulls and only materializes a block when it is first written.

Example

For a device with 1000 measurements, writing one row with one measurement:

  • Before: each row allocated new Object[1000] and new BitMap[1000] (about 16 KB of reference arrays), then scanned all 1000 device columns twice for value copy/null marking.
  • After: the write allocates Object[1], BitMap[1] and int[1], and copies/masks only the written column. The first getStructureRamSize() after a structural change still computes the full cost, but subsequent calls in the same write return the cached value.

Behavior is unchanged: unwritten measurements remain null for the new row, including when their value arrays were materialized earlier in the same block.

Validation

mvn -pl iotdb-core/datanode -am \
  -Dtest=AlignedTVListTest,AlignedWritableMemChunkTest,AlignedTVListIteratorTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Result: 51 tests run, 0 failures, 0 errors.

New tests cover sparse column mapping with bitmaps/results across a block boundary, structure-cost cache invalidation, lazy prefixes over multiple blocks, and single-measurement writes to a 10,000-column device.

Sparse aligned writes now pass only the columns touched by the insert plan to AlignedTVList instead of materializing device-wide value and bitmap arrays. Newly added value columns keep their implicit null prefix lazy, and the structure RAM cost is cached and invalidated on structural changes so per-row memory reconciliation no longer re-estimates every column.

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of commit 3aa3bef9aaf0650508deb9fc76000daadbce0683 (the local PR snapshot and confirmed remote head match). Two RAM-accounting findings are attached inline. I did not rerun the Maven tests; the PR description reports 51 passing tests.

private final List<Integer> materializedColumns = new ArrayList<>();
// Marks which columns belong to the sparse write currently in progress, so the null-marking pass
// can skip the columns whose nulls are handled together with their values.
private long[] columnWriteEpoch;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Include the new write metadata in RAM accounting. getStructureRamSize() delegates to calculateContainerRamCost(), but that calculation does not include this long[] or the new materializedColumns list. A 10,000-column TVList now carries about 80 KB of epoch entries alone that calculateRamSize() does not report, so memory reservations can undercount wide lists. Please account for these structures and add a test against the expected size.

List<Object> columnValue =
existingBlockCount == 0
? new ArrayList<>()
: new NullPrefixedValueList(existingBlockCount);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Account for the lazy prefix as allocated, not logical, slots. calculateContainerRamCost() measures every value column with listRamCostWithReferences(), which sizes an Object[] from List.size(). Here the size includes implicitNullCount even while prefix is null, so adding C columns after B existing blocks still reports O(C * B) references despite allocating only O(C) lazy wrappers. Since this estimate feeds TVList memory accounting, sparse extensions can trigger premature memory pressure and lose the intended benefit. Please account for the actual prefix/tail storage and invalidate the cached cost when the prefix is materialized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants