Skip to content

[Fix] Do not create a deletion resource manager just to close it when deleting a region - #19314

Open
luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:fix/deletion-close-warning
Open

luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:fix/deletion-close-warning

Conversation

@luoluoyuyu

@luoluoyuyu luoluoyuyu commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Problem

With IoTConsensusV2 data regions, deleting a region logs

WARN ... [pool-1-IoTDB-DAL-Serialize(group-N)-1] ...PageCacheDeletionBuffer$PersistTask - Interrupted when waiting for taking DeletionResource from blocking queue to serialize.

once per deleted region and without a stack trace. A suite that drops databases/tables therefore produces one WARN per deleted region (e.g. 1860 WARNs for 1860 region deletions).

StorageEngine.deleteDataRegion() -> DataRegion.deleteDALFolderAndClose() called DeletionResourceManager.getInstance(), which is a computeIfAbsent and therefore created an empty manager even when nothing had ever registered a deletion. The constructor creates the DAL folder, allocates the deletion buffer and starts the persist thread, which immediately blocks on the empty deletion queue. deleteDALFolderAndClose() then closed the manager, and close() interrupts the idle persist thread with shutdownNow(), so the expected shutdown interrupt was logged as WARN. The manager is normally created lazily only when a pipe deletion arrives (PipeInsertionDataNodeListener.listenToDeleteData), so for a region without pipes this was a pure create-then-close cycle: DAL folder created and deleted within the same milliseconds, and the buffer/thread built only to be torn down.

Fix

  • DataRegion.deleteDALFolderAndClose() now uses DeletionResourceManager.closeAndRemoveDAL(), which closes the manager only if it already exists (getInstanceIfPresent(), no computeIfAbsent) and always removes the region's DAL folder without creating a manager.
  • PageCacheDeletionBuffer.persistDeletion() logs the take() interruption at DEBUG when the buffer is already closed and all deletions are flushed. This also covers managers that do exist (a region that had pipe deletions): a genuine interruption, or a close that timed out with unflushed deletions, still logs at WARN.

Behavior

Scenario Before After
Delete a region with no pipe/DAL activity (IoTConsensusV2) creates buffer + thread + DAL dir, closes them, logs 1 WARN no manager created, DAL dir removed, no WARN
Delete a region whose manager exists and is idle WARN on the expected shutdown interrupt closed as before, expected interrupt logged at DEBUG
Unexpected interrupt, or close timed out with unflushed deletions WARN WARN (unchanged)

Tests

New DeletionResourceManagerTest:

  • testDeleteDALFolderDoesNotCreateManager: a leftover DAL folder is removed and getInstanceIfPresent() stays null.
  • testCloseExistingManagerRemovesItAndDALFolder: an existing manager is closed, removed from the map and its DAL folder is deleted.
  • testClosingIdleManagerDoesNotWarnOnExpectedInterrupt: attaches a logback appender to PageCacheDeletionBuffer, closes an idle manager and asserts the expected shutdown interrupt is logged at DEBUG and not WARN. Verified to fail before the fix.

Regression run: DeletionResourceManagerTest, DeletionResourceTest, DeletionRecoverTest, PageCacheDeletionBufferTest, DataRegionTest, StorageEngineTest — 72 tests, 0 failures (1 pre-existing skip). mvn spotless:apply applied.

… deleting a region

With IoTConsensusV2, deleting a data region called
DeletionResourceManager.getInstance(), which is a computeIfAbsent and therefore created an empty
manager on demand. Its constructor allocates the deletion buffer and starts the persist thread,
which immediately blocks on the empty deletion queue. deleteDALFolderAndClose() then closed the
manager, and close() interrupts the idle persist thread with shutdownNow(), so the expected
shutdown interrupt was logged as

  WARN ... Interrupted when waiting for taking DeletionResource from blocking queue to serialize.

once per deleted region and without a stack trace.

deleteDALFolderAndClose() now closes the manager only when it already exists
(getInstanceIfPresent) and always removes the region's DAL folder without instantiating a manager.
In addition, the take() interruption is logged at DEBUG when the buffer is already closed and all
deletions are flushed. This covers managers that do exist (e.g. the region had pipe deletions):
a genuine interruption, or a close that timed out with unflushed deletions, still logs at WARN.

Reproduce: drop a database or table on an IoTConsensusV2 data region and observe one WARN per
deleted region, together with a DAL folder that is created and deleted within the same
milliseconds. After the fix the WARN is gone while the region, its DAL folder and the deletion
buffer are still closed and removed as before.

Tests: DeletionResourceManagerTest covers closing without a manager (no manager is created, the
leftover DAL folder is removed), closing an existing manager, and the absence of the WARN (the
expected interrupt is logged at DEBUG) when closing an idle manager.
@luoluoyuyu luoluoyuyu changed the title [Bug] Do not create a deletion resource manager just to close it when deleting a region Fix spurious WARN when closing an idle deletion resource manager Nov 18, 2026
@luoluoyuyu luoluoyuyu changed the title Fix spurious WARN when closing an idle deletion resource manager [Fix] Do not create a deletion resource manager just to close it when deleting a region Nov 18, 2026

/** Closes the manager of the data region if it exists, then removes its DAL folder. */
public static void closeAndRemoveDAL(int dataRegionId) {
DeletionResourceManager manager = getInstanceIfPresent(dataRegionId);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please serialize this lookup/close/remove operation with getInstance(). getInstanceIfPresent() is only a point-in-time read: if a concurrent getInstance(dataRegionId) creates and registers a manager after this returns null but before removeDALFolder, this call skips manager.close() and deletes the new manager DAL files while leaving its persist thread and map entry alive. This is reachable from PipeHistoricalDataRegionTsFileAndDeletionSource.start(), which can obtain the DataRegion reference before deletion and call getInstance() under writeLock(); writeLock() does not check deleted, so a caller queued behind markDeleted() can still proceed. Please coordinate manager creation with region teardown or prevent new creation before folder removal.

@luoluoyuyu luoluoyuyu Nov 18, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: after an offline discussion about this PR's size, the branch is back to the minimal WARN fix and no longer changes how manager creation is coordinated with region teardown. You're right that the race described above is real; I'll handle it in a separate change (serialize manager creation with the region teardown and stop creating managers for deleted regions). If you'd rather have that coordination in this PR, let me know and I can add it back.

@luoluoyuyu
luoluoyuyu force-pushed the fix/deletion-close-warning branch from 7308cd6 to 5d86bfd Compare November 18, 2026 07:00
@luoluoyuyu luoluoyuyu changed the title [Fix] Do not create a deletion resource manager just to close it when deleting a region [Fix] Avoid the spurious WARN when closing a deletion resource manager Nov 18, 2026
@luoluoyuyu
luoluoyuyu force-pushed the fix/deletion-close-warning branch 3 times, most recently from b70691e to 7308cd6 Compare November 18, 2026 07:32
@luoluoyuyu luoluoyuyu changed the title [Fix] Avoid the spurious WARN when closing a deletion resource manager [Fix] Do not create a deletion resource manager just to close it when deleting a region Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants