Repository navigation
[Fix] Do not create a deletion resource manager just to close it when deleting a region - #19314
luoluoyuyu wants to merge 1 commit into
Conversation
… deleting a region With IoTConsensusV2, deleting a data region called DeletionResourceManager.getInstance(), which is a computeIfAbsent and therefore created an empty manager on demand. Its constructor allocates the deletion buffer and starts the persist thread, which immediately blocks on the empty deletion queue. deleteDALFolderAndClose() then closed the manager, and close() interrupts the idle persist thread with shutdownNow(), so the expected shutdown interrupt was logged as WARN ... Interrupted when waiting for taking DeletionResource from blocking queue to serialize. once per deleted region and without a stack trace. deleteDALFolderAndClose() now closes the manager only when it already exists (getInstanceIfPresent) and always removes the region's DAL folder without instantiating a manager. In addition, the take() interruption is logged at DEBUG when the buffer is already closed and all deletions are flushed. This covers managers that do exist (e.g. the region had pipe deletions): a genuine interruption, or a close that timed out with unflushed deletions, still logs at WARN. Reproduce: drop a database or table on an IoTConsensusV2 data region and observe one WARN per deleted region, together with a DAL folder that is created and deleted within the same milliseconds. After the fix the WARN is gone while the region, its DAL folder and the deletion buffer are still closed and removed as before. Tests: DeletionResourceManagerTest covers closing without a manager (no manager is created, the leftover DAL folder is removed), closing an existing manager, and the absence of the WARN (the expected interrupt is logged at DEBUG) when closing an idle manager.
|
|
||
| /** Closes the manager of the data region if it exists, then removes its DAL folder. */ | ||
| public static void closeAndRemoveDAL(int dataRegionId) { | ||
| DeletionResourceManager manager = getInstanceIfPresent(dataRegionId); |
There was a problem hiding this comment.
Please serialize this lookup/close/remove operation with getInstance(). getInstanceIfPresent() is only a point-in-time read: if a concurrent getInstance(dataRegionId) creates and registers a manager after this returns null but before removeDALFolder, this call skips manager.close() and deletes the new manager DAL files while leaving its persist thread and map entry alive. This is reachable from PipeHistoricalDataRegionTsFileAndDeletionSource.start(), which can obtain the DataRegion reference before deletion and call getInstance() under writeLock(); writeLock() does not check deleted, so a caller queued behind markDeleted() can still proceed. Please coordinate manager creation with region teardown or prevent new creation before folder removal.
There was a problem hiding this comment.
Update: after an offline discussion about this PR's size, the branch is back to the minimal WARN fix and no longer changes how manager creation is coordinated with region teardown. You're right that the race described above is real; I'll handle it in a separate change (serialize manager creation with the region teardown and stop creating managers for deleted regions). If you'd rather have that coordination in this PR, let me know and I can add it back.
7308cd6 to
5d86bfd
Compare
b70691e to
7308cd6
Compare
Problem
With
IoTConsensusV2data regions, deleting a region logsonce per deleted region and without a stack trace. A suite that drops databases/tables therefore produces one WARN per deleted region (e.g. 1860 WARNs for 1860 region deletions).
StorageEngine.deleteDataRegion()->DataRegion.deleteDALFolderAndClose()calledDeletionResourceManager.getInstance(), which is acomputeIfAbsentand therefore created an empty manager even when nothing had ever registered a deletion. The constructor creates the DAL folder, allocates the deletion buffer and starts the persist thread, which immediately blocks on the empty deletion queue.deleteDALFolderAndClose()then closed the manager, andclose()interrupts the idle persist thread withshutdownNow(), so the expected shutdown interrupt was logged as WARN. The manager is normally created lazily only when a pipe deletion arrives (PipeInsertionDataNodeListener.listenToDeleteData), so for a region without pipes this was a pure create-then-close cycle: DAL folder created and deleted within the same milliseconds, and the buffer/thread built only to be torn down.Fix
DataRegion.deleteDALFolderAndClose()now usesDeletionResourceManager.closeAndRemoveDAL(), which closes the manager only if it already exists (getInstanceIfPresent(), nocomputeIfAbsent) and always removes the region's DAL folder without creating a manager.PageCacheDeletionBuffer.persistDeletion()logs thetake()interruption at DEBUG when the buffer is already closed and all deletions are flushed. This also covers managers that do exist (a region that had pipe deletions): a genuine interruption, or a close that timed out with unflushed deletions, still logs at WARN.Behavior
Tests
New
DeletionResourceManagerTest:testDeleteDALFolderDoesNotCreateManager: a leftover DAL folder is removed andgetInstanceIfPresent()stays null.testCloseExistingManagerRemovesItAndDALFolder: an existing manager is closed, removed from the map and its DAL folder is deleted.testClosingIdleManagerDoesNotWarnOnExpectedInterrupt: attaches a logback appender toPageCacheDeletionBuffer, closes an idle manager and asserts the expected shutdown interrupt is logged at DEBUG and not WARN. Verified to fail before the fix.Regression run:
DeletionResourceManagerTest,DeletionResourceTest,DeletionRecoverTest,PageCacheDeletionBufferTest,DataRegionTest,StorageEngineTest— 72 tests, 0 failures (1 pre-existing skip).mvn spotless:applyapplied.