storage: the profile store grew to 153 GB with growth measurement unavailable

On 2026-10-16 ~/.tracedecay measured 153 GB before the owner had it deleted. Nothing reported the growth: tracedecay_runtime(doctor_report=true) listed storage.table_growth for both tracedecay.db and global.db as "table payload growth measurement is unavailable", so the doctor had no size finding at all.

What the folder held (listing taken before deletion; sizes per entry were not captured): projects/ with more than 30 proj_* stores, one per registered root, so each agent worktree under /fast/tmp that ran TraceDecay carried its own full code-graph index and sessions shard; lcm-payloads; user-sessions.db and user-sessions.grafeo with their WALs; global.db; maintenance, response-handles, hook_analytics.jsonl, host-bundle-stage, host-components. The live project_sessions shard for proj_a5b3d7e3ebe14ca7 had a 229 MB WAL whose checkpoint stayed pending through 245,264 retry wakes, and the host projection drain had been failing on a corrupt head since Oct 8 (separate lane). A past note records a staging container that grew to 8.6 GB and was loaded whole into heap.

Needed:

Make size observable: fix the table-growth measurement, and make the doctor and tracedecay storage report show per-store sizes (each project store, LCM payloads, Grafeo and WALs) with a finding when a store or the profile crosses a bound.
Retention and GC: project stores whose registered root no longer exists on disk (deleted worktrees); LCM payload retention; Grafeo WAL checkpointing; projection history; hook analytics rotation. Every GC goes through the daemon's own maintenance path, never a direct file delete.
A regression check in the suites: a store that grows without bound under the ingest journey must trip the finding.

Acceptance: after a fresh tracedecay init, 24 hours of normal use on this host shows per-store sizes in the doctor, and no store grows past its bound without a named finding.

Status 2026-10-16. Landed or in the merge queue for this issue:

fix(storage): measure every store and bound each family by purpose #4123 (merged): every store is measured, the profile footprint is censused against per-family bounds, doctor and tracedecay storage report show it, Grafeo WAL segments are sized.
fix(maintenance): stop the shared git dir keeping dead stores alive #4120: a deleted linked worktree's store is no longer kept alive by the shared git directory.
fix(maintenance): collect unregistered project dirs every tick #4126: project directories with no registry row are collected, one bounded page per maintenance tick, never one that holds durable memory.
fix(maintenance): sweep response handles, rotate hook analytics (#4099) #4128: expired response handles are swept every full tick; hook_analytics.jsonl is truncated under its append lock once fully imported past 64 MiB.
fix(rusqlite-runtime): retire ledger rows of exited daemon incarnations #4129: idempotency rows of daemon runs quiet for 7 days are pruned on commit.
fix(graph-db): checkpoint the Grafeo WAL during commits (#4099) #4125: Grafeo WAL is checkpointed on commit once it exceeds max(128 MiB, container size).
fix(maintenance): restart for stores deleted under the daemon (#4099) #4130: a store file deleted or replaced under a running daemon (the 112 GB written into unlinked inodes) is detected each tick and reported as reset-required.
test(storage): ingest growth must trip the footprint finding (#4099) #4127: a session-suite test proves ingest growth trips the footprint finding.
fix(rusqlite-runtime): checkpoint WAL pressure with RESTART #4097: WAL pressure checkpoints with RESTART (interim, until the shared executor's fenced TRUNCATE).

Remaining growth sources are assigned to the storage redesign cutover (docs/plans/storage/README.md, #4118): retention for LCM payloads no summary covers and for unmounted session databases, observation projection history, the disposition ledger, generation staging release (step 9), the response-handle cap, hook events into a telemetry table (step 8), closing a lost store's lane instead of requiring a restart (step 7), and a 24-hour bounded-growth acceptance check on a fresh profile. This issue stays open until those land.

Merge pull request #4125 from ScriptedAlchemy/fix/4099-grafeo-wal-che…

Merge pull request #4127 from ScriptedAlchemy/fix/4099-footprint-regr…

Merge pull request #4128 from ScriptedAlchemy/fix/4099-sidecar-retention

Merge pull request #4130 from ScriptedAlchemy/fix/4099-store-identity

Navigation Menu

Repository navigation

storage: the profile store grew to 153 GB with growth measurement unavailable #4099

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

storage: the profile store grew to 153 GB with growth measurement unavailable #4099

Description

Activity

ScriptedAlchemy commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions