daemon: high idle CPU from session refresh scheduler, store over size bound, repeated corruption retries #4481

Summary

On an idle macOS machine, the daemon uses about 170–300% CPU for a long time. One project store is more than twice its size bound. The log also shows repeated SQLite corruption warnings and a Pi discovery warning every 2 minutes.

Environment

• tracedecay 1.0.0-beta.90+93164b3f05447886afacf61f8643ead23e8e4c02.dirty
• macOS (Darwin 27.0.0), 10 CPUs, 32 GB RAM
• Daemon runs as a launchd user service; uptime about 12 hours at capture
• 8 registered projects

1. High CPU while idle

No agent session was active. The scheduler log shows only skipped tasks (no_new_session_activity, scheduler_interval_not_elapsed).

• ps and top -l 3 -s 2 show 145–297% CPU for the daemon, sustained over 2+ minutes.
• sample (3 s): most of the active stacks are in pread, __psynch_mutexwait and _platform_memmove. The binary has no symbols, so I cannot name the functions.
• In the current log (about 32k lines after a rotation), the warning native task held its worker for one poll past 100ms appears 14,405 times. Nearly all come from one site:

14371 spawn_site=crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/registry.rs:372:40
   12 spawn_site=crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/registry.rs:398:40

• native runtime woke a timer late also appears (89 times; examples: late=882ms, late=405ms).

My guess: the session-temporal refresh scheduler re-reads large session stores in a loop. The largest project sessions.db is 9.0 GB, and the profile user-sessions.db is 3.4 GB. This is not confirmed.

2. Memory (no confirmed leak)

• RSS: 1,707,520 KB → 1,740,464 KB over 2 minutes (+33 MB), sampled every 10 s.
• vmmap --summary: physical footprint 1.4 G, peak 1.6 G. Most of it is IOAccelerator (1.4 G resident). The malloc zones total only 29 MB.
• Doctor: admission pressure 1176 MiB of a 16384 MiB limit; 7 owners retain 19 MiB measured, 1 unmeasured.
• retained_parses of one worktree reports unmeasured bytes.

The window is too short to show a slow leak. The IOAccelerator share may be expected (a GPU model?), but nothing in the runtime telemetry reports it.

3. Project store over its bound

From tracedecay tool runtime --args '{"doctor_report":true}':

project store ... (root present); graph container 487.2 MB, replay pool 49.5 MB: 22.4 GB of 10.0 GB bound
WAL .../sessions.grafeo.wal: 487.1 MB of 128.0 MB bound
profile total: 34.9 GB of 64.0 GB bound

Disk use for that store: sessions.db 9.0 GB, tracedecay.sealed 8.8 GB, code-index-v1 4.2 GB.

The doctor shows these as plain entries, not as warnings or errors. Nothing appears to enforce the bound. Also, table payload growth is unmeasured: the store is too large to walk within the measurement deadline.

Another project's root is gone from disk, but its 484 MB store stays: (root missing on disk): 484.3 MB of 10.0 GB bound.

Related: #4099.

4. SQLite corruption warnings on a project sessions.db

272 times, from 2026-10-23T06:59:55Z to 09:11:59Z:

WARN tracedecay_global_db::observation_projection::rebuild: projection hit SQLite corruption; discarding the output state cache and retrying error=projection store is corrupt: read projection output owner failed: SQLite advance query failed: database disk image is malformed observation="sha256:1eca9317..." database=~/.tracedecay/projects/<project>/sessions.db
WARN tracedecay_host_admission::projection_drain: projection store operation failed during host drain error=projection store is corrupt: discard projection output state cache failed: SQLite execute batch failed: database disk image is malformed ...

The same observation retried 272 times. The warning has not appeared since. Now session_temporal_health reports 0 for every finding, but status: partial (bounded_row_probe_incomplete on every check). quick_check_ok is null. So I cannot say if the file is healthy now.

Related: #4152.

5. Pi discovery warning every ~2 minutes

WARN tracedecay_sessions::runtime::hosts::pi: additional Pi discovery failures were bounded provider="pi" failure_count=174 reported_failures=16

11,080 times in the current log. The count stays at 174, so the same failing files are retried each cycle. This makes the log larger and possibly uses CPU too.

Suggested next steps

• Back off or cache the refresh at session_temporal_refresh_scheduler/registry.rs:372 when nothing changed.
• Report a doctor warning when a store is over its bound, and compact or prune it.
• Stop retrying the same corrupt observation without limit; mark it as poisoned after N tries.
• Remember failed Pi files and skip them until they change.
• Report IOAccelerator / GPU memory in tracedecay_runtime.