[bug/rfe] Rebuild or mark repos stale when zoekt shard files are missing but DB marks repos indexed #1210

Summary

If the zoekt index directory is deleted while the Sourcebot database and repo clones remain intact, Sourcebot starts successfully but does not automatically rebuild the missing shards. Repos remain marked as indexed in the DB, while zoekt has no shard files to search, so search returns incomplete or empty results from an otherwise healthy instance.

This matters for Kubernetes deployments that want to place only .sourcebot/index on more performant ephemeral/local storage while keeping DB/repos on durable storage.

Version

Observed on:

• Sourcebot image: ghcr.io/sourcebot-dev/sourcebot:v4.17.1

Background / Use Case

For performance reasons, we are evaluating a hybrid storage layout:

• durable PVC for /data
• node-local / ephemeral storage for /data/.sourcebot/index

The zoekt index is derived/rebuildable, while the DB and repo clones should remain durable. This can improve cold/random read performance for large zoekt indexes, but pod replacement would delete only the index directory.

In that scenario, Sourcebot should ideally detect that indexed repos no longer have shard files and enqueue rebuilds, or otherwise mark those repos as not indexed.

Current Behavior

When /data/.sourcebot/index is empty but the DB still has repos with fresh indexedAt values:

• Sourcebot starts normally.
• zoekt tolerates the empty index directory.
• The repo remains marked as indexed/completed in the DB.
• No new indexing job is scheduled.
• Search returns zero/incomplete results.
• Manual reindex through /api/index-repo successfully rebuilds the missing shard.

Expected Behavior

On startup, or during repo-index scheduling, Sourcebot should detect repos that are marked indexed in the DB but have no corresponding zoekt shard files.

Possible expected behaviors:

• enqueue index jobs for affected repos,
• set indexedAt = null for affected repos so the existing scheduler picks them up,
• or expose a supported startup/recovery command that rebuilds all missing index shards.

Relevant Code Paths

From a quick read of the code:

• packages/backend/src/constants.ts

  • REPOS_CACHE_DIR = DATA_CACHE_DIR/repos
  • INDEX_CACHE_DIR = DATA_CACHE_DIR/index

• packages/backend/src/index.ts

  • creates the repos and index directories at startup if absent

• packages/backend/src/repoIndexManager.ts

  • scheduleIndexJobs() selects repos where indexedAt is null or older than reindexIntervalMs
  • cleanupOrphanedDiskResources() removes shard files with no corresponding DB repo, but does not detect the inverse: DB repo exists and is marked indexed, but shard files are missing

• packages/backend/src/zoekt.ts

  • indexGitRepository() invokes zoekt-git-index -index ${INDEX_CACHE_DIR}

Suggested Fix

Add startup or scheduler reconciliation that compares DB repo state with shard files in INDEX_CACHE_DIR.

For each repo with indexedAt != null, verify that at least one shard exists with that repo's shard prefix. If no shard exists, either:

1. enqueue an index job for that repo, or
2. mark the repo as stale by clearing indexedAt, allowing the existing scheduler to enqueue it.

This would make derived/ephemeral index storage safe to use with durable DB/repos.

To reproduce

Reproduction

Use a small Sourcebot instance with a local file:// repo.

Example config:

{
  "connections": {
    "local-repos": {
      "type": "git",
      "url": "file:///data/local-repos/*"
    }
  },
  "settings": {
    "reindexIntervalMs": 86400000,
    "reindexRepoPollingIntervalMs": 1000,
    "maxRepoIndexingJobConcurrency": 1
  }
}

The long reindexIntervalMs is intentional so we can distinguish immediate missing-shard recovery from the normal periodic reindex.

Steps:

1. Create a local git repo under /data/local-repos/testrepo.
2. Start Sourcebot.
3. Wait for the repo to index successfully.
4. Confirm:
   • a shard exists under /data/.sourcebot/index
   • the repo has indexedAt set
   • search returns a result from the repo
5. Delete only the zoekt shard files:

rm -f /data/.sourcebot/index/*.zoekt \
      /data/.sourcebot/index/*.zoekt.meta \
      /data/.sourcebot/index/*.tmp

6. Restart Sourcebot with the same DB and repo data.
7. Observe that no new index job is created and search returns no results.

Manual recovery works:

curl -s -X POST http://localhost:3060/api/index-repo \
  -H "Content-Type: application/json" \
  -d '{"repoId":1}'

After this, the shard is rebuilt and search works again.

Sourcebot deployment information

ghcr.io/sourcebot-dev/sourcebot:v4.17.1

Additional information

No response