v4: Valkey index drift — orphaned vectors not cleaned up on memory deletion #11

New issue

Closed

opened 2026-04-07 19:38:53 +00:00 by ric_harvey · 1 comment

ric_harvey commented

2026-04-07 19:38:53 +00:00

(Migrated from codeberg.org)

Summary

The health endpoint reports index counts directly from Valkey's search indexes, but these counts can diverge significantly from the actual number of memory records in the system. Running memory_audit reveals the true picture.

Observed discrepancy

Source	episodic	project	knowledge	Total
`health` (Valkey index)	347	192	754	1,293
`memory_audit` (actual records)	—	—	—	372

That's ~921 orphaned index entries — vectors present in the Valkey search index with no corresponding memory record.

The web UI dashboard (which queries records directly) showed 235 knowledge entries, consistent with the audit, and much lower than the 754 reported by health.

Root cause (suspected)

When memories are deleted, the Valkey search index entries are not being removed. This compounds over time, especially after bulk deletes, restores, or expired entries being purged. The index grows without bound while the record store stays in sync.

Proposed fix

Add a reindex command (or integrate into memory_audit) that:

Iterates all keys in each Valkey search index (idx:episodic, idx:project, idx:knowledge)
Checks whether a corresponding record exists
Removes any index entries with no backing record

This could be exposed as:

A new reindex MCP tool
An option on memory_audit (e.g. --fix or auto_clean=True)
A scheduled maintenance task (could hook into the existing briefing-interval maintenance introduced in v3.6.0)

The health endpoint should ideally report record counts, not raw index counts — or report both separately so the drift is visible.

Impact

health gives misleading counts, making it hard to trust reported index sizes
The web UI dashboard appears correct (queries records not indexes)
No data loss — records are intact, only the index has phantom entries

## Summary The `health` endpoint reports index counts directly from Valkey's search indexes, but these counts can diverge significantly from the actual number of memory records in the system. Running `memory_audit` reveals the true picture. ## Observed discrepancy | Source | episodic | project | knowledge | Total | |--------|----------|---------|-----------|-------| | `health` (Valkey index) | 347 | 192 | 754 | 1,293 | | `memory_audit` (actual records) | — | — | — | 372 | That's **~921 orphaned index entries** — vectors present in the Valkey search index with no corresponding memory record. The web UI dashboard (which queries records directly) showed 235 knowledge entries, consistent with the audit, and much lower than the 754 reported by `health`. ## Root cause (suspected) When memories are deleted, the Valkey search index entries are not being removed. This compounds over time, especially after bulk deletes, restores, or expired entries being purged. The index grows without bound while the record store stays in sync. ## Proposed fix Add a `reindex` command (or integrate into `memory_audit`) that: 1. Iterates all keys in each Valkey search index (`idx:episodic`, `idx:project`, `idx:knowledge`) 2. Checks whether a corresponding record exists 3. Removes any index entries with no backing record This could be exposed as: - A new `reindex` MCP tool - An option on `memory_audit` (e.g. `--fix` or `auto_clean=True`) - A scheduled maintenance task (could hook into the existing briefing-interval maintenance introduced in v3.6.0) The `health` endpoint should ideally report record counts, not raw index counts — or report both separately so the drift is visible. ## Impact - `health` gives misleading counts, making it hard to trust reported index sizes - The web UI dashboard appears correct (queries records not indexes) - No data loss — records are intact, only the index has phantom entries

ric_harvey commented

2026-07-08 11:11:11 +00:00

(Migrated from codeberg.org)

Closing — this shipped, matching the proposed fix almost exactly, and was verified against the current v5 code during the 2026-07-08 audit:

reindex(namespace?) MCP tool (v5.1.1): drops and recreates the search index for a namespace so valkey-search rebuilds from existing hashes only, clearing phantom entries. Data-safe — only the index is touched. Returns before/after num_docs, the actual record count, and removed_phantoms. (The valkey-py .dropindex() wrapper turned out to be unreliable, so it uses FT.DROPINDEX directly.)
health() reports both sides plus the drift (v5.1.1, extended to the preference namespace): per-namespace index num_docs, the authoritative SCAN-based record count, and a drift entry whenever they diverge — so the discrepancy in this issue's table is now visible at a glance rather than misleading. As of v5.3.0 the record counts come from a single keyspace scan instead of one per namespace.

The "orphaned vectors accumulate silently" failure mode is what reindex + the drift report were built for.

Closing — this shipped, matching the proposed fix almost exactly, and was verified against the current `v5` code during the 2026-07-08 audit: - **`reindex(namespace?)` MCP tool** (v5.1.1): drops and recreates the search index for a namespace so valkey-search rebuilds from existing hashes only, clearing phantom entries. Data-safe — only the index is touched. Returns before/after `num_docs`, the actual record count, and `removed_phantoms`. (The valkey-py `.dropindex()` wrapper turned out to be unreliable, so it uses `FT.DROPINDEX` directly.) - **`health()` reports both sides plus the drift** (v5.1.1, extended to the preference namespace): per-namespace index `num_docs`, the authoritative SCAN-based record count, and a `drift` entry whenever they diverge — so the discrepancy in this issue's table is now visible at a glance rather than misleading. As of v5.3.0 the record counts come from a single keyspace scan instead of one per namespace. The "orphaned vectors accumulate silently" failure mode is what `reindex` + the drift report were built for.

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#11

No description provided.

Rows
Columns