ric/omnimem

Watch

Fork 0

Index drift recurs on 6.7.0 — 762 phantom entries across four namespaces #28

New issue

Closed

opened 2026-09-08 13:28:49 +00:00 by ric · 1 comment

ric commented

2026-09-08 13:28:49 +00:00

Owner

Observed on: 6.7.0

What happened

health reported drift on every namespace on a long-running instance:

namespace	index num_docs	actual records	drift
episodic	718	573	145
project	574	521	53
knowledge	2680	2150	530
preference	289	255	34

reindex() cleared all 762. health afterwards returns "drift": {} with index counts matching record counts exactly on all five indexes. Recall was verified unaffected — same top hit at the same score to 9 d.p. before and after.

Why this needs more than the cleanup

reindex() fixed the symptom. Two things suggest the cause is still live:

Uptime was ~110s when drift was first observed, so this is persisted in the index rather than a runtime artefact — it survived a restart.
762 orphans have accumulated since the last manual reindex, across all four namespaces rather than one.

The health docstring already names the likely cause — deletes the search module didn't observe, e.g. keyspace notifications disabled. If that's it, then reindex() is a recurring chore rather than a repair, and any instance whose operator never calls health carries a silently growing set of phantom vectors.

Suggested investigation

Check notify-keyspace-events on the Valkey side, and confirm what valkey-search actually needs configured to observe hash deletes.
If notifications can't be relied on, reconcile at startup: health already does a record scan, so the same comparison could run on boot and either self-heal or log a warning.
Decide whether drift should be surfaced more loudly than a field in health. Nothing currently prompts an operator to look.

Open question worth answering first

What is the actual impact of a phantom entry? If an orphaned vector can be returned as a search hit with no backing hash, this is a correctness bug and should be prioritised accordingly. If it only costs memory and a little recall latency, it's housekeeping. Worth settling before deciding how much of the above to build.

**Observed on:** 6.7.0 ## What happened `health` reported drift on every namespace on a long-running instance: | namespace | index num_docs | actual records | drift | |---|---|---|---| | episodic | 718 | 573 | 145 | | project | 574 | 521 | 53 | | knowledge | 2680 | 2150 | 530 | | preference | 289 | 255 | 34 | `reindex()` cleared all 762. `health` afterwards returns `"drift": {}` with index counts matching record counts exactly on all five indexes. Recall was verified unaffected — same top hit at the same score to 9 d.p. before and after. ## Why this needs more than the cleanup `reindex()` fixed the symptom. Two things suggest the cause is still live: 1. Uptime was ~110s when drift was first observed, so this is persisted in the index rather than a runtime artefact — it survived a restart. 2. 762 orphans have accumulated since the last manual reindex, across all four namespaces rather than one. The `health` docstring already names the likely cause — deletes the search module didn't observe, e.g. keyspace notifications disabled. If that's it, then `reindex()` is a recurring chore rather than a repair, and any instance whose operator never calls `health` carries a silently growing set of phantom vectors. ## Suggested investigation - Check `notify-keyspace-events` on the Valkey side, and confirm what valkey-search actually needs configured to observe hash deletes. - If notifications can't be relied on, reconcile at startup: `health` already does a record scan, so the same comparison could run on boot and either self-heal or log a warning. - Decide whether drift should be surfaced more loudly than a field in `health`. Nothing currently prompts an operator to look. ## Open question worth answering first What is the actual impact of a phantom entry? If an orphaned vector can be returned as a search hit with no backing hash, this is a correctness bug and should be prioritised accordingly. If it only costs memory and a little recall latency, it's housekeeping. Worth settling before deciding how much of the above to build.

ric referenced this issue from a commit

2026-09-08 14:20:27 +00:00

fix: relevance floors for recall and find_skills, phantom guard (#28, #29, #30, #31)

ric referenced this issue from a commit

2026-09-08 14:20:27 +00:00

fix: full held_back text, drift surfacing, stale exemption (#28, #32, #33, #34)

ric referenced this issue from a commit

2026-09-08 14:20:27 +00:00

fix: recalibrate the recall floor against the real embedder, mark the overlap

ric commented

2026-09-08 15:11:02 +00:00

Author

Owner

Fixed in 6.7.1 (f9e8f64, b4822f3, 32e5f9d, 7eaeafc on v6.7.x).

Answering the open question first, because it changes the priority: a phantom could reach a caller. An index entry whose backing hash is gone comes back from FT.SEARCH as a document id and a score with no fields, and recall built a result out of it — real key, empty content, a slot in top_k. So this was a correctness bug, not only housekeeping. Recall now skips those and logs them.

One subtlety worth recording, because the first version of that guard was wrong. It tested "content is empty", justified by remember() rejecting empty content. True for episodic, knowledge and preference — but not for project: compile_project_context(auto_save=True) writes content="" whenever a project has no description yet and keeps its real text in current_state. That made a fully populated project context — the one the documented session-start bootstrap creates — permanently unrecallable, with a log line telling the operator to run reindex(), which cannot help. A phantom is now identified by carrying no fields at all beyond the key and score, which needs no per-namespace knowledge.

On the cause. The shipped compose files have set --notify-keyspace-events AKE since #11 in April, so disabled notifications are not the explanation for drift on a current deployment. health() on the live instance after your reindex() showed "drift": {} with index counts matching record counts exactly across 28 minutes and new writes, so observation is working in steady state. I could not reproduce fresh drift to attribute it, and I have not claimed a cause the evidence doesn't support.

On surfacing. You were right that this is the real problem: drift was only ever a field in health(), and nothing prompts an operator to call health(), which is how 762 orphans accumulated before anyone looked. The comparison moved into store.index_report() so health(), briefing() and a new startup check share one definition, and it is now logged at boot (INDEX_DRIFT_CHECK=false opts out, costs one SCAN of mem:*) and reported in every briefing.

Deliberately not self-healing. Dropping and recreating every index automatically on startup is a much bigger hammer than the problem and reindex() is one call away. Related: positive and negative drift are now described separately, because they are different faults — reindex() clears orphans and does nothing for an index that is merely behind, which is the expected state moments after _migrate_indexes() recreates one. The startup check no longer warns in that case.

Fixed in 6.7.1 (`f9e8f64`, `b4822f3`, `32e5f9d`, `7eaeafc` on `v6.7.x`). **Answering the open question first, because it changes the priority: a phantom could reach a caller.** An index entry whose backing hash is gone comes back from `FT.SEARCH` as a document id and a score with no fields, and recall built a result out of it — real key, empty content, a slot in `top_k`. So this was a correctness bug, not only housekeeping. Recall now skips those and logs them. One subtlety worth recording, because the first version of that guard was wrong. It tested "content is empty", justified by `remember()` rejecting empty content. True for episodic, knowledge and preference — but not for project: `compile_project_context(auto_save=True)` writes `content=""` whenever a project has no description yet and keeps its real text in `current_state`. That made a fully populated project context — the one the documented session-start bootstrap creates — permanently unrecallable, with a log line telling the operator to run `reindex()`, which cannot help. A phantom is now identified by carrying no fields at all beyond the key and score, which needs no per-namespace knowledge. **On the cause.** The shipped compose files have set `--notify-keyspace-events AKE` since #11 in April, so disabled notifications are not the explanation for drift on a current deployment. `health()` on the live instance after your `reindex()` showed `"drift": {}` with index counts matching record counts exactly across 28 minutes and new writes, so observation is working in steady state. I could not reproduce fresh drift to attribute it, and I have not claimed a cause the evidence doesn't support. **On surfacing.** You were right that this is the real problem: drift was only ever a field in `health()`, and nothing prompts an operator to call `health()`, which is how 762 orphans accumulated before anyone looked. The comparison moved into `store.index_report()` so `health()`, `briefing()` and a new startup check share one definition, and it is now logged at boot (`INDEX_DRIFT_CHECK=false` opts out, costs one `SCAN` of `mem:*`) and reported in every briefing. Deliberately **not** self-healing. Dropping and recreating every index automatically on startup is a much bigger hammer than the problem and `reindex()` is one call away. Related: positive and negative drift are now described separately, because they are different faults — `reindex()` clears orphans and does nothing for an index that is merely behind, which is the expected state moments after `_migrate_indexes()` recreates one. The startup check no longer warns in that case.

ric closed this issue

2026-09-08 15:12:28 +00:00

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#28

No description provided.

Rows
Columns