recall() has no relevance floor — top_k fills with unrelated results #30
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Observed on: 6.7.0
Problem
recall()fillstop_kregardless of whether the remaining candidates are relevant. There is no score floor, so a query with two good matches and nothing else still returns five results.Concrete run — query: "valkey-search index drift and FT.DROPINDEX behaviour",
top_k=5:dropindex()postmortemget_fields_multi()projected batch fetchThere's a clear cliff after result 2. Results 3 and 4 share no subject matter with the query at all — they're matching on generic "search" and "index" tokens.
Why it matters
Recall output goes straight into an agent's context. Three irrelevant results is three chunks of tokens spent, plus a live risk of the agent trying to make one of them relevant — the WWDC Spotlight note is about search UI, so it isn't obviously noise to a model skimming for connections. That's a worse failure than returning nothing.
The cost is asymmetric: an empty or short result set is cheap and honest, while padding is expensive and occasionally misleading.
top_kreads naturally as a ceiling, not a quota.Suggested fix
A configurable score floor (env var, consistent with
RECALL_EXPAND_QUERIES) applied after ranking, defaulting to something around 0.4 based on the above. Returning fewer thantop_kshould be normal.If a hard cut feels too blunt given scores aren't calibrated across namespaces, a softer version: keep returning them but mark anything below the floor with a
weak_match: truefield, so the consumer can decide. A relative floor (drop anything below some fraction of the top score) would also handle the cliff shape here, though it behaves badly when the best hit is itself poor.Related: #31 —
find_skills()has the same problem, likely the same fix.Fixed in 6.7.1 (
f9e8f64,9466069,32e5f9d,7eaeafc,8d647cfonv6.7.x) — but not with the 0.4 floor suggested here, and the reason matters.The scores in the table are adjusted, not raw.
recall()reportsadjusted_score, which is why the top hit reads 1.133 — not a reachable cosine. A floor has to gate the raw similarity, because that is the only number in the pipeline measuring "is this about the same subject"; the multipliers stacked on top encode policy (this is old, this was abandoned, this is a derived fact whose source should outrank it), and gating on those would silently hide the two classes deliberately scored down — extracted facts at surface 0.5 from #20, and abandoned experiences at 0.1 — which are exactly what should still come back when nothing better matches. So 0.4 was a cliff read off a different scale from the one being compared against.Measured on the shipped ONNX embedder, over query/memory pairs written in this repo's own style:
They overlap, so no single cut is both safe and useful. A floor at 0.4 kept 6 of 8 correct answers — a user experiences that as recall having forgotten something they know is stored, with nothing to say a filter fired. A floor low enough to keep them all leaves the worst noise in.
So the shipped behaviour is both halves:
RECALL_MIN_SCORE(default 0.15) drops only what is unambiguously noise.top_kis now a ceiling, and an empty return is a normal answer.RECALL_WEAK_SCORE(default 0.35) flags everything in the overlap band asweak_match: true, and the agent instructions tell it to read a weak match but not build on it and not to go looking for a connection.That second part is your own alternative from this issue, and on this evidence it is the honest one: the row count was the symptom, but the failure you described was an agent constructing a relationship to the WWDC Spotlight note because nothing said not to. Both knobs are configurable —
RECALL_MIN_SCORE=0.4is one env var away if you'd rather have the hard cut.Abandoned-approach warnings and reinstate candidates are exempt from both: the first fires on a keyword scan before the query is even embedded, the second matches on its own hints, so neither earned its place on a similarity it can fairly be judged against.
The web UI search opts out entirely and marks instead. The floor's whole justification is agent context cost and agent misreading; a person reading a results page pays neither, and an empty page for a memory they know is stored is the worse answer.
Two things found in review, both reproduced before fixing:
adjusted_scorewhile the floor readsscore, so a fact at 1.0 against a differently-worded source at 0.39 returned nothing at all. Enrichment routinely distils short facts out of long memories, which is exactly that shape.remember()accepts 50,000 characters un-chunked, so a long write-up embeds as its first 256 tokens; and a short keyword query against long prose scores low by construction.