find_skills() returns every skill regardless of score — 0.0013 match presented as a result #31
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Observed on: 6.7.0
Problem
find_skills()returns every compiled skill regardless of score.find_skills("python")returned all three skills in the store:preferences-localopentofu-localwcag-accessibility-localA score of 0.0013 is indistinguishable from no match. None of these is a Python skill — there isn't one — so the correct answer to this query is an empty list.
Why this is worse than the equivalent in recall()
find_skills()is a discovery step: the documented flow is find, thenget_skill()on the winner. It returns a ranked list with no signal that the top of that list is itself a bad match, so "the winner" is whatever sorted first. An agent following the intended workflow here loadspreferences-localat 0.23 and treats it as the relevant skill for Python work.Skills are also loaded whole and are much larger than a memory, so a wrong pick is expensive in a way a stray recall hit isn't.
Suggested fix
Same shape as #30 — a floor, returning
[]when nothing clears it. Skills are few and their descriptions are long and human-written, so the score distribution differs from the memory namespaces and probably wants its own threshold rather than sharing recall's.Given the load-the-winner workflow, this one may also want a
confidencesignal or an explicit "no matching skill" response so a caller can tell the difference between a weak best match and a good one.Fixed in 6.7.1 (
f9e8f64onv6.7.x).SKILL_MIN_SCORE(default 0.25) gates the semantic pass, and an empty list is now the answer when nothing covers the work —find_skills("python")on a store with no Python skill returns[]and says so, rather than handing backpreferences-localat 0.2341.You were right that skills want their own threshold rather than sharing recall's, though not quite for the reason in the issue. It isn't that skill descriptions are long — it's that the comparison is a different one: a query about the work at hand against a short, deliberately written summary of what a skill is for. That distribution separates cleanly, where a query against a memory's raw content does not (see #30, where the two overlap and a hard cut was not defensible).
Calibrated against the three skills on the live store, so these are real numbers rather than a guess:
"opentofu"→ opentofu-local"accessibility"→ wcag"what git forge and commit message style"→ preferences"a modal dialog that works with a keyboard and screen reader"→ wcag"python"→ preferences"k8s ingress + cert-manager"→ opentofuLowest true positive 0.3042 against highest false positive 0.2341 puts 0.25 in the gap with about 0.05 either side. Worth noting if this is ever retuned: the lowest true positive is the verbose query. Length dilutes cosine similarity, so the floor has to clear the worst honest query, not the best one.
Exact domain hits are identity matches, never scored and never gated.
On the confidence signal you asked for — added, because you're right that the load-the-winner workflow needs it: each entry carries
confidenceofhigh(exact domain, or semantic ≥ 0.45) orlow, and a result set that is entirely low says so in a note. The agent instructions now tell it that an empty list means no stored skill covers the work rather than that discovery failed, and that alowentry must have its description read before loading, because a skill loads whole and a wrong pick costs far more context than a stray recall hit.