ric/omnimem

Watch

Fork 0

Broad domains compile from project narrative, not generalisable rules — 'python' can never clear reinforcement #33

New issue

Closed

opened 2026-09-08 13:29:57 +00:00 by ric · 2 comments

ric commented

2026-09-08 13:29:57 +00:00

Owner

Observed on: 6.7.0

Observation

compile_skill(domain='python', mode='propose') found a pool of 29 candidates and produced zero rules — nothing recurred across the min_reinforcement=2 threshold. The gate behaved correctly. The interesting part is what the candidates were:

"Community PR merged cleanly despite significant changes to maintenance.py on bot…"
"Root cause was a data structure mismatch — find_all_duplicates() returned list[l…"
"HINCRBY creates fields with value 1 if absent, so no backfill migration needed f…"
"Leveraged existing find_all_duplicates() and _has_negation_pair() functions from…"

These aren't Python rules. They're OmniMem project history that happens to carry a python tag. Some are Python-adjacent facts about Valkey commands; others are narrative about what happened in a particular PR.

Why this won't fix itself

The domain will never compile, and lowering min_reinforcement is the wrong lever — it would admit the noise rather than find the signal. Two candidate readings, and it's worth deciding which:

1. python is too broad for this corpus. Nearly every Python memory here belongs to one project, so "the Python domain" and "OmniMem's history" are the same set. Reinforcement can't distinguish a recurring language pattern from two unrelated episodes in the same codebase. Narrower domains (valkey-search, mcp-server-design) might have enough internal repetition to clear the gate, which is roughly what the existing three compiled skills look like.

2. The breakthrough field is written as narrative, not as a rule. record_experience() captures what happened, in past tense, tied to the specific incident. A rule needs to be a generalisable claim. "Community PR merged cleanly despite significant changes" can never become a rule no matter how often something like it recurs, because it isn't the kind of sentence a rule is. The valkey-py dropindex() memory is the counter-example that proves the point — its breakthrough field explicitly generalises ("when a fake stands in for a component with known divergent behaviour, the fake must reproduce the divergence"), and that is rule-shaped.

If (2) is the main factor, the fix is upstream of the compiler: prompt for the generalisable lesson at record time, separately from the narrative of what happened. That's a schema change, so worth deciding before more experience data accumulates in the current shape.

Suggested next step

Try compile_skill() against a narrower domain that maps to genuine cross-project work (valkey, testing) and see whether reinforcement clears there. That distinguishes reading 1 from reading 2 cheaply, without changing anything.

#32 — the held_back text is truncated, which makes assessing these candidates harder than it should be.

**Observed on:** 6.7.0 ## Observation `compile_skill(domain='python', mode='propose')` found a pool of 29 candidates and produced zero rules — nothing recurred across the `min_reinforcement=2` threshold. The gate behaved correctly. The interesting part is what the candidates were: - "Community PR merged cleanly despite significant changes to maintenance.py on bot…" - "Root cause was a data structure mismatch — `find_all_duplicates()` returned list[l…" - "HINCRBY creates fields with value 1 if absent, so no backfill migration needed f…" - "Leveraged existing `find_all_duplicates()` and `_has_negation_pair()` functions from…" These aren't Python rules. They're OmniMem project history that happens to carry a `python` tag. Some are Python-adjacent facts about Valkey commands; others are narrative about what happened in a particular PR. ## Why this won't fix itself The domain will never compile, and lowering `min_reinforcement` is the wrong lever — it would admit the noise rather than find the signal. Two candidate readings, and it's worth deciding which: **1. `python` is too broad for this corpus.** Nearly every Python memory here belongs to one project, so "the Python domain" and "OmniMem's history" are the same set. Reinforcement can't distinguish a recurring language pattern from two unrelated episodes in the same codebase. Narrower domains (`valkey-search`, `mcp-server-design`) might have enough internal repetition to clear the gate, which is roughly what the existing three compiled skills look like. **2. The `breakthrough` field is written as narrative, not as a rule.** `record_experience()` captures what happened, in past tense, tied to the specific incident. A rule needs to be a generalisable claim. "Community PR merged cleanly despite significant changes" can never become a rule no matter how often something like it recurs, because it isn't the kind of sentence a rule is. The valkey-py `dropindex()` memory is the counter-example that proves the point — its breakthrough field explicitly generalises ("when a fake stands in for a component with known divergent behaviour, the fake must reproduce the divergence"), and that *is* rule-shaped. If (2) is the main factor, the fix is upstream of the compiler: prompt for the generalisable lesson at record time, separately from the narrative of what happened. That's a schema change, so worth deciding before more experience data accumulates in the current shape. ## Suggested next step Try `compile_skill()` against a narrower domain that maps to genuine cross-project work (`valkey`, `testing`) and see whether reinforcement clears there. That distinguishes reading 1 from reading 2 cheaply, without changing anything. ## Related - #32 — the held_back text is truncated, which makes assessing these candidates harder than it should be.

ric referenced this issue from a commit

2026-09-08 14:20:27 +00:00

fix: full held_back text, drift surfacing, stale exemption (#28, #32, #33, #34)

ric commented

2026-09-08 15:12:12 +00:00

Author

Owner

Partly addressed in 6.7.1 (b4822f3 on v6.7.x) — diagnostics only, and the design question you raise is deliberately left open.

What shipped: insufficient_reinforcement now reports pool_concentration, and when every candidate in the pool comes from a single project it says so outright. That is the whole of reading 1, available at the point of use — you no longer have to run the experiment to discover that "the python domain" and "this project's history" are the same set of memories here, and that reinforcement therefore cannot distinguish a recurring language pattern from two unrelated episodes in one codebase.

The note also stopped recommending that you lower min_reinforcement, which it previously did. You're right that it's the wrong lever: when the candidates are narrative rather than generalisable claims, a lower gate admits the noise instead of finding the signal, turning a domain that correctly compiled nothing into one that compiles the wrong thing. It points at bless() and at narrower domains instead, and tells you what to look for in the held_back text — which is now readable in full (#32).

What is not fixed, and why. Reading 2 — that breakthrough is written as narrative of what happened rather than as a generalisable claim, so the fix belongs upstream at record_experience() — is a schema change, and this was a patch release. Your own framing is the argument for not rushing it: it's worth deciding before more experience data accumulates in the current shape, and that decision is yours rather than something to settle as a side effect of fixing six other issues.

I'd suggest a fresh issue for that specifically — capturing the lesson separately from the narrative at record time — since it's a design question with a real migration behind it, not a bug. Happy to open it if you want.

Closing this one as the diagnostic work is done; reopen or link the follow-up as you prefer.

Partly addressed in 6.7.1 (`b4822f3` on `v6.7.x`) — **diagnostics only, and the design question you raise is deliberately left open.** What shipped: `insufficient_reinforcement` now reports `pool_concentration`, and when every candidate in the pool comes from a single project it says so outright. That is the whole of reading 1, available at the point of use — you no longer have to run the experiment to discover that "the `python` domain" and "this project's history" are the same set of memories here, and that reinforcement therefore cannot distinguish a recurring language pattern from two unrelated episodes in one codebase. The note also **stopped recommending that you lower `min_reinforcement`**, which it previously did. You're right that it's the wrong lever: when the candidates are narrative rather than generalisable claims, a lower gate admits the noise instead of finding the signal, turning a domain that correctly compiled nothing into one that compiles the wrong thing. It points at `bless()` and at narrower domains instead, and tells you what to look for in the held_back text — which is now readable in full (#32). **What is not fixed, and why.** Reading 2 — that `breakthrough` is written as narrative of what happened rather than as a generalisable claim, so the fix belongs upstream at `record_experience()` — is a schema change, and this was a patch release. Your own framing is the argument for not rushing it: it's worth deciding before more experience data accumulates in the current shape, and that decision is yours rather than something to settle as a side effect of fixing six other issues. I'd suggest a fresh issue for that specifically — capturing the lesson separately from the narrative at record time — since it's a design question with a real migration behind it, not a bug. Happy to open it if you want. Closing this one as the diagnostic work is done; reopen or link the follow-up as you prefer.

ric closed this issue

2026-09-08 15:12:48 +00:00

ric referenced this issue

2026-09-08 15:58:48 +00:00

record_experience() captures narrative, the skill compiler needs a claim — separate the lesson from what happened #35

ric commented

2026-09-08 15:58:54 +00:00

Author

Owner

Follow-up opened as #35 for the schema half — separating the generalisable claim from the narrative at record_experience() time, with the four decisions it needs.

Follow-up opened as #35 for the schema half — separating the generalisable claim from the narrative at `record_experience()` time, with the four decisions it needs.

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#33

No description provided.

Rows
Columns