bug: fact extraction replaces verbatim content causing 50% score regression — should supplement not replace #20

New issue

Closed

opened 2026-04-13 12:38:08 +00:00 by ric_harvey · 3 comments

ric_harvey commented 2026-04-13 12:38:08 +00:00 (Migrated from codeberg.org)

Copy link

Problem

LongMemEval benchmark results show v5-full mode scores 33.6% overall — nearly half of v5-raw's 65.6%. The fact extraction pipeline is actively hurting retrieval rather than helping it.

Benchmark evidence

Run	Overall	temporal	preference	multi-session
v5-raw	65.6%	53.4%	46.7%	53.4%
v5-full	33.6%	7.5%	6.7%	37.6%

165 empty recalls in full mode vs 1 in raw mode — the extracted facts simply don't match recall queries.

Root cause: semantic gap between facts and questions

Fact extraction transforms conversational content:

Original: "User: I graduated with a BSc in Computer Science from Edinburgh in 2019.
           Assistant: Congratulations! That's a great achievement."

Extracted fact: "User has BSc in Computer Science from Edinburgh (2019)"

The recall query is: "What degree did I graduate with?"

The embedding similarity between the natural question and the compact extracted fact is lower than between the question and the original verbatim text. Valkey can't find the fact, returns empty, Claude abstains.

Two categories are catastrophic:

Temporal reasoning (7.5%) — fact extraction strips or normalises timestamps. "I moved to Edinburgh last March" loses its temporal anchor when converted to a fact. The timestamp metadata in haystack_dates is available but not being preserved in extracted facts.

Preference (6.7%) — subtle preferences buried in conversation ("I prefer leather straps", "I like earthy tones") are either missed during extraction or phrased differently, breaking recall.

Root cause: facts replace verbatim content

The current pipeline stores extracted facts instead of the original verbatim chunks. This is the wrong architecture for a retrieval system — you lose the original vocabulary that the recall query would naturally match against.

Proposed fix: supplementary storage, not replacement

Store both the extracted facts AND the verbatim chunks:

remember_document(content, mode='full'):
  1. Chunk into turn-pairs (verbatim) → store as mem:episodic:*
  2. Extract facts from each chunk → store as mem:knowledge:* (supplementary)
  3. Both are searchable at recall time

This way:

• Natural recall queries match verbatim chunks (high similarity)
• Fact-based queries and contradiction detection use the extracted facts
• No semantic gap — the original vocabulary is preserved

Additional fix: preserve timestamps in extracted facts

Extracted facts should carry the session timestamp as a field, not just in content:

# Current
fact = "User has BSc in Computer Science from Edinburgh"

# Proposed
fact = "User has BSc in Computer Science from Edinburgh (2019)"  # timestamp in content
# AND
metadata["fact_date"] = "2019"  # timestamp as searchable field

This directly addresses the 7.5% temporal reasoning score.

Expected impact

Switching from replacement to supplementary storage should recover most of the raw mode score (65.6%) while adding the benefits of fact extraction for contradiction detection and knowledge updates. Combined with timestamp preservation, temporal reasoning should improve significantly above the raw baseline.

Related issues

• #17 — v5 full mode benchmark improvements
• #19 — async fact extraction

## Problem LongMemEval benchmark results show v5-full mode scores 33.6% overall — nearly half of v5-raw's 65.6%. The fact extraction pipeline is actively hurting retrieval rather than helping it. ## Benchmark evidence | Run | Overall | temporal | preference | multi-session | |-----|---------|----------|------------|---------------| | v5-raw | 65.6% | 53.4% | 46.7% | 53.4% | | v5-full | 33.6% | 7.5% | 6.7% | 37.6% | 165 empty recalls in full mode vs 1 in raw mode — the extracted facts simply don't match recall queries. ## Root cause: semantic gap between facts and questions Fact extraction transforms conversational content: ``` Original: "User: I graduated with a BSc in Computer Science from Edinburgh in 2019. Assistant: Congratulations! That's a great achievement." Extracted fact: "User has BSc in Computer Science from Edinburgh (2019)" ``` The recall query is: `"What degree did I graduate with?"` The embedding similarity between the natural question and the compact extracted fact is **lower** than between the question and the original verbatim text. Valkey can't find the fact, returns empty, Claude abstains. Two categories are catastrophic: **Temporal reasoning (7.5%)** — fact extraction strips or normalises timestamps. "I moved to Edinburgh last March" loses its temporal anchor when converted to a fact. The timestamp metadata in `haystack_dates` is available but not being preserved in extracted facts. **Preference (6.7%)** — subtle preferences buried in conversation ("I prefer leather straps", "I like earthy tones") are either missed during extraction or phrased differently, breaking recall. ## Root cause: facts replace verbatim content The current pipeline stores extracted facts **instead of** the original verbatim chunks. This is the wrong architecture for a retrieval system — you lose the original vocabulary that the recall query would naturally match against. ## Proposed fix: supplementary storage, not replacement Store **both** the extracted facts AND the verbatim chunks: ``` remember_document(content, mode='full'): 1. Chunk into turn-pairs (verbatim) → store as mem:episodic:* 2. Extract facts from each chunk → store as mem:knowledge:* (supplementary) 3. Both are searchable at recall time ``` This way: - Natural recall queries match verbatim chunks (high similarity) - Fact-based queries and contradiction detection use the extracted facts - No semantic gap — the original vocabulary is preserved ## Additional fix: preserve timestamps in extracted facts Extracted facts should carry the session timestamp as a field, not just in content: ```python # Current fact = "User has BSc in Computer Science from Edinburgh" # Proposed fact = "User has BSc in Computer Science from Edinburgh (2019)" # timestamp in content # AND metadata["fact_date"] = "2019" # timestamp as searchable field ``` This directly addresses the 7.5% temporal reasoning score. ## Expected impact Switching from replacement to supplementary storage should recover most of the raw mode score (65.6%) while adding the benefits of fact extraction for contradiction detection and knowledge updates. Combined with timestamp preservation, temporal reasoning should improve significantly above the raw baseline. ## Related issues - #17 — v5 full mode benchmark improvements - #19 — async fact extraction

ric_harvey commented 2026-04-13 12:43:08 +00:00 (Migrated from codeberg.org)

Copy link

Highest-leverage single fix: timestamp preservation in extracted facts

The temporal reasoning collapse from 53.4% → 7.5% is almost entirely explained by timestamps being stripped during fact extraction.

The problem

Session date:   2023/05/20
Original chunk: "User: I moved to Edinburgh last March for a new job at a startup.
                 Assistant: That's exciting! How are you settling in?"

Extracted fact: "User moved to Edinburgh for a new job at a startup"
                ← timestamp gone, "last March" gone

Recall query: "When did the user move to Edinburgh?"

The extracted fact has no temporal anchor. Valkey embedding similarity between a temporal question and a dateless fact is near zero → empty recall → abstention.

The fix — prepend timestamp to every extracted fact

# In extract_facts() or remember_document() enrichment step:
timestamp = session_metadata.get("timestamp", "")
if timestamp:
    fact = f"[{timestamp}] {fact}"

# Result:
# "[2023/05/20] User moved to Edinburgh for a new job at a startup"

Also store timestamp as a searchable metadata field:

metadata["fact_date"] = timestamp  # enables date-range filtering later

Expected impact

This single change should recover most of the 7.5% → ~40-50% temporal reasoning score, because:

• Temporal recall queries now have a date anchor to match against
• "When did..." queries embed with higher similarity to timestamped facts
• The haystack_dates field is already available in LongMemEval sessions — it just needs threading through to the extracted fact

Supplementary storage implementation notes

For the verbatim + facts dual storage approach:

1. Linking — extracted facts should carry source_key metadata pointing to their parent verbatim chunk. When the verbatim chunk is deleted, its facts are deleted too. doc_id from remember_document() is the natural grouping key.

2. Recall priority — verbatim chunks should score slightly higher than extracted facts on direct recall (they preserve original vocabulary). Facts should score higher on contradiction checks and knowledge updates.

3. Quoted phrase / name boost (from MemPalace hybrid_v4) — post-retrieval reranking step that boosts results containing exact proper nouns or quoted phrases from the query. Implementable as a rerank() step, doesn't change storage.

4. Nostalgia patterns — temporal anchor phrases ("last year", "used to", "when I was") in the query should boost chunks with fact_date metadata. Complements the timestamp preservation fix.

Priority order for maximum benchmark impact

1. ✅ Verbatim + facts dual storage (issue #20 main fix)
2. 🔴 Timestamp preservation in extracted facts (this comment — highest leverage)
3. 🟡 source_key linking for clean deletion
4. 🟢 Quoted phrase / name boost reranking
5. 🟢 Nostalgia pattern temporal boosting

## Highest-leverage single fix: timestamp preservation in extracted facts The temporal reasoning collapse from 53.4% → 7.5% is almost entirely explained by timestamps being stripped during fact extraction. ### The problem ``` Session date: 2023/05/20 Original chunk: "User: I moved to Edinburgh last March for a new job at a startup. Assistant: That's exciting! How are you settling in?" Extracted fact: "User moved to Edinburgh for a new job at a startup" ← timestamp gone, "last March" gone ``` Recall query: "When did the user move to Edinburgh?" The extracted fact has no temporal anchor. Valkey embedding similarity between a temporal question and a dateless fact is near zero → empty recall → abstention. ### The fix — prepend timestamp to every extracted fact ```python # In extract_facts() or remember_document() enrichment step: timestamp = session_metadata.get("timestamp", "") if timestamp: fact = f"[{timestamp}] {fact}" # Result: # "[2023/05/20] User moved to Edinburgh for a new job at a startup" ``` Also store timestamp as a searchable metadata field: ```python metadata["fact_date"] = timestamp # enables date-range filtering later ``` ### Expected impact This single change should recover most of the 7.5% → ~40-50% temporal reasoning score, because: - Temporal recall queries now have a date anchor to match against - "When did..." queries embed with higher similarity to timestamped facts - The `haystack_dates` field is already available in LongMemEval sessions — it just needs threading through to the extracted fact ### Supplementary storage implementation notes For the verbatim + facts dual storage approach: 1. **Linking** — extracted facts should carry `source_key` metadata pointing to their parent verbatim chunk. When the verbatim chunk is deleted, its facts are deleted too. `doc_id` from `remember_document()` is the natural grouping key. 2. **Recall priority** — verbatim chunks should score slightly higher than extracted facts on direct recall (they preserve original vocabulary). Facts should score higher on contradiction checks and knowledge updates. 3. **Quoted phrase / name boost** (from MemPalace hybrid_v4) — post-retrieval reranking step that boosts results containing exact proper nouns or quoted phrases from the query. Implementable as a `rerank()` step, doesn't change storage. 4. **Nostalgia patterns** — temporal anchor phrases ("last year", "used to", "when I was") in the query should boost chunks with `fact_date` metadata. Complements the timestamp preservation fix. ### Priority order for maximum benchmark impact 1. ✅ Verbatim + facts dual storage (issue #20 main fix) 2. 🔴 Timestamp preservation in extracted facts (this comment — highest leverage) 3. 🟡 `source_key` linking for clean deletion 4. 🟢 Quoted phrase / name boost reranking 5. 🟢 Nostalgia pattern temporal boosting

ric_harvey commented 2026-04-13 12:54:53 +00:00 (Migrated from codeberg.org)

Copy link

Fix Plan

Root cause (confirmed)

The issue description identifies the semantic gap correctly, but the actual mechanism is namespace crowding, not replacement. Both verbatim chunks and extracted facts coexist, but extracted facts are stored in the same namespace as source chunks (e.g. both in episodic). During recall, vector search returns top_k=20 per namespace then clamps to top_k=5 final results. Extracted facts (semantically narrower, 1-2 sentences) crowd out the richer verbatim content that would otherwise match recall queries.

Additionally, temporal info from source memories is lost when extracted facts don't carry their own event_date.

Fix: 3 files, 6 changes

1. Route facts to knowledge namespace — mcp_server/memory/enrichment.py:124

# Before
target_ns = "preference" if fact.kind == "preference" else namespace
# After
target_ns = "preference" if fact.kind == "preference" else "knowledge"

The knowledge namespace already has its own HNSW index and is already searched during recall. Verbatim chunks stay in episodic with their own top_k=20 budget. No crowding.

2. Preserve source timestamps — mcp_server/memory/enrichment.py

In _enrich(), read event_date and created_at from the source memory and apply a fallback chain:

if fact.event_date is not None:
    fields["event_date"] = str(fact.event_date)
elif source_event_date is not None:
    fields["event_date"] = str(source_event_date)
elif source_created_at is not None:
    fields["event_date"] = str(source_created_at)

Pass created_at through enqueue() and enqueue_batch() payloads for batch mode (where the worker doesn't read individual source memories).

3. Pass created_at at call sites — mcp_server/tools/core.py

All three enqueue()/enqueue_batch() call sites updated to pass created_at=now.

4. Update dead code — mcp_server/tools/core.py:116

_store_extracted_fact (currently unused) updated for consistency.

5. Tests — mcp_server/tests/test_enrichment.py

• Update 2 existing tests to expect knowledge namespace
• Add 3 new tests: namespace routing regression, timestamp inheritance, timestamp precedence

No migration required

Existing enriched facts in episodic remain searchable. New facts go to knowledge going forward. Old enriched facts age out naturally via lifecycle.

## Fix Plan ### Root cause (confirmed) The issue description identifies the semantic gap correctly, but the actual mechanism is **namespace crowding**, not replacement. Both verbatim chunks and extracted facts coexist, but extracted facts are stored in the **same namespace** as source chunks (e.g. both in `episodic`). During recall, vector search returns `top_k=20` per namespace then clamps to `top_k=5` final results. Extracted facts (semantically narrower, 1-2 sentences) crowd out the richer verbatim content that would otherwise match recall queries. Additionally, temporal info from source memories is lost when extracted facts don't carry their own `event_date`. ### Fix: 3 files, 6 changes **1. Route facts to `knowledge` namespace** — `mcp_server/memory/enrichment.py:124` ```python # Before target_ns = "preference" if fact.kind == "preference" else namespace # After target_ns = "preference" if fact.kind == "preference" else "knowledge" ``` The `knowledge` namespace already has its own HNSW index and is already searched during recall. Verbatim chunks stay in `episodic` with their own `top_k=20` budget. No crowding. **2. Preserve source timestamps** — `mcp_server/memory/enrichment.py` In `_enrich()`, read `event_date` and `created_at` from the source memory and apply a fallback chain: ```python if fact.event_date is not None: fields["event_date"] = str(fact.event_date) elif source_event_date is not None: fields["event_date"] = str(source_event_date) elif source_created_at is not None: fields["event_date"] = str(source_created_at) ``` Pass `created_at` through `enqueue()` and `enqueue_batch()` payloads for batch mode (where the worker doesn't read individual source memories). **3. Pass `created_at` at call sites** — `mcp_server/tools/core.py` All three `enqueue()`/`enqueue_batch()` call sites updated to pass `created_at=now`. **4. Update dead code** — `mcp_server/tools/core.py:116` `_store_extracted_fact` (currently unused) updated for consistency. **5. Tests** — `mcp_server/tests/test_enrichment.py` - Update 2 existing tests to expect `knowledge` namespace - Add 3 new tests: namespace routing regression, timestamp inheritance, timestamp precedence ### No migration required Existing enriched facts in `episodic` remain searchable. New facts go to `knowledge` going forward. Old enriched facts age out naturally via lifecycle.

ric_harvey commented 2026-07-08 11:05:14 +00:00 (Migrated from codeberg.org)

Copy link

Fixed in v5.3.1 on branch v5 (commit 80e43fe).

The fix follows the plan in the comments above, with one correction to the diagnosis: it was indeed namespace crowding rather than replacement — both verbatim chunks and facts coexisted, but compact facts competed for the same per-namespace KNN candidate budget.

What landed

1. Facts route to the knowledge namespace (memory/enrichment.py)
Preference-shaped facts still go to preference. Verbatim chunks keep the episodic candidate budget to themselves, so facts can no longer crowd out the original wording that recall queries naturally match.

2. event_date fallback chain — the temporal fix
A fact carries its own date if extraction found one, else it inherits the source memory's event_date, else the source's ingest time. enqueue()/enqueue_batch() now thread created_at through the queue payload so batch mode gets timestamps too, and batch mode also falls back to reading the first chunk's timestamps so pre-upgrade queued payloads still get an anchor. Date-shaped recall queries ("when did I…") hit the existing temporal boost (1.0–1.5x within ±7 days, linear falloff to ±60 days) against the inherited anchor.

3. Verbatim-first ranking + source linking
Facts are written with surface_score 0.5 so the verbatim chunk outranks its own derived fact on direct recall. When a fact and its enriched_from source both match a query, the merge keeps only the source, promoted to the fact's score if the fact ranked higher — so the verbatim stands in at the fact's rank in both orderings, and no result slot is wasted on a near-duplicate. enriched_from was already being written; it's now returned by search and exposed in recall() output.

4. The silent project-filter drop
_NAMESPACE_RETURN_FIELDS for knowledge was missing project entirely, so a project-filtered recall discarded every knowledge result without a trace — facts would have been invisible the moment they moved namespaces. Knowledge results now carry project, event_date, tags and enriched_from, and the knowledge index gains an indexed project tag (picked up automatically by the startup index migration) so the project filter is pushed into the vector search itself.

Validation

• 583 unit tests pass (13 new for this issue: namespace routing, all three fallback-chain branches, batch-mode timestamps incl. pre-upgrade payloads, source promotion when the fact ranks higher, project-filtered fact recall, temporal boost inheritance)
• End-to-end against a live valkey-search container: index migration to the new field set, worker routing, return fields, project-filtered fact recall, source-present suppression, and the inherited event_date driving the temporal boost
• Adversarial multi-agent review of the diff caught and fixed two issues pre-merge: a rank-inversion in the suppression logic (fact shown instead of source when the fact ranked higher) and the pre-upgrade batch payload anchor loss

Not included (from the priority list above)

• Quoted phrase / name boost reranking (🟢) and nostalgia-pattern boosting (🟢) — the existing dateparser-based temporal boost already covers parseable anchors like "last March"; the fuzzy-phrase variants remain future work
• No data migration, as planned: old enriched facts stay in episodic and age out naturally

Benchmark re-run against LongMemEval is the remaining validation step — tracked under #17.

Fixed in **v5.3.1** on branch `v5` (commit 80e43fe). The fix follows the plan in the comments above, with one correction to the diagnosis: it was indeed namespace crowding rather than replacement — both verbatim chunks and facts coexisted, but compact facts competed for the same per-namespace KNN candidate budget. ## What landed **1. Facts route to the `knowledge` namespace** (`memory/enrichment.py`) Preference-shaped facts still go to `preference`. Verbatim chunks keep the episodic candidate budget to themselves, so facts can no longer crowd out the original wording that recall queries naturally match. **2. `event_date` fallback chain** — the temporal fix A fact carries its own date if extraction found one, else it inherits the source memory's `event_date`, else the source's ingest time. `enqueue()`/`enqueue_batch()` now thread `created_at` through the queue payload so batch mode gets timestamps too, and batch mode also falls back to reading the first chunk's timestamps so pre-upgrade queued payloads still get an anchor. Date-shaped recall queries ("when did I…") hit the existing temporal boost (1.0–1.5x within ±7 days, linear falloff to ±60 days) against the inherited anchor. **3. Verbatim-first ranking + source linking** Facts are written with `surface_score` 0.5 so the verbatim chunk outranks its own derived fact on direct recall. When a fact and its `enriched_from` source both match a query, the merge keeps only the source, promoted to the fact's score if the fact ranked higher — so the verbatim stands in at the fact's rank in both orderings, and no result slot is wasted on a near-duplicate. `enriched_from` was already being written; it's now returned by search and exposed in `recall()` output. **4. The silent project-filter drop** `_NAMESPACE_RETURN_FIELDS` for knowledge was missing `project` entirely, so a project-filtered recall discarded every knowledge result without a trace — facts would have been invisible the moment they moved namespaces. Knowledge results now carry `project`, `event_date`, `tags` and `enriched_from`, and the knowledge index gains an indexed `project` tag (picked up automatically by the startup index migration) so the project filter is pushed into the vector search itself. ## Validation - 583 unit tests pass (13 new for this issue: namespace routing, all three fallback-chain branches, batch-mode timestamps incl. pre-upgrade payloads, source promotion when the fact ranks higher, project-filtered fact recall, temporal boost inheritance) - End-to-end against a live valkey-search container: index migration to the new field set, worker routing, return fields, project-filtered fact recall, source-present suppression, and the inherited `event_date` driving the temporal boost - Adversarial multi-agent review of the diff caught and fixed two issues pre-merge: a rank-inversion in the suppression logic (fact shown instead of source when the fact ranked higher) and the pre-upgrade batch payload anchor loss ## Not included (from the priority list above) - Quoted phrase / name boost reranking (🟢) and nostalgia-pattern boosting (🟢) — the existing dateparser-based temporal boost already covers parseable anchors like "last March"; the fuzzy-phrase variants remain future work - No data migration, as planned: old enriched facts stay in episodic and age out naturally Benchmark re-run against LongMemEval is the remaining validation step — tracked under #17.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

v6.5.x

pages

v6.4.x

badges

v6

v5

v4

feat/rss-reader-improvements

v6.4.2

V6.3.2

v6.3.2

v6.1.0

v5.5.3

v3.13.1

v3.13.0

v3.12.1

v3.10.4

v3.10.2

v3.10.1

v3.10.0

v3.9.5

v3.8.2

v3.8.1

v3.8.0

v3.7.1

v3.7.0

v0.2.0

v0.1.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#20

No description provided.