feat: v5 — LongMemEval benchmark improvements (fact extraction, timestamp-aware retrieval) #17

New issue

Open

opened 2026-04-08 22:13:12 +00:00 by ric_harvey · 1 comment

ric_harvey commented

2026-04-08 22:13:12 +00:00

(Migrated from codeberg.org)

Background

OmniMem v2-raw was benchmarked against LongMemEval (500 items, GPT-4o judge) with the following results:

Question type	Score
single-session-assistant	92.9%
single-session-user	84.3%
knowledge-update	79.5%
multi-session	51.9%
temporal-reasoning	51.1%
single-session-preference	50.0%
Overall	65.0%

Single-session retrieval is strong. The weak spots are multi-session (51.9%) and temporal-reasoning (51.1%), which together make up 266/500 items (53% of the benchmark). These are exactly what OmniMem's pipeline is designed to address — but this run used raw mode (verbatim storage, no enrichment).

Proposed v5 improvements

1. Fact extraction at ingest (full mode)

Instead of storing raw conversation turns, use Claude to extract discrete facts before storing:

Session: "User: I moved to Edinburgh last March for a new job at a startup."
→ Stored fact: "User moved to Edinburgh in March [year] for a job at a startup."

This directly improves multi-session reasoning — facts are compact, semantically precise, and deduplicable across sessions. OmniMem's contradiction detection then handles knowledge updates correctly.

2. Timestamp-aware query expansion

For temporal questions ("What did I do last Tuesday?"), include date context in search variants. The haystack_dates field in LongMemEval provides session timestamps — these should be surfaced in recall scoring.

3. Preference detection at ingest

Single-session-preference scored 50.0% — preferences are subtle ("I prefer leather straps", "I like earthy tones") and often buried mid-conversation. A dedicated preference extraction pass at ingest would surface these as structured preference memory types (related to the preference memory type design from earlier work).

Expected impact

Based on the category breakdown, fixing multi-session and temporal alone (266 items) could move the overall score from 65% to 80%+. Fact extraction is the highest-leverage change.

Benchmark repo

Full benchmark code, results, and evaluation scripts: ~/Downloads/omnimem-longmemeval (to be pushed to Forgejo).

#15 remember_document() — chunking strategies needed for fact extraction pipeline
#16 built-in query expansion — timestamp-aware variant needed for temporal questions

## Background OmniMem v2-raw was benchmarked against LongMemEval (500 items, GPT-4o judge) with the following results: | Question type | Score | |---|---| | single-session-assistant | 92.9% | | single-session-user | 84.3% | | knowledge-update | 79.5% | | multi-session | 51.9% | | temporal-reasoning | 51.1% | | single-session-preference | 50.0% | | **Overall** | **65.0%** | Single-session retrieval is strong. The weak spots are multi-session (51.9%) and temporal-reasoning (51.1%), which together make up 266/500 items (53% of the benchmark). These are exactly what OmniMem's pipeline is designed to address — but this run used raw mode (verbatim storage, no enrichment). ## Proposed v5 improvements ### 1. Fact extraction at ingest (full mode) Instead of storing raw conversation turns, use Claude to extract discrete facts before storing: ``` Session: "User: I moved to Edinburgh last March for a new job at a startup." → Stored fact: "User moved to Edinburgh in March [year] for a job at a startup." ``` This directly improves multi-session reasoning — facts are compact, semantically precise, and deduplicable across sessions. OmniMem's contradiction detection then handles knowledge updates correctly. ### 2. Timestamp-aware query expansion For temporal questions ("What did I do last Tuesday?"), include date context in search variants. The `haystack_dates` field in LongMemEval provides session timestamps — these should be surfaced in recall scoring. ### 3. Preference detection at ingest Single-session-preference scored 50.0% — preferences are subtle ("I prefer leather straps", "I like earthy tones") and often buried mid-conversation. A dedicated preference extraction pass at ingest would surface these as structured `preference` memory types (related to the preference memory type design from earlier work). ## Expected impact Based on the category breakdown, fixing multi-session and temporal alone (266 items) could move the overall score from 65% to 80%+. Fact extraction is the highest-leverage change. ## Benchmark repo Full benchmark code, results, and evaluation scripts: `~/Downloads/omnimem-longmemeval` (to be pushed to Forgejo). ## Related issues - #15 `remember_document()` — chunking strategies needed for fact extraction pipeline - #16 built-in query expansion — timestamp-aware variant needed for temporal questions

ric_harvey commented

2026-07-08 11:11:49 +00:00

(Migrated from codeberg.org)

Status update (2026-07-08) — leaving this open because its acceptance criterion is a benchmark number, but all three proposed improvements are now code-complete on v5:

Fact extraction at ingest — shipped v5.0.0 (INGEST_MODE=full), made asynchronous in v5.2.0 (#19). The architecture flaw that made it hurt scores (facts crowding out verbatim content, timestamps stripped — the 33.6% vs 65.6% regression) was fixed in v5.3.1 (#20): facts now supplement verbatim chunks from the knowledge namespace, inherit temporal anchors via an event_date fallback chain, and the verbatim wording wins direct recall.
Timestamp-aware retrieval — shipped v5.0.0 (dateparser-based query date detection + 1.0–1.5x event_date proximity boost), and as of v5.3.1 the boost also applies to query-expansion variants and to extracted facts via inherited anchors.
Preference detection at ingest — shipped v5.0.0: preference-shaped facts route to the dedicated preference namespace with its own index and recall path.

Related recall-quality work that should help the multi-session categories specifically: v5.3.0 pushed project/state filters into the vector search and scaled the candidate pool, fixing result starvation under project filters.

What remains for this issue: re-run LongMemEval (raw vs full) against current v5 to confirm full mode now beats the 65.0% raw baseline, especially on temporal-reasoning and multi-session. Closing then, with the numbers.

Status update (2026-07-08) — **leaving this open** because its acceptance criterion is a benchmark number, but all three proposed improvements are now code-complete on `v5`: 1. **Fact extraction at ingest** — shipped v5.0.0 (`INGEST_MODE=full`), made asynchronous in v5.2.0 (#19). The architecture flaw that made it *hurt* scores (facts crowding out verbatim content, timestamps stripped — the 33.6% vs 65.6% regression) was fixed in v5.3.1 (#20): facts now supplement verbatim chunks from the knowledge namespace, inherit temporal anchors via an `event_date` fallback chain, and the verbatim wording wins direct recall. 2. **Timestamp-aware retrieval** — shipped v5.0.0 (dateparser-based query date detection + 1.0–1.5x `event_date` proximity boost), and as of v5.3.1 the boost also applies to query-expansion variants and to extracted facts via inherited anchors. 3. **Preference detection at ingest** — shipped v5.0.0: preference-shaped facts route to the dedicated `preference` namespace with its own index and recall path. Related recall-quality work that should help the multi-session categories specifically: v5.3.0 pushed project/state filters into the vector search and scaled the candidate pool, fixing result starvation under project filters. **What remains for this issue:** re-run LongMemEval (raw vs full) against current `v5` to confirm full mode now beats the 65.0% raw baseline, especially on temporal-reasoning and multi-session. Closing then, with the numbers.

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#17

No description provided.

Rows
Columns