perf: async fact extraction for remember() and remember_document() #19

New issue

Closed

opened 2026-04-09 14:04:54 +00:00 by ric_harvey · 5 comments

ric_harvey commented 2026-04-09 14:04:54 +00:00 (Migrated from codeberg.org)

Copy link

Problem

remember_document() with mode='full' takes ~3.4s for a tiny 3-turn session. A realistic LongMemEval session (~10-15 turns, ~1000 words) takes ~8-15s. With 53 sessions per benchmark item and 500 items, full mode ingestion would take ~74 hours — completely impractical.

The bottleneck is extract_facts() — a synchronous Haiku API call that runs on every chunk before returning. The caller blocks waiting for fact extraction before the next chunk can be stored.

Timing breakdown (v5.1.1-beta, localhost)

remember(force=True)          → 0.252s   (embed + write, no enrichment)
remember(force=False, raw)    → 0.252s   (same)
remember_document (tiny, full) → 3.405s  (2 turn-pairs, fact extraction blocking)
remember_document (real, full) → ~10-15s (10-15 turns, multiple Haiku calls)

Proposed solution — async fact extraction

Decouple ingestion from enrichment:

1. remember() and remember_document() store chunks immediately (embed + write) and return keys — same speed as force=True
2. A background worker queue picks up new keys and runs extract_facts() + contradiction check asynchronously
3. Facts/preferences are written back to the memory as enrichment completes
4. Caller gets an immediate response; enrichment happens in the background

# Current (blocking)
remember_document(content, mode='full')
# → embed chunk 1 → extract_facts(chunk 1) → store → embed chunk 2 → extract_facts(chunk 2) → ...
# Total: n_chunks × (embed_time + haiku_call_time)

# Proposed (async enrichment)
remember_document(content, mode='full')
# → embed chunk 1 → store → embed chunk 2 → store → ... → return keys
# Background: extract_facts(chunk 1), extract_facts(chunk 2), ...
# Total ingest: n_chunks × embed_time (~0.25s/chunk)

Impact on benchmark

With async enrichment, full mode ingestion time drops from ~74 hours to ~2 hours for 500 LongMemEval items — making the v5-full benchmark run feasible.

Benchmark evidence

Mode	Ingest time/item	500-item total	Feasible?
raw (force=True)	~62s	~9hrs	marginal
raw (force=True, 0.25s/chunk)	~62s	~9hrs	marginal
full (sync, current)	~530s	~74hrs	❌ no
full (async, proposed)	~62s	~9hrs	✅ yes

Note: the 9hr figure assumes 250 chunks/item × 0.25s. A batch ingest endpoint (separate issue) would reduce this further.

Implementation notes

OmniMem already uses Valkey + Apalis for job queuing in the Mycelium architecture. The same pattern applies here — new memories could be pushed onto an enrichment queue immediately after storage, with the RSS worker pattern reused for the enrichment consumer.

Related issues

• #18 — force=True should be raw write (partially addresses this for benchmark use)
• #15 — remember_document() implementation (now in v5)
• #17 — v5 full mode benchmark improvements

## Problem `remember_document()` with `mode='full'` takes ~3.4s for a tiny 3-turn session. A realistic LongMemEval session (~10-15 turns, ~1000 words) takes ~8-15s. With 53 sessions per benchmark item and 500 items, full mode ingestion would take ~74 hours — completely impractical. The bottleneck is `extract_facts()` — a synchronous Haiku API call that runs on every chunk before returning. The caller blocks waiting for fact extraction before the next chunk can be stored. ## Timing breakdown (v5.1.1-beta, localhost) ``` remember(force=True) → 0.252s (embed + write, no enrichment) remember(force=False, raw) → 0.252s (same) remember_document (tiny, full) → 3.405s (2 turn-pairs, fact extraction blocking) remember_document (real, full) → ~10-15s (10-15 turns, multiple Haiku calls) ``` ## Proposed solution — async fact extraction Decouple ingestion from enrichment: 1. `remember()` and `remember_document()` store chunks immediately (embed + write) and return keys — same speed as `force=True` 2. A background worker queue picks up new keys and runs `extract_facts()` + contradiction check asynchronously 3. Facts/preferences are written back to the memory as enrichment completes 4. Caller gets an immediate response; enrichment happens in the background ```python # Current (blocking) remember_document(content, mode='full') # → embed chunk 1 → extract_facts(chunk 1) → store → embed chunk 2 → extract_facts(chunk 2) → ... # Total: n_chunks × (embed_time + haiku_call_time) # Proposed (async enrichment) remember_document(content, mode='full') # → embed chunk 1 → store → embed chunk 2 → store → ... → return keys # Background: extract_facts(chunk 1), extract_facts(chunk 2), ... # Total ingest: n_chunks × embed_time (~0.25s/chunk) ``` ## Impact on benchmark With async enrichment, full mode ingestion time drops from ~74 hours to ~2 hours for 500 LongMemEval items — making the v5-full benchmark run feasible. ## Benchmark evidence | Mode | Ingest time/item | 500-item total | Feasible? | |------|-----------------|----------------|-----------| | raw (force=True) | ~62s | ~9hrs | marginal | | raw (force=True, 0.25s/chunk) | ~62s | ~9hrs | marginal | | full (sync, current) | ~530s | ~74hrs | ❌ no | | full (async, proposed) | ~62s | ~9hrs | ✅ yes | Note: the 9hr figure assumes 250 chunks/item × 0.25s. A batch ingest endpoint (separate issue) would reduce this further. ## Implementation notes OmniMem already uses Valkey + Apalis for job queuing in the Mycelium architecture. The same pattern applies here — new memories could be pushed onto an enrichment queue immediately after storage, with the RSS worker pattern reused for the enrichment consumer. ## Related issues - #18 — `force=True` should be raw write (partially addresses this for benchmark use) - #15 — `remember_document()` implementation (now in v5) - #17 — v5 full mode benchmark improvements

ric_harvey commented 2026-04-09 14:05:30 +00:00 (Migrated from codeberg.org)

Copy link

Updated timing with real LongMemEval data

Session: 1434 chars, 2 turns
remember_document (full, turn_pairs): 4.369s → 5 keys stored

Extrapolated for real benchmark:
- 53 sessions/item × 4.37s = ~232s/item
- 500 items × 232s = ~32 hours total

Still not feasible but better than the 74hr estimate (which assumed 10-15 turns per session — real sessions average 2 turns).

The blocker remains the synchronous Haiku call per chunk. Even at 2 turns per session, extract_facts() is running synchronously before the next session can be stored.

Also noted

The v5 remember_document() tool schema does not expose a mode parameter — mode is only on remember(). This means remember_document() currently always runs in whatever the INGEST_MODE env var is set to. For the benchmark to control full vs raw mode via remember_document(), either:

1. Add mode parameter to remember_document() (consistent with remember())
2. Or rely on INGEST_MODE env var switching between runs

Both are valid — just needs documenting.

## Updated timing with real LongMemEval data ``` Session: 1434 chars, 2 turns remember_document (full, turn_pairs): 4.369s → 5 keys stored Extrapolated for real benchmark: - 53 sessions/item × 4.37s = ~232s/item - 500 items × 232s = ~32 hours total ``` Still not feasible but better than the 74hr estimate (which assumed 10-15 turns per session — real sessions average 2 turns). The blocker remains the synchronous Haiku call per chunk. Even at 2 turns per session, `extract_facts()` is running synchronously before the next session can be stored. ## Also noted The v5 `remember_document()` tool schema does not expose a `mode` parameter — `mode` is only on `remember()`. This means `remember_document()` currently always runs in whatever the `INGEST_MODE` env var is set to. For the benchmark to control full vs raw mode via `remember_document()`, either: 1. Add `mode` parameter to `remember_document()` (consistent with `remember()`) 2. Or rely on `INGEST_MODE` env var switching between runs Both are valid — just needs documenting.

ric_harvey commented 2026-04-09 14:15:33 +00:00 (Migrated from codeberg.org)

Copy link

Benchmark runner architecture for async enrichment

The benchmark runner needs to split into three phases to work correctly with async fact extraction:

Phase 1 — Ingest (fast)

ingest_manifest = {}
for item in items:
    doc_ids = []
    for session in item['haystack_sessions']:
        result = remember_document(content, chunk_strategy='turn_pairs', mode='full')
        doc_ids.append(result['doc_id'])
    ingest_manifest[item['question_id']] = doc_ids
# Save manifest to disk — survives interruption
json.dump(ingest_manifest, open('results/ingest_manifest.json', 'w'))

Phase 2 — Wait for enrichment queue to drain

while True:
    status = queue_status()  # needs new MCP tool
    if status['pending'] == 0:
        break
    print(f"Waiting... {status['pending']} enrichment jobs pending")
    time.sleep(30)

Phase 3 — Query (after enrichment complete)

for qid, doc_ids in ingest_manifest.items():
    chunks = recall_full(question, project=f'bench_{qid}', expand_queries=True)
    answer = generate_answer(question, chunks)
    # write result to JSONL

New MCP tools needed to support this

1. queue_status() — returns {"pending": int, "processing": int, "completed": int}. Benchmark polls this between Phase 2 and Phase 3 to know when enrichment is complete before querying.

2. remember_document() already returns doc_id — cleanup can be done by doc_id rather than by project scan, making it reliable.

Why the manifest matters

Saving the ingest manifest to disk means Phase 1 and Phase 3 can run at different times — you can ingest overnight, let enrichment run, then query the next day. Supports --resume cleanly.

## Benchmark runner architecture for async enrichment The benchmark runner needs to split into three phases to work correctly with async fact extraction: ### Phase 1 — Ingest (fast) ```python ingest_manifest = {} for item in items: doc_ids = [] for session in item['haystack_sessions']: result = remember_document(content, chunk_strategy='turn_pairs', mode='full') doc_ids.append(result['doc_id']) ingest_manifest[item['question_id']] = doc_ids # Save manifest to disk — survives interruption json.dump(ingest_manifest, open('results/ingest_manifest.json', 'w')) ``` ### Phase 2 — Wait for enrichment queue to drain ```python while True: status = queue_status() # needs new MCP tool if status['pending'] == 0: break print(f"Waiting... {status['pending']} enrichment jobs pending") time.sleep(30) ``` ### Phase 3 — Query (after enrichment complete) ```python for qid, doc_ids in ingest_manifest.items(): chunks = recall_full(question, project=f'bench_{qid}', expand_queries=True) answer = generate_answer(question, chunks) # write result to JSONL ``` ### New MCP tools needed to support this 1. **`queue_status()`** — returns `{"pending": int, "processing": int, "completed": int}`. Benchmark polls this between Phase 2 and Phase 3 to know when enrichment is complete before querying. 2. **`remember_document()` already returns `doc_id`** — cleanup can be done by `doc_id` rather than by project scan, making it reliable. ### Why the manifest matters Saving the ingest manifest to disk means Phase 1 and Phase 3 can run at different times — you can ingest overnight, let enrichment run, then query the next day. Supports `--resume` cleanly.

ric_harvey commented 2026-04-09 14:34:20 +00:00 (Migrated from codeberg.org)

Copy link

Proposed implementation: ENRICHMENT_BATCH_MODE

Rather than per-memory async enrichment via a queue, a simpler approach is ENRICHMENT_BATCH_MODE:

How it works

ENRICHMENT_BATCH_MODE=false (default):
  remember_document() → chunk → extract_facts(chunk 1) → store → extract_facts(chunk 2) → store...
  Cost: n_chunks × (embed + haiku_call) per call — slow, blocking

ENRICHMENT_BATCH_MODE=true:
  remember_document() → store raw session → queue enrichment job → return doc_id immediately
  Cost: ~0.5s per session — fast, non-blocking
  Background: enrichment worker processes queued jobs

Benchmark runner flow with batch mode

# Phase 1 — Ingest (fast, ~3.6 hours for 500 items)
# Set ENRICHMENT_BATCH_MODE=true in .env
for item in items:
    for session in item['haystack_sessions']:
        result = remember_document(content, chunk_strategy='turn_pairs')
        # Returns immediately, enrichment queued
    manifest[item['question_id']] = project

# Phase 2 — Wait for enrichment queue to drain
while queue_status()['pending'] > 0:
    sleep(30)

# Phase 3 — Query (enriched facts now searchable, ~40 min)
for qid in manifest:
    chunks = recall_full(question, project=f'bench_{qid}', expand_queries=True)
    answer = generate_answer(question, chunks)

Timing estimate with batch mode

Phase	Time
Ingest (53 sessions × ~0.5s × 500 items)	~3.6 hours
Enrichment (background, runs during ingest)	overlaps
Query (500 items × ~5s)	~40 min
Total	~4-5 hours

This makes the v5-full benchmark feasible in a single overnight run.

queue_status() tool still needed

The runner needs to poll queue_status() between Phase 2 and Phase 3 to know when enrichment is complete and querying will return enriched results.

## Proposed implementation: ENRICHMENT_BATCH_MODE Rather than per-memory async enrichment via a queue, a simpler approach is `ENRICHMENT_BATCH_MODE`: ### How it works ``` ENRICHMENT_BATCH_MODE=false (default): remember_document() → chunk → extract_facts(chunk 1) → store → extract_facts(chunk 2) → store... Cost: n_chunks × (embed + haiku_call) per call — slow, blocking ENRICHMENT_BATCH_MODE=true: remember_document() → store raw session → queue enrichment job → return doc_id immediately Cost: ~0.5s per session — fast, non-blocking Background: enrichment worker processes queued jobs ``` ### Benchmark runner flow with batch mode ```python # Phase 1 — Ingest (fast, ~3.6 hours for 500 items) # Set ENRICHMENT_BATCH_MODE=true in .env for item in items: for session in item['haystack_sessions']: result = remember_document(content, chunk_strategy='turn_pairs') # Returns immediately, enrichment queued manifest[item['question_id']] = project # Phase 2 — Wait for enrichment queue to drain while queue_status()['pending'] > 0: sleep(30) # Phase 3 — Query (enriched facts now searchable, ~40 min) for qid in manifest: chunks = recall_full(question, project=f'bench_{qid}', expand_queries=True) answer = generate_answer(question, chunks) ``` ### Timing estimate with batch mode | Phase | Time | |-------|------| | Ingest (53 sessions × ~0.5s × 500 items) | ~3.6 hours | | Enrichment (background, runs during ingest) | overlaps | | Query (500 items × ~5s) | ~40 min | | **Total** | **~4-5 hours** | This makes the v5-full benchmark feasible in a single overnight run. ### queue_status() tool still needed The runner needs to poll `queue_status()` between Phase 2 and Phase 3 to know when enrichment is complete and querying will return enriched results.

ric_harvey commented 2026-04-13 13:45:39 +00:00 (Migrated from codeberg.org)

Copy link

Update: ENRICHMENT_BATCH_MODE partially addresses this

ENRICHMENT_BATCH_MODE=true has been implemented and solves the bulk ingestion performance problem. During the LongMemEval benchmark:

• Phase 1 (ingest): remember_document() returns immediately, enrichment queued
• Phase 2: enrichment worker processes queue in background overnight
• Phase 3 (query): recall runs against enriched data after queue drains

This makes bulk benchmark ingestion feasible (~3.6 hours vs ~32 hours synchronous).

What remains open

ENRICHMENT_BATCH_MODE is an opt-in env var for bulk use cases. Regular daily use of remember() still hits the synchronous Haiku call (~1.2s) on every write:

Normal user storing a memory:
  remember("I moved to Edinburgh") → extract_facts() → 1.7s response
  
With ENRICHMENT_BATCH_MODE=true:
  remember("I moved to Edinburgh") → queue enrichment → 0.25s response ✓

The remaining work in this issue is making async enrichment the default behaviour for all remember() calls, not just when ENRICHMENT_BATCH_MODE is explicitly set. This would:

• Bring all single remember() calls from ~1.7s to ~0.25s
• Make OmniMem feel snappier in daily Claude conversation use
• Remove the need for users to know about ENRICHMENT_BATCH_MODE

The queue infrastructure is already in place — this is now a matter of making it the default path rather than opt-in.

## Update: ENRICHMENT_BATCH_MODE partially addresses this `ENRICHMENT_BATCH_MODE=true` has been implemented and solves the bulk ingestion performance problem. During the LongMemEval benchmark: - Phase 1 (ingest): `remember_document()` returns immediately, enrichment queued - Phase 2: enrichment worker processes queue in background overnight - Phase 3 (query): recall runs against enriched data after queue drains This makes bulk benchmark ingestion feasible (~3.6 hours vs ~32 hours synchronous). ## What remains open `ENRICHMENT_BATCH_MODE` is an opt-in env var for bulk use cases. Regular daily use of `remember()` still hits the synchronous Haiku call (~1.2s) on every write: ``` Normal user storing a memory: remember("I moved to Edinburgh") → extract_facts() → 1.7s response With ENRICHMENT_BATCH_MODE=true: remember("I moved to Edinburgh") → queue enrichment → 0.25s response ✓ ``` The remaining work in this issue is making async enrichment the **default behaviour** for all `remember()` calls, not just when `ENRICHMENT_BATCH_MODE` is explicitly set. This would: - Bring all single `remember()` calls from ~1.7s to ~0.25s - Make OmniMem feel snappier in daily Claude conversation use - Remove the need for users to know about `ENRICHMENT_BATCH_MODE` The queue infrastructure is already in place — this is now a matter of making it the default path rather than opt-in.

ric_harvey commented 2026-07-08 11:11:41 +00:00 (Migrated from codeberg.org)

Copy link

Closing — shipped across v5.2.0/v5.2.1. remember() and remember_document() in full mode store content raw (embed + write, fast) and return immediately with enrichment: "queued"; a background EnrichmentWorker daemon thread pops jobs from a Valkey-backed queue (queue:enrich, survives restarts) and writes extracted facts as linked memories. ENRICHMENT_BATCH_MODE=true collapses a whole document into one Haiku call, and queue_status() lets benchmark runners poll until the queue drains.

Two fixes landed since that are worth knowing about: v5.3.0 restored the near-duplicate guard that the async migration had dropped (re-remembering similar content was silently piling up duplicate facts), and v5.3.1 rerouted extracted facts to the knowledge namespace with inherited timestamps (#20).

Closing — shipped across v5.2.0/v5.2.1. `remember()` and `remember_document()` in full mode store content raw (embed + write, fast) and return immediately with `enrichment: "queued"`; a background `EnrichmentWorker` daemon thread pops jobs from a Valkey-backed queue (`queue:enrich`, survives restarts) and writes extracted facts as linked memories. `ENRICHMENT_BATCH_MODE=true` collapses a whole document into one Haiku call, and `queue_status()` lets benchmark runners poll until the queue drains. Two fixes landed since that are worth knowing about: v5.3.0 restored the near-duplicate guard that the async migration had dropped (re-remembering similar content was silently piling up duplicate facts), and v5.3.1 rerouted extracted facts to the knowledge namespace with inherited timestamps (#20).

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

v6.5.x

pages

v6.4.x

badges

v6

v5

v4

feat/rss-reader-improvements

v6.4.2

V6.3.2

v6.3.2

v6.1.0

v5.5.3

v3.13.1

v3.13.0

v3.12.1

v3.10.4

v3.10.2

v3.10.1

v3.10.0

v3.9.5

v3.8.2

v3.8.1

v3.8.0

v3.7.1

v3.7.0

v0.2.0

v0.1.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#19

No description provided.