ric/omnimem

Watch

Fork 0

feat: bulk delete by project name #14

New issue

Closed

opened 2026-04-08 15:15:30 +00:00 by ric_harvey · 2 comments

ric_harvey commented

2026-04-08 15:15:30 +00:00

(Migrated from codeberg.org)

Problem

There is currently no way to delete all memories belonging to a project in a single operation. The only deletion path is forget which operates on individual keys.

This became a real problem during the LongMemEval benchmark run (~500 items × 53 sessions = ~26,500 memories). The cleanup strategy in the runner used recall_index with project_filter to find keys, then deleted them one by one. This has two failure modes:

Semantic search misses — if recall returns zero results for a project (45% of items in our run), those memories are never found and never deleted, leaving thousands of orphaned memories in Valkey.
Performance — deleting 26,500 memories individually via MCP at ~0.1s per call takes ~45 minutes.

After the benchmark run, OmniMem had 24,176 memories instead of the expected ~200, requiring a separate cleanup script to recover.

Proposed solution

Add a delete_project MCP tool (and corresponding internal method) that:

delete_project(project: str, confirm: bool = False) -> {"deleted": int}

Looks up all Valkey keys tagged with the given project directly (no semantic search)
Deletes them in batch
Returns count of deleted memories
Requires confirm=True as a safety gate (consistent with forget)

Optionally also support a delete_by_tag variant for similar bulk cleanup use cases.

Why not just use forget_project workaround?

The current workaround of querying semantically then deleting is fundamentally broken for this use case — you can't reliably find memories by querying for them if the whole point is that recall doesn't work well on that content. Direct project-scoped deletion needs to bypass the search layer entirely and operate on the metadata index.

Context

Discovered during the OmniMem × LongMemEval benchmark project. Related to issue #11 (Valkey index drift).

## Problem There is currently no way to delete all memories belonging to a project in a single operation. The only deletion path is `forget` which operates on individual keys. This became a real problem during the LongMemEval benchmark run (~500 items × 53 sessions = ~26,500 memories). The cleanup strategy in the runner used `recall_index` with `project_filter` to find keys, then deleted them one by one. This has two failure modes: 1. **Semantic search misses** — if recall returns zero results for a project (45% of items in our run), those memories are never found and never deleted, leaving thousands of orphaned memories in Valkey. 2. **Performance** — deleting 26,500 memories individually via MCP at ~0.1s per call takes ~45 minutes. After the benchmark run, OmniMem had 24,176 memories instead of the expected ~200, requiring a separate cleanup script to recover. ## Proposed solution Add a `delete_project` MCP tool (and corresponding internal method) that: ``` delete_project(project: str, confirm: bool = False) -> {"deleted": int} ``` - Looks up all Valkey keys tagged with the given project directly (no semantic search) - Deletes them in batch - Returns count of deleted memories - Requires `confirm=True` as a safety gate (consistent with `forget`) Optionally also support a `delete_by_tag` variant for similar bulk cleanup use cases. ## Why not just use forget_project workaround? The current workaround of querying semantically then deleting is fundamentally broken for this use case — you can't reliably find memories by querying for them if the whole point is that recall doesn't work well on that content. Direct project-scoped deletion needs to bypass the search layer entirely and operate on the metadata index. ## Context Discovered during the OmniMem × LongMemEval benchmark project. Related to issue #11 (Valkey index drift).

ric_harvey commented

2026-04-08 15:30:43 +00:00

(Migrated from codeberg.org)

Workaround until this is implemented

Direct Valkey Lua script — bypasses OmniMem entirely and operates on the hash fields directly. Clears 24k memories in seconds.

Step 1 — write the script into the container:

docker compose -f ~/docker-compose.yml exec valkey \
  sh -c 'cat > /tmp/cleanup_bench.lua << EOF
local cursor = "0"
local deleted = 0
repeat
  local result = redis.call("SCAN", cursor, "MATCH", "mem:episodic:*", "COUNT", "200")
  cursor = result[1]
  local keys = result[2]
  for _, key in ipairs(keys) do
    local project = redis.call("HGET", key, "project")
    if project and string.sub(project, 1, 6) == "bench_" then
      redis.call("DEL", key)
      deleted = deleted + 1
    end
  end
until cursor == "0"
return deleted
EOF'

Step 2 — run it:

docker compose -f ~/docker-compose.yml exec valkey \
  valkey-cli -a YOUR_PASSWORD --no-auth-warning \
  --eval /tmp/cleanup_bench.lua 0

Returns the count of deleted keys. The prefix match (bench_) can be adjusted for other project naming patterns.

Note: this only deletes the raw hash keys. A proper delete_project implementation in OmniMem should also handle the search index cleanup to avoid the index drift described in issue #11.

## Workaround until this is implemented Direct Valkey Lua script — bypasses OmniMem entirely and operates on the hash fields directly. Clears 24k memories in seconds. **Step 1 — write the script into the container:** ```bash docker compose -f ~/docker-compose.yml exec valkey \ sh -c 'cat > /tmp/cleanup_bench.lua << EOF local cursor = "0" local deleted = 0 repeat local result = redis.call("SCAN", cursor, "MATCH", "mem:episodic:*", "COUNT", "200") cursor = result[1] local keys = result[2] for _, key in ipairs(keys) do local project = redis.call("HGET", key, "project") if project and string.sub(project, 1, 6) == "bench_" then redis.call("DEL", key) deleted = deleted + 1 end end until cursor == "0" return deleted EOF' ``` **Step 2 — run it:** ```bash docker compose -f ~/docker-compose.yml exec valkey \ valkey-cli -a YOUR_PASSWORD --no-auth-warning \ --eval /tmp/cleanup_bench.lua 0 ``` Returns the count of deleted keys. The prefix match (`bench_`) can be adjusted for other project naming patterns. Note: this only deletes the raw hash keys. A proper `delete_project` implementation in OmniMem should also handle the search index cleanup to avoid the index drift described in issue #11.

ric_harvey commented

2026-07-08 11:08:13 +00:00

(Migrated from codeberg.org)

Shipped in v5.4.0 on branch v5 (commit 08b1554).

delete_project(project_name, confirm=False, include_context=False) is now a registered MCP tool:

Direct key scan, no semantic search — memories are found by scanning all four namespaces and matching the project/project_name fields with a projected batch fetch, so it catches everything, including memories recall can't surface (the failure mode that left ~24k orphans after the benchmark run).
Pipelined batch deletion — the new ValkeyStore.delete_many() deletes in batches of 500 per pipeline round-trip. Verified live against valkey-search: 1,200 keys in ~15ms, so the 26,500-memory cleanup that took ~45 minutes would now finish in well under a second.
Safety gates — confirm=False (default) returns a preview with per-namespace counts; the project context entry (mem:project:<name>) is kept unless you pass include_context=True; project names are validated against the standard allowlist.
Deleting also invalidates the recall pipeline's abandoned-approach cache, so dead ends recorded on deleted memories stop surfacing immediately.

The optional delete_by_tag variant mentioned in the proposal wasn't included — happy to raise a separate issue if it's still wanted, but delete_project covers the benchmark-cleanup case that motivated this.

589 tests pass (8 new covering preview counts, cross-namespace deletion, context-entry handling, cache invalidation, and name validation).

Shipped in **v5.4.0** on branch `v5` (commit 08b1554). `delete_project(project_name, confirm=False, include_context=False)` is now a registered MCP tool: - **Direct key scan, no semantic search** — memories are found by scanning all four namespaces and matching the `project`/`project_name` fields with a projected batch fetch, so it catches everything, including memories recall can't surface (the failure mode that left ~24k orphans after the benchmark run). - **Pipelined batch deletion** — the new `ValkeyStore.delete_many()` deletes in batches of 500 per pipeline round-trip. Verified live against valkey-search: 1,200 keys in ~15ms, so the 26,500-memory cleanup that took ~45 minutes would now finish in well under a second. - **Safety gates** — `confirm=False` (default) returns a preview with per-namespace counts; the project context entry (`mem:project:<name>`) is kept unless you pass `include_context=True`; project names are validated against the standard allowlist. - Deleting also invalidates the recall pipeline's abandoned-approach cache, so dead ends recorded on deleted memories stop surfacing immediately. The optional `delete_by_tag` variant mentioned in the proposal wasn't included — happy to raise a separate issue if it's still wanted, but `delete_project` covers the benchmark-cleanup case that motivated this. 589 tests pass (8 new covering preview counts, cross-namespace deletion, context-entry handling, cache invalidation, and name validation).

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ric/omnimem#14

No description provided.

Rows
Columns