OmniMem test harness

Black-box tests for OmniMem. One command downloads a release (or takes a local build), starts a fresh, named instance of it, loads an invented sample dataset, runs the suites, and writes everything to:

results/<instance-name>/<YYYY-MM-DD>/<HHMMSS>/

The suites come from the Agent Memory Atlas benchmarks page. The reasons for picking these and not the others are in PLAN.md.

Runs and suites still to do, LongMemEval among them, are in TODO.md.

Testing a version in CI

Push a tag named test-<version>:

git tag test-7.0.8 && git push origin test-7.0.8

The Harness workflow (.forgejo/workflows/harness.yml) then does four things:

1. It downloads omnimem-7.0.8-linux-amd64.tar.gz and checks it against the published .sha256.
2. It runs the core suites against an instance named after the tag.
3. It commits the run folder to the results branch.
4. It attaches the folder as a workflow artifact.

If any gate fails, the job turns red. The results are published either way.

When it publishes, the job also compares the run with the latest run of the highest earlier OmniMem version on the results branch that has the same enrichment setting. For test-7.0.9 that's the newest test-7.0.8 run. The comparison is written into the run folder as compare-<baseline>.html and compare-<baseline>.pdf (see Comparing two versions). The first run of all has nothing to compare with, so it skips this step.

To compare any two runs that are already on the results branch, start the Compare workflow (.forgejo/workflows/compare.yml) from the Actions page with Run workflow:

• Candidate: the run to check. Give an instance name such as test-7.0.9 for its latest run, or name/date/time for one particular run.
• Baseline: the run to compare it with, in the same form, or auto.

The workflow doesn't run any suites. It commits compare-<baseline>.html and .pdf into the candidate's run folder and attaches them to the workflow run.

To see what an API key changes, compare the two runs of one version: candidate test-7.0.9-enrich, baseline test-7.0.9. The page is then titled not enriched vs enriched. Every page says whether its runs were enriched, in the title, the lede and the provenance table.

You can run the same version again with test-7.0.8+rerun (the instance is then named test-7.0.8-rerun). You can also start a run from the Actions page with Run workflow, which lets you pick the suites.

The job runs on [linux, amd64] inside ci/Dockerfile, so the runner needs Docker and nothing else. To push to results, it uses the job's own token. If Forgejo's job token can't push to this repo, add a repository secret HARNESS_PUSH_TOKEN: a Forgejo access token with write access to this repository.

Testing with an Anthropic API key

Without a key, OmniMem stores memories exactly as written, and contradictions are linked only by its word-pair heuristic. With a key, it also extracts facts in the background, can expand queries, and has Claude confirm contradictions. That's a different configuration, so the harness never uses a key unless you ask. A keyed run is marked Enrichment: on in report.md and manifest.json. The key itself is never written anywhere.

In CI: add a repository secret named ANTHROPIC_API_KEY (Settings → Actions → Secrets). Then push a tag with +enrich in it:

git tag test-7.0.8+enrich && git push origin test-7.0.8+enrich   # instance test-7.0.8-enrich

You can also start the workflow by hand with enrichment ticked. Plain test-<version> tags never pass the secret to the container.

Locally: export the key and add --enrichment:

export ANTHROPIC_API_KEY=sk-ant-...
./omh run --name rel-7.0.8-enrich --omnimem 7.0.8 --suites core --enrichment

Without --enrichment, a key in your shell is stripped before the server starts. With it on, each suite waits for the enrichment queue to empty before it checks anything. S6 then also covers the Atlas's step 9 (facts extracted from a memory must go when it does), and S2 records whether Claude confirms each linked pair. S1 never gets the key: the pinned script strips it, and its judge calls go through the claude CLI.

Fact extraction calls Claude Haiku for every memory written, including the roughly 875-record baseline that each suite loads. Expect a few hundred Haiku calls per suite.

Running it locally

You need Python 3.13 and uv. This command runs from a checkout with the submodule present:

git submodule update --init
uv sync
./omh fetch --suites s4                                        # Lethe's ForgetEval, pinned
./omh run --name rel-7.0.8 --omnimem 7.0.8 --suites core       # a released build
./omh run --name main-head --binary ../omnimem/target/release/omnimem --suites smoke
./omh run --name span --omnimem 7.0.8 --config RECALL_EXPERIENCE_SPAN=0.05 --suites s5
./omh report results/rel-7.0.8/latest                          # re-render report.md
uv run pytest                                                  # the harness's own tests
OMH_BINARY=/path/to/omnimem uv run pytest -m integration       # plus an end-to-end smoke run

Each suite gets its own throwaway server on a port from 8770 to 8789. The harness refuses ports 6399, 8080 and 8765–8767, and refuses any port where something is already listening. It also checks that the store is empty before writing anything.

Suites

Suite	Tests	Gate
smoke	Remember one memory and recall it back	It comes back as the top hit
s2	The Atlas's Contradiction Test, without an API key: replacement, polarity and retraction pairs, at gaps of 1 minute and 120 days	No STALE value after resolution, maintenance or re-entry
s4	Lethe's ForgetEval (smoke set plus the adversarial layer), using its own case checks through an OmniMem adapter	Reported per family, not gated
s5	Ranking weights against configuration.md (recency curve, DEPRIORITISED_WEIGHT) and feed-article expiry	Expired articles are archived
s6	The Atlas's 13-step deletion sequence, in two arms: forget, and forget plus suppress_topic	The cells listed in PLAN.md §4.3
s7	Recall on questions the store knows nothing about	Nothing comes back above the weak band
s8	Project isolation in recall, recall_index, briefing and the PreToolUse hook	No cross-project leaks
s1	LongMemEval through omnimem's own run_memory_bench.py, unchanged. This spends money: about $30 at n=60	Memory beats no memory

The groups are core (s8, s6, s7, s5, s2, s4), judged (s1) and all.

Before any case runs, each suite runs a positive fixture (a case it must pass) and a negative fixture (a broken case it must fail). If either gives the wrong answer, the suite is reported INVALID, because a check that can't tell a pass from a failure proves nothing.

Comparing two versions

./omh compare results/rel-7.0.8/latest results/dev-7.0.9/latest   # baseline, then candidate
./omh compare auto results/dev-7.0.9/latest                        # the highest earlier version under results/
./omh run --name dev-7.0.9 --binary ../omnimem/target/release/omnimem --compare auto

This writes compare-<baseline>.html and an A4 compare-<baseline>.pdf into the candidate's run folder. The page uses the same styling as the OmniMem investor brief: Space Grotesk, IBM Plex Sans and JetBrains Mono, in ink and green on paper. Its sections are:

• the run status
• the gates held
• what got worse
• a chart of each suite's headline measure
• S4 by family
• every gate
• every measure
• the candidate's notes
• where each run came from

Every number is read from the two run folders. The fonts and logo are embedded from harness/assets/, so the page needs no network.

The PDF is printed by a headless Chromium. The harness looks for one in this order:

1. OMH_CHROMIUM
2. the PATH
3. /Applications
4. Playwright's cache

The CI image installs Debian's chromium. Pass --no-pdf for the HTML alone.

What a run folder holds

File	Contents
manifest.json	The instance name, the OmniMem version the server reported, the binary's sha256 and source, config overrides, and the harness and submodule commits
<suite>.jsonl	One record per case. Records are written as cases finish, so --resume <run_dir> can carry on after a crash
<suite>.summary.json	Counts, failing case IDs, fixture results, and p50/p95/p99 for settle time and latency
gates.json	Every gate, and the run's status: PASSED, FAILED, INVALID or ERROR
report.md	A table of every suite, then each gate
compare-<baseline>.html, .pdf	The comparison with an earlier run, when one was made

Layout

omh                       CLI entry point
harness/                  instance, adapters, suites, results, manifest
fixtures/build_baseline.py  generates fixtures/baseline/baseline.json from a fixed seed
vendor/omnimem/           omnimem, pinned as a submodule (its scripts/benchmarking is reused)
data/manifest.lock        pinned third-party data; everything else under data/ is fetched
.forgejo/workflows/       the tag-triggered CI run
docs/review/              the design review of PLAN.md