Benchmarks

ZMem should compete on benchmark accuracy, latency, token efficiency, and proof-backed reproducibility. The current benchmark harness is designed for internal development and honest public evidence, not unsupported leaderboard claims.

Current Local LoCoMo Evidence

ZMem has a verified four-mode baseline over the same 1,986 questions, SHA-pinned LoCoMo conversion, code checkpoint (871f23d), seed (42), and provisional local scorer. A verified follow-up uses the same dataset, split, seed, and scorer to measure the next adaptive-routing checkpoint:

Mode	Accuracy	Mean query tokens	Single-hop	Multi-hop	Temporal	Open-domain	Adversarial abstention
fts	0.5967	536.0	0.6005	0.0709	0.6293	0.1250	1.0000
fts-multihop	0.6067	425.8	0.6219	0.0780	0.6168	0.1667	1.0000
pseudo-embedding	0.5967	536.0	0.6005	0.0709	0.6293	0.1250	1.0000
pseudo-embedding-rerank	0.5967	536.0	0.6005	0.0709	0.6293	0.1250	1.0000
fts-adaptive	0.6143	533.3	0.6290	0.0780	0.6417	0.1771	1.0000

Always-on fts-multihop improved overall accuracy by 0.0101 and reduced mean query tokens by 20.6% relative to fts, but it gained 98 questions and lost 78, including a net four-question temporal regression. The first conservative adaptive checkpoint gained 29 and lost 1 against FTS.

The v0.1.4 morphology follow-up adds only regular inflection matches behind a two-exact-anchor and two-match-gain gate. It passed the stable 227-question cohort at 158/227 versus 156/227, then improved full adaptive LoCoMo from 1,213 to 1,218 correct with five gains and zero losses. Three gains are single-hop and two are temporal. Mean query tokens moved from about 532.8 to 533.3.

The next completion-support checkpoint bridges one already-retrieved subject/object nucleus to at most one paraphrased completion fact. It moved the stable cohort from 158/227 to 159/227 and full adaptive LoCoMo from 1,218 to 1,219 correct, with one gain and zero losses in each comparison. LongMemEval remained 386/500. The rule and selected support are recorded in the retrieval receipt under zerker.support_expansion.v1.

The transcript-neighbor checkpoint adds one narrower bridge for structured onset questions. It requires an exact event head in both memories, the same speaker, transcript session, and timestamp, and an earlier turn no more than two positions away. It moved the stable cohort from 159/227 to 160/227 and full adaptive LoCoMo from 1,219 to 1,220 correct. Exactly one answer and one retrieval context changed, there were zero losses, and LongMemEval remained retrieval-identical at 386/500. This is bounded transcript support, not unrestricted adjacency or general graph traversal.

Pseudo-embedding and pseudo-rerank match plain FTS on every scored category and add no measured quality on this deterministic local path. The baseline matrix/comparison hashes are 9f8b77ca... and 18cdca15...; the adaptive follow-up hashes are 7e8825aa... and 3bcd5d22.... Both use conversation-bounded ephemeral stores and leave zero SQLite artifacts.

These values use ZMem's provisional-local retrieval-recall scorer. The receipts set public_benchmark_claim: false. An older token F1/EM artifact (0.3752 F1, 0.3721 EM) also exists, but it uses a different scoring path and must not be compared numerically with the accuracy table above.

Run An Isolated LoCoMo Matrix

Use a new run id for every matrix and keep concurrent jobs out of the same target. For LoCoMo, --compact-artifacts uses one ephemeral store per conversation and leaves no run database behind; it preserves result hashes, traces, summaries, and receipt commitments without writing a bundle for every question. When per-question bundles are enabled, they use compact zerker.receipt_bundle.v2 event witnesses by default instead of embedding the full pre-action history.

RUN_ID="locomo-$(date -u +%Y%m%dT%H%M%SZ)"

zmem bench matrix locomo \
  --dataset data/locomo/locomo_official_zmem.json \
  --out .zerker/bench/runs \
  --run-id "$RUN_ID" \
  --seed 42 \
  --compact-artifacts \
  --summary-only

That single command runs the local mode matrix sequentially and writes comparison artifacts under .zerker/bench/runs/$RUN_ID/. Do not launch another job with the same run id.

To isolate one mode, give it its own run id:

zmem bench matrix locomo \
  --dataset data/locomo/locomo_official_zmem.json \
  --out .zerker/bench/runs \
  --run-id "${RUN_ID}-adaptive" \
  --mode fts-adaptive \
  --seed 42 \
  --compact-artifacts \
  --summary-only

A one-mode matrix records the verified result once; it does not compare the result to itself. Summary-only output bounds long question-id lists and per-question delta rows to ten examples plus an omitted count.

Render and verify after the matrix finishes:

zmem bench report ".zerker/bench/runs/$RUN_ID" --summary-only
zmem bench dashboard ".zerker/bench/runs/$RUN_ID"
zmem bench public-page ".zerker/bench/runs/$RUN_ID"
zmem bench verify ".zerker/bench/runs/$RUN_ID/benchmark-matrix.json" --summary-only

Category Deltas

The harness already writes per-category summaries and comparison rows. Use the rendered report/dashboard to answer:

• does adaptive routing improve multi-hop,
• does it preserve temporal,
• does it avoid unnecessary decomposition,
• does any mode improve adversarial abstention.

There is not a category-filter CLI yet. If full runs are too expensive, add category filtering before running many category-only sweeps.

Token Budget Sweep

Use distinct run ids at a few budgets to separate "retrieval found the wrong thing" from "retrieval found the right thing but packing dropped it."

zmem bench matrix locomo \
  --dataset data/locomo/locomo_official_zmem.json \
  --out .zerker/bench/runs \
  --run-id "locomo-budget-384-$(date -u +%Y%m%dT%H%M%SZ)" \
  --mode fts-multihop \
  --seed 42 \
  --context-budget-tokens 384 \
  --compact-artifacts

zmem bench matrix locomo \
  --dataset data/locomo/locomo_official_zmem.json \
  --out .zerker/bench/runs \
  --run-id "locomo-budget-768-$(date -u +%Y%m%dT%H%M%SZ)" \
  --mode fts-multihop \
  --seed 42 \
  --context-budget-tokens 768 \
  --compact-artifacts

Add the same sweep for pseudo-embedding-rerank only if multihop shows signal or budget-dropped context appears in the reports.

ActiveGraph Compact Trace

Use ActiveGraph for storage-safe trace runs and use-case proof. It writes trace.jsonl and scored_receipt.json instead of per-question receipt bundles.

Smoke first:

zmem-bench-locomo \
  --dataset data/locomo/locomo_official_zmem.json \
  --out .zerker/bench/activegraph-locomo-smoke \
  --run-id activegraph-smoke-fts-multihop \
  --retrieval-mode fts-adaptive \
  --split default \
  --event-batch-size 128

The full 227-question acceptance run produced 908 replayable events in eight commits, a 1 MB causal event database, a 196 KB trace, and zero receipt bundles. Its deterministic answerer is a trace smoke, not an official quality score. See ActiveGraph for the real pack-loader verification flow.

BEAM Scale

Point the adapter at the official BEAM repository's chats/<scale> directory or one conversation inside it. ZMem does not download or redistribute the dataset.

RUN_ID="beam-100k-$(date -u +%Y%m%dT%H%M%SZ)"

zmem bench run beam \
  --dataset /path/to/BEAM/chats/100K \
  --split 100K \
  --out .zerker/bench/runs \
  --run-id "$RUN_ID" \
  --retrieval-mode fts-adaptive \
  --context-budget-tokens 4096 \
  --trace \
  --compact-artifacts

zmem bench verify \
  ".zerker/bench/runs/$RUN_ID/benchmark-result.json" \
  --summary-only

The adapter supports 100K, 500K, 1M, and 10M. It hashes each chat and probing file, records observed messages/tokens/bytes, preserves official source chat ids, and reports source-reference coverage.

The first untouched official 100K smoke covered 188 messages, 63,411 observed whitespace tokens, 20 probing questions across all ten categories, and 53/53 resolved source references. The reusable trace was 31 KB and the complete proof directory was about 700 KB.

The first isolated official-layout 500K conversation run covered 796 messages, 247,175 observed whitespace tokens, 20 probing questions, and 83/83 resolved source references. It measured 0.30 deterministic local evidence recall with p50/p95 retrieval latency of 265.2 ms / 821.4 ms. Both BEAM runs are local scale/evidence instrumentation with public_benchmark_claim: false, not official model-judged BEAM scores.

The isolated 1M conversation covered 1,802 messages, 490,991 observed whitespace tokens, 20 probing questions, and 107/107 resolved references. Its verified compact proof directory is about 756 KB and reports 0.15 local evidence recall.

The isolated 10M conversation covered 19,895 messages, 6,209,948 observed whitespace tokens, 20 probing questions, and 201/201 resolved references. Its verified compact proof directory is about 656 KB and reports 0.10 local evidence recall. An additive events(memory_id, seq) index removed a correlated observation-order scan: the same fts-adaptive event-ordering question fell from 112,976 ms to 8,344 ms (13.54x) with the same outcome and the same retrieved/injected memories.

These runs prove bounded local execution, complete source-reference resolution, and compact verifiable artifacts at the sampled scales. They also expose the remaining product gap: evidence recall is weak at 10M. They are not official model-judged BEAM scores or evidence that long-context retrieval quality is solved.

Synthetic Matrix

zmem bench matrix synthetic \
  --out .zerker/bench \
  --seed 0 \
  --run-id synthetic-local

Generate the engineering and public evidence surfaces:

zmem bench dashboard .zerker/bench/synthetic-local
zmem bench public-page .zerker/bench/synthetic-local

Verify Evidence

zmem bench verify .zerker/bench/synthetic-local/fts/benchmark-result.json
zmem bench verify .zerker/bench/synthetic-local/fts-multihop/benchmark-result.json

LongMemEval

The verified local 500-question matrix uses the same dataset, seed, code, and provisional scorer for all modes:

Mode	Accuracy	Mean query tokens	Knowledge update	Multi-session	Temporal reasoning	Abstention
fts	0.740	2452.1	0.7821	0.5338	0.7143	1.0000
fts-multihop	0.780	2550.5	0.8462	0.5865	0.7669	1.0000
pseudo-embedding	0.740	2452.1	0.7821	0.5338	0.7143	1.0000
pseudo-embedding-rerank	0.740	2452.1	0.7821	0.5338	0.7143	1.0000
fts-adaptive	0.772	2510.6	0.8205	0.5714	0.7744	1.0000

Always-on multihop recovers 20 questions with zero losses: +5 knowledge update, +7 multi-session, +7 temporal, and +1 single-session user recall. The first adaptive checkpoint kept 13 of those gains with zero losses. The v0.1.4 morphology gate adds three more regression-free wins, one knowledge update and two temporal, for 386/500. It costs 11,968 additional tokens across the run.

With --compact-artifacts, LongMemEval uses one ephemeral store per session, preserves reuse within a session, and writes no run database or per-question receipt bundles. The complete four-mode matrix finished in 174.84s and independently verified. Normal runs still retain the shared database, snapshots, and proof bundles.

Download LongMemEval separately, then convert the local official file. ZMem does not download or redistribute the dataset:

python3 scripts/bench/longmemeval_to_zmem.py \
  --in /path/to/longmemeval_oracle.json \
  --out data/longmemeval/longmemeval_oracle_zmem.json

zmem bench matrix longmemeval \
  --dataset data/longmemeval/longmemeval_oracle_zmem.json \
  --split default \
  --out .zerker/bench/runs \
  --seed 42 \
  --run-id "longmemeval-$(date -u +%Y%m%dT%H%M%SZ)" \
  --compact-artifacts \
  --summary-only

The same flag also works with zmem bench run longmemeval, so the reproducibility command recorded in each compact manifest can be executed directly.

LoCoMo-Style Fixtures

zmem bench matrix locomo \
  --dataset /path/to/local-locomo.jsonl \
  --split dev \
  --out .zerker/bench \
  --seed 0 \
  --run-id locomo-dev-local

Frontier Queue

Next benchmark priorities:

• Stable-miss analysis: use the 768 adaptive LoCoMo misses and 114 adaptive LongMemEval misses to select one bounded retrieval-quality slice.
• LoCoMo multi-hop and open-domain: improve the still-low categories without giving back temporal accuracy.
• BEAM: broaden the verified 1M and 10M evidence across more conversations and add an official model-judged scoring path before making scale-quality claims.

Public Claim Rules

Allowed before official submissions:

• "ZMem publishes proof-backed local benchmark evidence."
• "This matrix is reproducible from the attached artifact hashes and receipts."
• "This local scaffold tracks retrieval accuracy, latency, tokens, and proof verification."

Do not claim official LongMemEval or LoCoMo ranking, vendor superiority, or canonical leaderboard score from local scaffold output.

What To Track

• accuracy and category accuracy,
• recall and precision evidence where ground-truth support exists,
• p50, p95, and p99 retrieval latency,
• total tokens and injected context tokens,
• retrieved, injected, withheld, and budget-dropped memory counts,
• verification status,
• matrix hash and comparison hash,
• optional Treeship proof URL when public proof publishing is enabled.