Research & benchmarks

Numbers you can rerun.

Every claim about Neurel's memory is measured, compared with a baseline, and stored in the repo with the commands that produced it.

At a glance

LongMemEval_S answer accuracy, all 50094.0%
Held out, run once93.8% (n = 420)
Right conversation in the top 598.3%
Recall latency, median19 ms
ReaderClaude Haiku 5.5, about 0.6¢ per question with the judge

How we report

  • Each experiment states the question, the method, the baselines, the results, the limitations and how to reproduce it.
  • Results are committed as JSON in bench/results/. This page is built from that file.
  • Datasets are pinned to a revision and a sha256. A run on different bytes refuses to start.
  • If a baseline beats Neurel, we publish it as measured.
Experiment 03 · run 2026-11-18 · reader claude-haiku-5-5 · judge gpt-6.1-terra

LongMemEval: does the assistant answer correctly from what Neurel recalls?

Question

Finding the right conversation is half the job. Given what Neurel recalls, does an assistant actually answer the question correctly? This is the number LongMemEval reports and other memory systems publish.

Method

  • Dataset: all 500 LongMemEval_S questions, including the 30 that ask about something never said, where the right answer is "I don't know".
  • Memory: each question's history is written into a fresh vault (one page per session, one timeline entry per turn, stamped with the session's date and time) and recalled through the production recall, at limit 20.
  • What the reader sees: the top 10 recalled sessions in full, as get_page returns them, and ranks 11–20 as the matched turn with one turn either side. Sessions are in date order with their dates, as LongMemEval does it. About 30k tokens per question.
  • Prompts: LongMemEval's own, unchanged: the step-by-step reader prompt and the per-type judge prompts from the official repository (commit 9e0b455). The question date is passed as "Current Date".
  • Reader: Claude Haiku 5.5, the cheapest model we tried. Judge: GPT-6.1 Terra, from a different vendor than the reader so no model grades its own family.
  • Split: 80 dev questions (stratified by type, chosen by a hash of the question id) for every decision, and 420 held out. The setup was frozen on dev, then the held-out set was run once.

Baselines

There is no official leaderboard. These are the strongest self-reported LongMemEval answer accuracies we found. They use different readers, judges and context budgets, so they are reference points, not a ranking.

SystemReported accuracy
Maximem Synap95.6%
Agent Zero Memory95.6%
Mastra Observational Memory (gpt-5-mini)94.87%
Hindsight94.6%
Mem094.4%
ByteRover92.8%
Neurel (Haiku 5.5 reader, all 500)94.0%

Results

SetnAccuracy95% interval
Held out (run once)42093.8%± 2.3
All 50050094.0%± 2.1
Dev, two samples8095.0% / 93.8%± 4.8

By question type

Typen (all)All 500Held out
knowledge update7298.6%98.3%
multi session12187.6%86.3%
single session assistant56100.0%100.0%
single session preference3096.7%96.0%
single session user6498.4%98.1%
temporal reasoning12793.7%94.4%
abstention3086.7%88.0%

What we tried that did not make it

Each idea ran on the 80 dev questions. Run twice, the same setup flips 5 of 80 answers by itself, so a change counts only if it fixes questions both baseline runs missed and breaks none both got right.

ChangeDev accuracyFixedBroke
Skill rules: "prefer the latest dated fact", "flag a false premise", "answer plainly"90.0%13
GPT-6.1 Terra as the reader instead of Haiku 5.592.5%12
Session time of day shown to the reader91.3%02
Time-aware recall ("four weeks ago" counted from the question date)95.0%02
Both of the above92.5%02

What we found

  • The cheapest reader is enough. Haiku 5.5 reaches 94.0% on all 500 questions for about $2.79 in answer and judge calls. GPT-6.1 Terra cost five times more on dev and did no better: it hedged where the answer was there ("the brand wasn't specified, Trader Joe's is the store").
  • Multi-session questions are the weak spot (87.6%): counting or adding up across several conversations fails if one is missed or misread.
  • None of our three improvements survived. Answering rules, time of day and time-aware recall each broke more than they fixed. Time-aware recall also lowered retrieval (experiment 02), so it was taken out. Session times are still recorded, because they cost nothing and users can see them.
  • Within reach of the top, not above it. The best self-reported scores sit about one to two points higher, inside our own margin of error.

Limitations

  • The judge is GPT-6.1 Terra, not the GPT-4o snapshot LongMemEval's paper uses. Scores judged by different models are not strictly comparable.
  • One held-out run. A second sample would move individual answers, and the total by about a point.
  • The recall fixes in experiment 02 were measured on all 470 retrieval questions, held-out ones included, before the dev/held-out split existed. Every answer-stage decision used dev only.
  • Raw transcripts are written straight into the vault. Real clients remember distilled facts and the nightly dream consolidates them; neither is tested here.
  • The reader is told each session's date but not its time; showing the time did not help (above).

Reproduce

Needs ANTHROPIC_API_KEY and OPENAI_API_KEY in .env. Held out cost $2.49; dev about $0.30 per run. Every call is cached, so a rerun only pays for what changed.

bun bench/longmemeval.ts --tag time      # retrieval + reader contexts, free, about 2 hours
bun bench/qa.ts --contexts time --split dev  --max-usd 1
bun bench/qa.ts --contexts time --split test --max-usd 5
# → bench/results/longmemeval-s-qa-time-haiku-5.5-full10of20-{dev,test}.json

The results files keep every answer and verdict. The split is in bench/splits/longmemeval-s.json.

Experiment 02 · run 2026-11-18

Recall fixes: fill the list, drop the placeholder

Question: experiment 01 found recall returning fewer sessions than asked for, and an identicalnone yet chunk on every page. Does fixing both help? Method: recall now widens its candidate pool until it has as many sessions as asked for (up to 1,024 chunks), and the placeholder is no longer indexed. Same harness and data as experiment 01. We also tried two time features on top. Baseline: experiment 01.

RecallR_any@5R_all@10R_all@20Chunks / qShort lists
Experiment 01 (before)98.7%95.5%–540133
Wider pool, no placeholder chunk98.7%96.6%98.1%4930
+ entries stamped with time of day (shipped)98.3%96.2%98.3%4930
+ time-aware recall (reverted)99.1%94.5%98.1%493–
  • Short lists are gone and Recall_all@10 now matches the flat dense baseline; one fewer chunk per page also makes indexing lighter.
  • Time-aware recall cost more than it gained. It fixed "the appliance I bought 10 days ago", but asking from the question's date also woke the recency boost, which pushed older evidence out of the top 10 on questions with no time phrase at all. It was reverted.
  • Limitation: the same as experiment 01. Retrieval is deterministic; the small differences between the last two shipped rows come from the timestamps now being part of the embedded text.
Experiment 01 · run 2026-11-17 · src/ at 7f69626

LongMemEval: does recall find the conversation that holds the answer?

Question

A user has talked to their assistant across about 48 sessions, roughly 115k tokens. Later they ask something only one or two of those sessions can answer. When the assistant calls recall with that question, does Neurel return the right session?

Method

  • Dataset: LongMemEval_S, cleaned(MIT), revision 98d7416. 470 questions. The 30 abstention questions are left out: they ask about something the user never said, so the right behaviour is to answer "I don't know", not to find a session.
  • Ingest: each question gets a fresh vault. Every session becomes one page, every turn (user and assistant) one timeline entry, then the server's own reindex embeds them into Qdrant. About 540 chunks per question.
  • Query: the question goes through the production recall path, with ranking, page dedup and link spread, at limits 1, 5 and 10, with track: false so one query cannot influence the next.
  • Metrics: session-level Recall_any@k (at least one evidence session in the top k), Recall_all@k (all of them) and NDCG@k. No LLM is involved.
  • Stack: bge-small-en-v1.5 (ONNX int8, CLS pooling, 512-token cap); Qdrant 1.18.3, collection configured as in production (int8 quantization, on-disk HNSW).

Baselines

  • Flat dense: the same vectors Neurel stored, read back and ranked by exact cosine. No HNSW, no quantization, no strength, recency or link spread. It tells us what Neurel's own machinery adds or costs.
  • BM25: keyword search (Okapi, k1 1.2, b 0.75) over the same turns. The classic lexical baseline.

All three rank the same chunks and score a session by its best chunk.

Results

SystemR_any@1R_any@5R_any@10R_all@5R_all@10NDCG@10
Neurel recall90.4%98.7%99.1%90.9%95.5%93.6%
Flat dense (baseline)90.4%98.7%99.1%91.1%96.0%93.6%
BM25 (baseline)85.7%94.7%97.7%79.4%89.4%88.2%

n = 470. Best in each column highlighted.

By question type (R_any@5)

TypenNeurel recallFlat denseBM25
knowledge update72100.0%100.0%100.0%
multi session12199.2%99.2%95.9%
single session assistant56100.0%100.0%100.0%
single session preference30100.0%100.0%63.3%
single session user64100.0%100.0%100.0%
temporal reasoning12796.1%96.1%92.9%

Speed

Recall answered in 19 ms at the median and 23 ms at p95 (limit 10, local machine). Indexing a 540-chunk history took 12.6 s, almost all of it embedding. The full run took 100 minutes.

What we found

  • The right session is almost always there. Neurel puts at least one evidence session in the top 5 for 98.7% of questions, against 94.7% for BM25. The widest gap is preference questions (100.0% vs 63.3%), where the question rarely uses the words the user once said.
  • The gain is the embedder's, not Neurel's ranking. Neurel and flat dense are identical on Recall_any at every k. Strength, recency and link spread have nothing to act on here (see limitations), so this run measures Neurel's retrieval floor, not its ceiling.
  • Neurel loses slightly where several sessions are needed. Recall_all@10 is 95.5% against 96.0% for flat dense, mostly on temporal reasoning (90.6% vs 92.1%). Recall fetches 4 × limit chunks and keeps one per page; when one session matches on many turns, fewer distinct sessions survive. Asked for 10, it returned fewer in 133 of 470 questions, and 2 of the 2 questions where flat dense found every session and Neurel did not were such short lists. Experiment 02 widens that pool.

Limitations

  • This is retrieval recall, not a LongMemEval score. The official score is question-answering accuracy judged by an LLM; finding the right session is necessary for that, not sufficient.
  • Raw transcripts are written straight into the vault. In real use a client calls remember with distilled facts and the nightly dream consolidates them, which this run skips.
  • Each turn is cut at 512 tokens by the embedder. Long assistant answers lose their tail.
  • With no links between pages and every memory equally old and equally used, recency, strength and spread barely act here. They are built for a vault that has lived, which this one has not.
  • About 540 chunks per question is small for a vector index. At this size Qdrant may not build the HNSW graph at all, so approximate search at production scale is not tested here.
  • A page with no Compiled Truth yet is saved with a none yet placeholder, and that placeholder is indexed as a chunk, here as in production until the dream fills it in. Every session carries one such identical chunk. Fixed in experiment 02.
  • One run on one machine. Retrieval is deterministic, latency is not.

Reproduce

Free: no API keys, no LLM calls. About 100 minutes on a laptop CPU.

# From the Neurel project directory
bun install
bun run fetch-model                 # bge-small-en-v1.5 ONNX
bun bench/fetch.ts                  # LongMemEval_S, sha256-checked
docker run -d --rm --name neurel-bench-qdrant -p 127.0.0.1:6399:6333 qdrant/qdrant:v1.18.3
bun bench/longmemeval.ts            # --limit 20 for a quick subset
# → bench/results/longmemeval-s.json

The harness is bench/longmemeval.ts. The results file keeps every question's ranked sessions, so any metric can be recomputed without rerunning.