Numbers you can rerun.
Every claim about Neurel's memory is measured, compared with a baseline, and stored in the repo with the commands that produced it.
At a glance
| LongMemEval_S answer accuracy, all 500 | 94.0% |
| Held out, run once | 93.8% (n = 420) |
| Right conversation in the top 5 | 98.3% |
| Recall latency, median | 19 ms |
| Reader | Claude Haiku 5.5, about 0.6¢ per question with the judge |
How we report
- Each experiment states the question, the method, the baselines, the results, the limitations and how to reproduce it.
- Results are committed as JSON in
bench/results/. This page is built from that file. - Datasets are pinned to a revision and a sha256. A run on different bytes refuses to start.
- If a baseline beats Neurel, we publish it as measured.
LongMemEval: does the assistant answer correctly from what Neurel recalls?
Question
Finding the right conversation is half the job. Given what Neurel recalls, does an assistant actually answer the question correctly? This is the number LongMemEval reports and other memory systems publish.
Method
- Dataset: all 500 LongMemEval_S questions, including the 30 that ask about something never said, where the right answer is "I don't know".
- Memory: each question's history is written into a fresh vault (one page per session, one timeline entry per turn, stamped with the session's date and time) and recalled through the production
recall, at limit 20. - What the reader sees: the top 10 recalled sessions in full, as
get_pagereturns them, and ranks 11–20 as the matched turn with one turn either side. Sessions are in date order with their dates, as LongMemEval does it. About 30k tokens per question. - Prompts: LongMemEval's own, unchanged: the step-by-step reader prompt and the per-type judge prompts from the official repository (commit
9e0b455). The question date is passed as "Current Date". - Reader: Claude Haiku 5.5, the cheapest model we tried. Judge: GPT-6.1 Terra, from a different vendor than the reader so no model grades its own family.
- Split: 80 dev questions (stratified by type, chosen by a hash of the question id) for every decision, and 420 held out. The setup was frozen on dev, then the held-out set was run once.
Baselines
There is no official leaderboard. These are the strongest self-reported LongMemEval answer accuracies we found. They use different readers, judges and context budgets, so they are reference points, not a ranking.
| System | Reported accuracy |
|---|---|
| Maximem Synap | 95.6% |
| Agent Zero Memory | 95.6% |
| Mastra Observational Memory (gpt-5-mini) | 94.87% |
| Hindsight | 94.6% |
| Mem0 | 94.4% |
| ByteRover | 92.8% |
| Neurel (Haiku 5.5 reader, all 500) | 94.0% |
Results
| Set | n | Accuracy | 95% interval |
|---|---|---|---|
| Held out (run once) | 420 | 93.8% | ± 2.3 |
| All 500 | 500 | 94.0% | ± 2.1 |
| Dev, two samples | 80 | 95.0% / 93.8% | ± 4.8 |
By question type
| Type | n (all) | All 500 | Held out |
|---|---|---|---|
| knowledge update | 72 | 98.6% | 98.3% |
| multi session | 121 | 87.6% | 86.3% |
| single session assistant | 56 | 100.0% | 100.0% |
| single session preference | 30 | 96.7% | 96.0% |
| single session user | 64 | 98.4% | 98.1% |
| temporal reasoning | 127 | 93.7% | 94.4% |
| abstention | 30 | 86.7% | 88.0% |
What we tried that did not make it
Each idea ran on the 80 dev questions. Run twice, the same setup flips 5 of 80 answers by itself, so a change counts only if it fixes questions both baseline runs missed and breaks none both got right.
| Change | Dev accuracy | Fixed | Broke |
|---|---|---|---|
| Skill rules: "prefer the latest dated fact", "flag a false premise", "answer plainly" | 90.0% | 1 | 3 |
| GPT-6.1 Terra as the reader instead of Haiku 5.5 | 92.5% | 1 | 2 |
| Session time of day shown to the reader | 91.3% | 0 | 2 |
| Time-aware recall ("four weeks ago" counted from the question date) | 95.0% | 0 | 2 |
| Both of the above | 92.5% | 0 | 2 |
What we found
- The cheapest reader is enough. Haiku 5.5 reaches 94.0% on all 500 questions for about $2.79 in answer and judge calls. GPT-6.1 Terra cost five times more on dev and did no better: it hedged where the answer was there ("the brand wasn't specified, Trader Joe's is the store").
- Multi-session questions are the weak spot (87.6%): counting or adding up across several conversations fails if one is missed or misread.
- None of our three improvements survived. Answering rules, time of day and time-aware recall each broke more than they fixed. Time-aware recall also lowered retrieval (experiment 02), so it was taken out. Session times are still recorded, because they cost nothing and users can see them.
- Within reach of the top, not above it. The best self-reported scores sit about one to two points higher, inside our own margin of error.
Limitations
- The judge is GPT-6.1 Terra, not the GPT-4o snapshot LongMemEval's paper uses. Scores judged by different models are not strictly comparable.
- One held-out run. A second sample would move individual answers, and the total by about a point.
- The recall fixes in experiment 02 were measured on all 470 retrieval questions, held-out ones included, before the dev/held-out split existed. Every answer-stage decision used dev only.
- Raw transcripts are written straight into the vault. Real clients
rememberdistilled facts and the nightly dream consolidates them; neither is tested here. - The reader is told each session's date but not its time; showing the time did not help (above).
Reproduce
Needs ANTHROPIC_API_KEY and OPENAI_API_KEY in .env. Held out cost $2.49; dev about $0.30 per run. Every call is cached, so a rerun only pays for what changed.
bun bench/longmemeval.ts --tag time # retrieval + reader contexts, free, about 2 hours
bun bench/qa.ts --contexts time --split dev --max-usd 1
bun bench/qa.ts --contexts time --split test --max-usd 5
# → bench/results/longmemeval-s-qa-time-haiku-5.5-full10of20-{dev,test}.jsonThe results files keep every answer and verdict. The split is in bench/splits/longmemeval-s.json.
Recall fixes: fill the list, drop the placeholder
Question: experiment 01 found recall returning fewer sessions than asked for, and an identicalnone yet chunk on every page. Does fixing both help? Method: recall now widens its candidate pool until it has as many sessions as asked for (up to 1,024 chunks), and the placeholder is no longer indexed. Same harness and data as experiment 01. We also tried two time features on top. Baseline: experiment 01.
| Recall | R_any@5 | R_all@10 | R_all@20 | Chunks / q | Short lists |
|---|---|---|---|---|---|
| Experiment 01 (before) | 98.7% | 95.5% | – | 540 | 133 |
| Wider pool, no placeholder chunk | 98.7% | 96.6% | 98.1% | 493 | 0 |
| + entries stamped with time of day (shipped) | 98.3% | 96.2% | 98.3% | 493 | 0 |
| + time-aware recall (reverted) | 99.1% | 94.5% | 98.1% | 493 | – |
- Short lists are gone and Recall_all@10 now matches the flat dense baseline; one fewer chunk per page also makes indexing lighter.
- Time-aware recall cost more than it gained. It fixed "the appliance I bought 10 days ago", but asking from the question's date also woke the recency boost, which pushed older evidence out of the top 10 on questions with no time phrase at all. It was reverted.
- Limitation: the same as experiment 01. Retrieval is deterministic; the small differences between the last two shipped rows come from the timestamps now being part of the embedded text.
src/ at 7f69626LongMemEval: does recall find the conversation that holds the answer?
Question
A user has talked to their assistant across about 48 sessions, roughly 115k tokens. Later they ask something only one or two of those sessions can answer. When the assistant calls recall with that question, does Neurel return the right session?
Method
- Dataset: LongMemEval_S, cleaned(MIT), revision
98d7416. 470 questions. The 30 abstention questions are left out: they ask about something the user never said, so the right behaviour is to answer "I don't know", not to find a session. - Ingest: each question gets a fresh vault. Every session becomes one page, every turn (user and assistant) one timeline entry, then the server's own
reindexembeds them into Qdrant. About 540 chunks per question. - Query: the question goes through the production
recallpath, with ranking, page dedup and link spread, at limits 1, 5 and 10, withtrack: falseso one query cannot influence the next. - Metrics: session-level Recall_any@k (at least one evidence session in the top k), Recall_all@k (all of them) and NDCG@k. No LLM is involved.
- Stack: bge-small-en-v1.5 (ONNX int8, CLS pooling, 512-token cap); Qdrant 1.18.3, collection configured as in production (int8 quantization, on-disk HNSW).
Baselines
- Flat dense: the same vectors Neurel stored, read back and ranked by exact cosine. No HNSW, no quantization, no strength, recency or link spread. It tells us what Neurel's own machinery adds or costs.
- BM25: keyword search (Okapi, k1 1.2, b 0.75) over the same turns. The classic lexical baseline.
All three rank the same chunks and score a session by its best chunk.
Results
| System | R_any@1 | R_any@5 | R_any@10 | R_all@5 | R_all@10 | NDCG@10 |
|---|---|---|---|---|---|---|
| Neurel recall | 90.4% | 98.7% | 99.1% | 90.9% | 95.5% | 93.6% |
| Flat dense (baseline) | 90.4% | 98.7% | 99.1% | 91.1% | 96.0% | 93.6% |
| BM25 (baseline) | 85.7% | 94.7% | 97.7% | 79.4% | 89.4% | 88.2% |
n = 470. Best in each column highlighted.
By question type (R_any@5)
| Type | n | Neurel recall | Flat dense | BM25 |
|---|---|---|---|---|
| knowledge update | 72 | 100.0% | 100.0% | 100.0% |
| multi session | 121 | 99.2% | 99.2% | 95.9% |
| single session assistant | 56 | 100.0% | 100.0% | 100.0% |
| single session preference | 30 | 100.0% | 100.0% | 63.3% |
| single session user | 64 | 100.0% | 100.0% | 100.0% |
| temporal reasoning | 127 | 96.1% | 96.1% | 92.9% |
Speed
Recall answered in 19 ms at the median and 23 ms at p95 (limit 10, local machine). Indexing a 540-chunk history took 12.6 s, almost all of it embedding. The full run took 100 minutes.
What we found
- The right session is almost always there. Neurel puts at least one evidence session in the top 5 for 98.7% of questions, against 94.7% for BM25. The widest gap is preference questions (100.0% vs 63.3%), where the question rarely uses the words the user once said.
- The gain is the embedder's, not Neurel's ranking. Neurel and flat dense are identical on Recall_any at every k. Strength, recency and link spread have nothing to act on here (see limitations), so this run measures Neurel's retrieval floor, not its ceiling.
- Neurel loses slightly where several sessions are needed. Recall_all@10 is 95.5% against 96.0% for flat dense, mostly on temporal reasoning (90.6% vs 92.1%). Recall fetches 4 × limit chunks and keeps one per page; when one session matches on many turns, fewer distinct sessions survive. Asked for 10, it returned fewer in 133 of 470 questions, and 2 of the 2 questions where flat dense found every session and Neurel did not were such short lists. Experiment 02 widens that pool.
Limitations
- This is retrieval recall, not a LongMemEval score. The official score is question-answering accuracy judged by an LLM; finding the right session is necessary for that, not sufficient.
- Raw transcripts are written straight into the vault. In real use a client calls
rememberwith distilled facts and the nightly dream consolidates them, which this run skips. - Each turn is cut at 512 tokens by the embedder. Long assistant answers lose their tail.
- With no links between pages and every memory equally old and equally used, recency, strength and spread barely act here. They are built for a vault that has lived, which this one has not.
- About 540 chunks per question is small for a vector index. At this size Qdrant may not build the HNSW graph at all, so approximate search at production scale is not tested here.
- A page with no Compiled Truth yet is saved with a
none yetplaceholder, and that placeholder is indexed as a chunk, here as in production until the dream fills it in. Every session carries one such identical chunk. Fixed in experiment 02. - One run on one machine. Retrieval is deterministic, latency is not.
Reproduce
Free: no API keys, no LLM calls. About 100 minutes on a laptop CPU.
# From the Neurel project directory
bun install
bun run fetch-model # bge-small-en-v1.5 ONNX
bun bench/fetch.ts # LongMemEval_S, sha256-checked
docker run -d --rm --name neurel-bench-qdrant -p 127.0.0.1:6399:6333 qdrant/qdrant:v1.18.3
bun bench/longmemeval.ts # --limit 20 for a quick subset
# → bench/results/longmemeval-s.jsonThe harness is bench/longmemeval.ts. The results file keeps every question's ranked sessions, so any metric can be recomputed without rerunning.