15% was three bugs, not a ceiling
In July we put our agent's memory system on LongMemEval for the first time. It scored 15.0%. Published baselines for comparable systems sit around 64–71%, and simply stuffing the whole conversation into a long-context model scores about 60. Fifteen is not a bad score. Fifteen is a broken system.
The number was real. Every conclusion we were tempted to draw from it was wrong. This is what it took to find out, including the four changes we were confident about that made things worse, and the measurement error we nearly published.
The diagnostic that reframed everything
We ran the benchmark twice, changing only the model that read the retrieved memories and wrote the answer. A stronger reader moved the score by 3.3 points. The gap to state of the art was fifty.
When swapping the model barely moves a score that is fifty points adrift, the model was never the bottleneck. Something upstream is not handing it the evidence.
The abstention rate said the same thing from the other side: on the questions it got wrong, the agent was not confabulating. It was correctly saying it did not know. It was not reasoning badly over what it found. It was never being handed the right passage.
Three bugs
All three were structural, all three were verified in code, and none of them threw an error or appeared in a log.
The episodic tier stored no conversational text. The table had no column for it. Consolidation dropped the message body on the floor and kept the metadata. For months the system faithfully recorded that conversations had happened, along with when and how long they ran, and not one word of what was said.
Retrieval was recency-only. Ordered by timestamp, limit N. There was a vector column sitting in the schema and nothing ever wrote to it, so the similarity endpoints were dead code that returned plausible-looking empty results. "What did I decide about X" returned the most recent N memories, not the most relevant.
Entity extraction was a regular expression. Capitalised words became entities. Words like "Pro" and "Remember" became entities. Nothing that depended on the knowledge graph could work, and nothing reported that.
Fixed, re-benchmarked on the same protocol: 83.3%. Not a tuning win — the system had never been measured doing the thing it was designed to do.
Two throttles nobody had looked at
With retrieval working, we found the client asking for eight passages and truncating each one at 300 characters. Raising the request to 25 fixed eight benchmark questions and broke zero — strictly dominant, worth 13 points. The truncation was worse than it looks: 58% of stored passages were longer than 300 characters, median 433. The majority of everything retrieved was being cut mid-sentence before the model saw it.
Both were one-line constants written early, never revisited, and invisible because a truncated memory looks exactly like a short memory.
Four things that made it worse
This is the part worth the reader's time, because we were confident about all four.
| Change | Result |
|---|---|
| Retrieve more passages (25 → 50) | no change |
| Expand each hit to its full session | −6.6 points |
| Decompose the question into per-entity sub-queries | −22 points |
| Add explicit reasoning rules to the prompt | −6.7 points |
Every one of them failed the same way. Once retrieval is adequate, additional context is not free — it is noise competing with signal, and the reader drowns. Session expansion grew the context nearly sixfold and cost accuracy. Query decomposition was the most instructive: splitting a question into focused sub-queries produced sub-queries too narrow to clear the similarity floor, so it retrieved less while appearing to do something cleverer.
Measure retrieval recall before improving retrieval. Ours was already 96.5%. Every "improvement" after that point was solving a problem we no longer had.
With recall at 96.5% and accuracy at 83%, the remaining gap was not retrieval at all. The evidence was present and the reader was mishandling it. That is a different repair.
The one structural fix that worked
A handful of questions resisted every reader, every prompt and every retrieval setting we tried. They shared a shape: who did I meet first, who finished first among three people, how many days ago.
These are not retrieval failures and no value of k fixes them. One query embedding cannot sit near three separate events at once. Ordering is not a similarity operation.
So we stopped trying to retrieve the answer and started indexing it: a small structured timeline extracted at consolidation time, one row per event, as subject, predicate, date. Ordering questions become a sort.
The load-bearing decision is which date you store. It must be when the event happened, not when it was mentioned. "She graduated yesterday", said on the 28th, has to store the 27th. Store the mention date and the index is worse than useless — it is confidently, consistently wrong, and every row looks fine.
On the questions it applies to, the timeline moved accuracy from 70.5% to 81.8%, replicating exactly across runs. Overall it is worth less than a point, because it only fires on about a third of questions. Both numbers are true and the second one is the honest headline.
The measurement error we nearly published
The first timeline run scored the same as baseline. We almost wrote it up as a null result.
The feature only activates on questions that ask about ordering or elapsed time; everything else gets a byte-identical prompt. So we scored those two groups separately, across four runs. The untouched group — identical input every time — scored 34, 34, 31 and 33.
A three-question swing from nothing at all. On one question we sent the exact same prompt five times and got the right answer once. That noise was larger than the effect we were trying to measure, and in the first run it happened to cancel it out precisely.
A single benchmark run is a draw from a distribution, not a measurement. If you cannot state the spread, you do not have a result — you have an anecdote with a decimal point.
We now pair every run and score the affected subset separately. It is the difference between "no improvement" and "+11.4 points where it applies, diluted by coverage." Those are opposite conclusions from the same data.
Two smaller traps, both expensive
A transcript looks like a conversation to continue. Feeding twelve thousand characters of dialogue to an extraction model and asking for JSON, the model ignored the instruction and replied in character — helpfully, fluently, and as a participant. Roughly 70% of sessions silently produced nothing. Three things fixed it: enforced JSON output, explicit delimiters around the transcript, and restating the task after the data rather than only before it.
The cheaper model invented a fact. Comparing two extraction models on the same transcript, one produced a person's name and a date. The name appears nowhere in the source. The model we had assumed was weaker was the one that refused to invent it. We changed models on the strength of that and would not have caught it without a diff.
What we would tell someone starting
- Benchmark early. Ours ran for months on an architecture that could not work, and no test caught it, because every component passed its own contract.
- Diagnose the layer before you fix anything. Swap one variable and see how far the score moves; if it barely moves, you are looking at the wrong layer.
- Measure retrieval recall separately from answer accuracy. They fail differently and the fixes are unrelated.
- More context is not free. Past a threshold it is actively harmful, and it is the most tempting change available.
- Pair your runs. The first single-run number we produced was one draw, and we quoted it.
- Silent degradation is the dominant failure mode. Every bug here returned an empty result that was indistinguishable from a legitimately empty result.
The last one is the theme. Fifteen percent was not a ceiling and not a model limitation. It was three bugs, none of which produced an error, in a system where every component was correct in isolation.