Memory Ep. 11: I raced an autonomous agent, and we hit the same wall

Last episode ended on a cliffhanger: recall is not an answer. Getting the right turn into the pool is necessary, not sufficient. So I closed the loop, measured the actual answers an LLM judge marks right or wrong, and at the same time ran a strange little experiment alongside it.

I gave an autonomous agent one instruction: build a retrieval system that scores 90% on this benchmark. No method, no guidance, no me. Then I let it run for days while I worked the same problem the slow way, one lever at a time. The point was not to win. It was to see whether two opposite ways of working would reach the same place.

Two opposite philosophies

The agent built a kitchen sink. Dense search plus BM25 plus an entity graph plus query decomposition plus intent routing plus evidence compression, all fused with RRF, described in its own docs as “a unique combination of SOTA techniques”. It went from 29% to 82% by piling things on.

I did the opposite. Add one lever, measure it alone, keep it only if it pays, kill it if it does not. Build, measure, kill. That is how the graph died back in Episode 5, and it is how three of the four things I tried here died too.

My results, on one conversation, answer accuracy, non-adversarial:

• Seed expansion (last episode’s keyword trick): +13 points. The one big lever.
• BM25 hybrid search: nothing. In natural chat, lexical matching adds nothing over semantic. Killed, like the graph.
• A reasoning answerer (concise, “connect the pieces, draw a conclusion”): a trade-off, more on that below.
• More context (feeding the answerer more retrieved turns): real, but it saturates fast.

That last one is worth a number. Going from 20 to 50 turns of context bought +9 points. Going from 50 to 100 bought zero more, at double the tokens. There is a ceiling on useful context, and past it you are just burning money. The minimisation thesis hipocampo was built on, proven the hard way on someone else’s data.

The wall we both hit

Here is the agreement, and it is the real result.

The agent’s single biggest jump did not come from any of its retrieval machinery. It came from rewriting the answerer’s prompt, telling it to connect pieces and draw a logical conclusion instead of just extracting. It already had the evidence in context; it just was not reasoning over it.

I found the same wall from the other side. When I broke the score down by category, one pattern was undeniable: every category has a different bottleneck. Multi-hop questions are starved for context, so more retrieved turns help. But open-domain questions, the “would Caroline likely enjoy Vivaldi?” kind that ask you to infer something nobody said, do not move with more context at all. They only move when the answerer is allowed to reason. That is not a retrieval problem. It is a generation problem.

Two methods, opposite in every way, both ended at the same sentence: in conversational memory, the bottleneck is not retrieval, it is the answerer. The agent discovered it by accident, tuning a prompt. I discovered it by measurement, slicing by category. The fact that we agree, having shared no method, is worth more than either number alone.

What each approach cost us

The method was not free, on either side.

The kitchen sink got the agent to a number fast, but it can never tell you which of its six components earned it. Half could be dead weight, like my graph was, and it would never know.

My discipline gave me the clean lever, the one thing that actually moved the needle, the seed expansion the agent never found (it tried reformulating questions into statements, a different thing, and it failed, so it filed all query rewriting under “does not work”). But discipline left me narrow. The agent had breadth I lack: BM25, decomposition, compression, things I never got to. The honest reading is that they are complementary.

The coda that almost fooled me

Everything above was measured on a lab version of my system. To vary one lever at a time, I had stripped hipocampo down to a straight line: cues, search, answer. But that is not the real thing. The real hipocampo has all the machinery I spent months building. It classifies what kind of question you asked. It chains a second search when you ask about a relationship. It filters by note type and by date. I had never run that against the benchmark. So I finally did.

It came dead last. Worse than the plain baseline by seventeen points. My own orchestration, the “smart” part, appeared to be actively hurting the system. It was a perfect headline: I spent months building machinery that makes my memory worse.

I almost wrote that post. Then it stopped adding up. If the real pipeline were that bad, it would not work on the corpus I actually built it for, my own vault, where it plainly does. So I stopped writing and looked. It was not the orchestration. It was one number. The real system was handing the answerer four to eight snippets where the lab version handed it twenty. Its rule for how much to retrieve is tuned for dense notes, the kind where a single rich note answers the question. On shredded chat turns, that same rule is starvation, not stupidity.

I equalised the budget. The two tied. I added the winning levers back on top. The real system climbed right up to match the best lab config.

That is the second time in this project a rigged comparison handed me the most spectacular, most false result. The first was a test question that had leaked into a prompt, inflating a number I was proud of. This was an uneven k, sinking one I was afraid of. The fix both times was not technical. It was distrusting the number that confirmed, a little too neatly, exactly what I already believed.

So the orchestration does not hurt. On this corpus it simply neither helps nor hurts: its weapons aim at structure, note types, dates, named referents, that casual conversation does not have. Which is the most honest lesson of the whole thing. What I built works for my corpus, and a stranger’s benchmark measures neither for it nor against it. It measures the trunk, not the crown.

The asterisks

I will not pretend the numbers are clean. The judge is the same model that wrote the answers, which inflates everything, so I trust the gap between configurations far more than any absolute. It is one conversation out of ten. And the run-to-run noise is around five points, so anything smaller than that I treat as a tie, not a result.

But the shape holds: comprehension over horsepower in retrieval, reasoning over recall in the answer, a hard ceiling on context, and orchestration that is neither villain nor hero on a corpus it was not built for. Three findings and two near-misses, and honestly the near-misses taught me more. The fastest route to a confident wrong answer is a comparison you forgot to make fair.