Memory Ep. 10: taking the brain to a stranger's benchmark

Every episode so far measured hipocampo on my own vault. My notes, my questions, my ground truth. Comfortable, and a little suspicious, because the examiner and the student were the same person. So I took it somewhere it had never seen: LoCoMo, a standard benchmark for long-term conversational memory. Ten long conversations between strangers, hundreds of annotated questions, public baselines. Neutral ground.

The first recall number was bad. And chasing why was, as usual, the whole point.

The answerer was not the problem

My first instinct was to blame the part that writes the answer. It was the opposite. When the evidence did not reach the answerer, it did the honest thing and said it did not know. The failure was upstream, in retrieval. The right turn was in the index, but buried: rank 8, rank 36, rank 85, sometimes past 150 out of a few hundred. The system was looking, it just was not looking in the right place.

So I looked at what it was searching for. Hipocampo plans its search before it runs it: a small step turns the question into search cues. And the cues were the problem.

Ask “What did Caroline research?” and the planner produced ["Caroline", "research"]. Perfectly literal. But the turn that holds the answer says “I looked into adoption agencies last week”. Not one shared word with “research”. The embedding puts them far apart, so the evidence sits halfway down the ranking and never makes the cut. Worse case: “What is Caroline’s identity?” produced just ["Caroline"], because the planner is tuned to fetch a person’s note, and in a corpus that is entirely about Caroline that cue discriminates nothing. The turn that answers it says “I came out as trans last year”. The word “identity” appears nowhere. No amount of brute force finds that.

The lever is the seed, not the engine

The fix is not a bigger engine. It is a better seed. Have the planner expand the cue into the words the turn would actually use, the casual, indirect phrasing of real conversation: research becomes {studies, looked into, investigated, explored, read about}; identity becomes {trans, gender, coming out}.

I measured it the only way I trust now: change nothing but the seed, same retrieval, same k. Evidence recall went from 0.43 to 0.69. Twenty-five points, no reranker, no graph, no hybrid index. Just asking the right question. The categories that need the most inference, multi-hop and single-hop, moved the most.

That is the headline of the whole experiment. In conversational memory the lever is not the retrieval machinery everyone obsesses over. It is understanding what the question is really asking and seeding the search with the vocabulary the answer would use.

Two things I got wrong, on the record

I was sure HyDE would win. Instead of keywords, you have the model hallucinate the turn that would answer the question and search with that, imitating the shape of the corpus. Elegant, and my favourite going in. Measured against plain keyword expansion, it lost: +16 points versus +25. One invented sentence is a single bet on phrasing; a fistful of keywords casts a wider net and something always overlaps. I also tried fusing both with RRF. It did not beat keywords alone, because fusing a weaker signal in just drags the good one down.

And a confession. My first expansion prompt had two of the test questions sitting in it as examples, the research and identity cases I just used to sell the idea. A leak. The two cleanest demos were the two I had fed the model the answer to. I caught it, rewrote the prompt with examples from a different domain, and re-ran. The headline dropped from +28 to +25. The mechanism held, but I do not get to use those two cases as proof, and I would rather tell you that than have you find it.

Next

So the lever in conversational retrieval turned out to be comprehension, not horsepower. The graph I killed in Episode 5, the reranker, the hybrid index: all downstream of a question the system did not understand in the first place.

But recall is not an answer. Getting the right turn into the pool is necessary, not sufficient. The next episode closes the loop, measures the actual answers, and races hipocampo against an autonomous agent I set loose on the same benchmark with no instructions at all. We crashed into the same wall.