41 Experiments in 6 Days: What the Human Was Actually For
TL;DR: We took long-term conversational memory from F1 0.392 to 0.666 on the LoCoMo benchmark in six days, across 41 experiments and about 1,400 lines of TypeScript. Raw F1 landed at 0.638, against a previous published best of 0.598. The number is not the interesting part. Claude wrote nearly all of the code and ran every experiment. The four interventions that decided the outcome were all a human asking whether we were measuring the right thing.
Six days, 41 experiments, 17 adopted and 24 reverted, 108 eval runs, 85 commits. We never designed the memory system upfront. We built a harness that could run an experiment in five minutes, pointed Claude at it, and let the loop find the answer.
That worked. It also made something clear that we did not expect going in: writing code was never the constraint. The agent produced hypotheses and implementations faster than we could...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
