Two weeks ago we published a benchmark: Vyazen Continuum against Claude Code's search agent, on a 200K-line codebase and a 2M-line one. Continuum led on both axes at both sizes.
The correct response to any benchmark a vendor wins is not to read the table. It is to ask how the table was produced. The numbers are only as good as the method behind them, and the method is usually missing.
So here is ours, in full: how the questions were written, how the answers were graded, and every place we could have tilted the result and what stopped us. The last section is the checklist we would run against anyone else's benchmark, including this one.
What We Were Measuring
The benchmark tests one capability: answering questions about a codebase. Where is this defined? Who calls this? What breaks if this changes? Everything an agent does on a real system (planning a migration, reviewing a change, debugging an incident) sits on that substrate. If the substrate is unreliable at scale, nothing built on top of it is reliable either.
Equally important is what it does not measure: end-to-end task execution, editing, latency, cost. Those are different evaluations. A search benchmark won't tell you which agent ships better pull requests. It will tell you whether either agent knows what it is talking about.
The Two Axes
Every answer was scored on two independent axes, because the two failure modes look similar in a transcript and cost different things in production.
| Axis | The question it asks | A failure looks like |
|---|---|---|
| Accuracy | Is the content true? | Names the wrong function as the caller. Claims a table is written in three places; it is written in two. |
| Adherence | Is it the answer to the question asked: grounded in this codebase, within the scope given? | A true, general description of a module when the question asked for the exact call path. An answer about a different service that happens to share a name. |
An answer can be accurate and non-adherent: true statements that never answer the question. It can be adherent and inaccurate: exactly the right shape of answer, attached to the wrong file. Collapsing both into a single "was it good?" score hides which failure you have, and the fix is different for each. So we scored them separately, on every question.
The Codebases
Two production systems, neither built by us, neither synthetic:
- 200K lines: a service in active development, one primary language, in its eighth year.
- 2M lines: a platform of dozens of services, fifteen years of history, three generations of conventions layered on top of each other.
Two sizes, because the claim under test was about the slope, not the point. "Our search is good" is a one-size claim. "Retrieval quality degrades with scale, and degrades differently depending on how the system works" requires at least two sizes run under identical conditions. A benchmark at a single size cannot show a trend, which is precisely why most benchmarks are run at a single size.
Building the Question Set
Questions fell into four categories:
| Category | Example | What it stresses |
|---|---|---|
| Locate | "Where is the retry policy for outbound payments defined?" | Resolving intent to a symbol, past naming drift |
| Trace | "What path does a request take from this handler to the ledger write?" | Transitive relationships, not adjacency |
| Impact | "What breaks if this column's type changes?" | Reverse dependencies, breadth across the repo |
| Cross-cutting | "Which modules write to this table, and do their schemas agree?" | Reasoning that only exists at system scale |
Every question had to pass four rules:
- Verifiable before grading. Ground truth (the specific files and lines) was recorded for every question and checked by a second engineer before either system answered anything. An answer is graded against recorded evidence, not a grader's impression on the day.
- Answerable only from this codebase. Nothing answerable from general knowledge or framework documentation. Every answer lives at a file and a line in the repository under test.
- Written by people who built neither system. The question authors had no stake in which tool won and no visibility into either system's internals.
- Identical construction at both sizes. Same categories, same difficulty mix, same phrasing discipline at 200K and 2M, so the difference between the two runs is the codebase, not the questions.
One hundred questions per codebase, twenty-five per category.
Running the Benchmark
Each system ran in its native mode:
- Vyazen Continuum indexed the repository once; every question was answered against the graph.
- Claude Code's search agent worked from the repository on disk, default settings, one fresh session per question.
No follow-up prompts, no hints, no "look harder." The first answer counts, because the first answer is what a real user gets.
The mode is the fairness question, so it deserves a straight answer. Claude Code rebuilds its understanding of a repository per question, by design. Continuum resolves relationships once, at index time, by design. We ran each the way it is meant to be run. Handicapping either would have produced a cleaner table and a meaningless one.
Grading
- Each answer scored 0 or 1 on each axis; multi-part questions scored per part.
- Graders were engineers who had built neither system, with repository access and the ground-truth notes. Transcripts were stripped of anything identifying their source before grading.
- Disputes went to a third engineer, and the tie-break was the recorded ground truth, not majority opinion.
THE RULE THAT MATTERS
Ground truth was recorded before any system answered. When a grader's opinion and the recorded evidence disagreed, the evidence won. Most benchmark arguments are really arguments about what a question meant, held after the fact. Freezing the evidence first removes that entire class of dispute, including the ones you would have won.
Where We Could Have Cheated
A benchmark is a hundred small decisions, and each one can quietly lean your way. The ones that matter:
| Temptation | What we did |
|---|---|
| Drop questions after seeing the results | None dropped. The set was frozen and versioned before the first run; the results cover every question asked. |
| Tune on the test | Continuum's configuration was frozen before the run and untouched after. |
| Write trick questions | Questions were written to be answerable. A benchmark full of traps measures the question author. |
| Grade our own homework | Graders were blind to the source of every transcript. |
| Hide the failures | The next section is about the failures. |
The honest reason to believe a vendor's benchmark is not that they say they did these things. It is that they can show you the artifacts: the frozen question set, the transcripts, the grading records. We can.
Where Each System Dropped Points
Continuum lost 3 points of adherence and 5 of accuracy between 200K and 2M. The misses were not random: they clustered on questions whose answers lived outside source code (build scripts, generated code, configuration) where the graph's coverage is thinner. That is a real gap, it is known, and it belongs to the indexer, not to a prompt.
Claude Code's 200K numbers (85 adherence, 82 accuracy) are good, and worth saying plainly: at that size, grep-and-read genuinely works. Its misses there were mostly near-identical names: services sharing a prefix, two clients for the same dependency. At 2M the pattern changed shape. More near-matches per query, more files read to tell them apart, and answers delivered with full confidence about the wrong symbol. The fall to 62 accuracy means roughly two in five answers were wrong, and nothing in the tone of the transcript warned you.
The Results, Again
| Vyazen Continuum | Claude Code Search Agent | |
|---|---|---|
| Adherence: 200K | 98 | 85 |
| Adherence: 2M | 95 | 72 |
| Accuracy: 200K | 92 | 82 |
| Accuracy: 2M | 87 | 62 |
Same numbers as the first post, because they are the same run. A methodology post that produced different results than the results post would be its own finding.
What This Benchmark Does Not Tell You
- One codebase per size point. Two points make a line, not a curve; the shape between 200K and 2M is interpolated.
- Question answering, not task execution. An agent can know exactly where everything is and still change the wrong thing; that is a different evaluation.
- A dated snapshot. Claude Code ships weekly and its numbers will move with every release; so will ours. Any benchmark presented without a run date and model versions is marketing, not measurement.
Questions to Ask About Any Benchmark, Including Ours
- Who wrote the questions, and did they build any of the systems being scored?
- Was ground truth recorded before the systems ran?
- Were transcripts graded blind?
- Was any question dropped, or any setting changed, after results were seen?
- At what codebase size, and how far is that from yours?
- Can the vendor show you their own failure cases, with the transcripts?
A vendor who can answer all six runs a benchmark. A vendor who cannot runs a demo.