18.09.26
The boundary between what a model has learned to do in its weights and what it must still search over at runtime.
Hypothesis
The optimal train/test-time allocation is a property of the domain, and can be predicted from,
Combinatorial search complexity: What the trained policy can’t already resolve.
Verification advantage: How cheaply candidates can be evaluated.
Most domains can be formulated as a ‘game’.
Harnesses can change that split. They are not just tool integrations. They can make a domain externally searchable by exposing state, actions, and feedback.
I arrived at this through two domains I work on: chip design and debate.
Chip design has an enormous combinatorial structure but strong verifiers: routing, timing, congestion, etc. At NVIDIA, the issue was that there was no ontology to build this search structure. Once I built that, I was able to let my agent rip overnight and produce usable floorplans for my team.
Debate has a similarly large search space but much weaker verification: for open-ended research questions, truth is often delayed, expensive, or contested rather than immediately checkable. With Crux, I’m externalizing the search over competing arguments to identify the smallest disagreement that changes the conclusion, and collecting the human judgments that could eventually teach models better research taste.
We need to get to letting it rip. And this is the amortization frontier I’m interested in.
Domains placed by how much search the trained policy can’t absorb against how cheaply a candidate can be judged. Verification here means as your harness realizes it. Color is the architecture the two terms predict.
Hover or tap any dot for the detail. Positions are my judgments, not measurements. The point is the geometry, not the coordinates. The orange arrows are the claim: a harness moves a domain sideways. Chip design already made that move. Debate is the one I’m attempting.
| Domain | Residual search | Verification advantage | Feedback | Expected architecture |
|---|
External search is not just more inference-time compute. It turns invisible reasoning into observable decisions: which branch to explore, what evidence to seek, when to abandon a hypothesis, when to stop. Internal reasoning leaves you a final answer. External search leaves state-action-outcome traces, and if humans can evaluate those decisions, the traces can train the next search policy.
A harness cannot manufacture ground truth, but it can break an ungradable final judgment into locally testable ones.
We couldn’t externalize search into the LLM because ordinary language completion is mostly amortized into the model. External search adds little because plausible continuations are easy to generate but difficult to verify against a cheap objective. Search becomes useful only when language generation is embedded in a verifiable task, such as coding, retrieval, or factual question answering.
Things I’m thinking about
The recent release of Jev has confirmed my suspicions.
Jev doesn’t generate text. You give it state and a bounded set of typed questions, and it returns decisions with calibrated probabilities. It was trained with reinforcement learning for calibrated decisions, which optimizes for honest probabilities rather than for outputs a human would rate highly. The demos are an agent playing Doom in real time and one navigating Wikipedia across thousands of link choices per step.
It makes the decision problem explicit, which is the interface search needs.
Persimmon is closer to the weak-verification regime. There is no single correct human response, so Humans& trains against an adaptive discriminator and evaluates whether behavior matches the distribution of real human interactions over time.
The analogous question for Crux is: what evaluator could distinguish genuine research taste?
The closest, most popular analogue is AlphaGo: training compresses search into a policy network but at inference time, the system still performs MCTS because the search space is so large.
I work on AI for GPU floorplans, alongside building Crux. Both projects make me ask whether the bottleneck is the model or the environment we give it.
The Bitter Lesson makes me wary of hand-built ontologies. Are we encoding an expert’s preferred answers, or providing a representation in which search and learning can discover better ones? That is the tension I want to investigate.
The views and opinions expressed in this article are my own and do not represent those of my employer.