The Centaur Advantage

A human can know something the model does not and still fail to improve the result. Somebody has to recognise when that knowledge matters. If the system asks the wrong person, defers too late, or follows the wrong recommendation, the possible advantage stays possible.

That is the useful part of the centaur idea for me: treat the decision about who acts as an engineering problem. The name comes from human-computer chess teams. It is an appealing image, but the image can hide an extra component. Alongside the human and the AI sits a router. It can be wrong too.

A chess result worth reading carefully

In Modeling the Centaur, David Shoresh and Yonatan Loewenstein study this selection problem using two chess models. Maia is trained on human games to produce human-like play. Leela learns through self-play. When the models disagree about a move, a manager selects one recommendation.

There were no live human teammates in this experiment. The researchers explicitly state that limitation. Maia is useful for modelling a kind of difference between players; it is not evidence that a person under time pressure, with a changing workload, will behave the same way.

The evaluation used 500 held-out opening positions, played from both sides for 1,000 games. The score is wins plus half of draws, divided by games played: a win-draw-loss score, or WDL. It is not a percentage of decisions that were correct, an Elo rating, or a measure of worker productivity.

In the approximately equal-strength pairing, the learned manager achieved a higher WDL than either model alone. But increasing the strength of Leela changed the result. The learned manager and the expert manager did not produce synergy in the more asymmetric teams. The existence of a weaker partner with occasional useful moves was not enough.

The approximate oracle still found considerable room for improvement. It could simulate continuations using access to the opponent's policy, access that would be inadmissible during normal play. This tells us that exploitable differences existed under that evaluation. It does not tell us that an ordinary router could find them.

The paper also trained its managers for particular players and opponents. Transfer to unfamiliar teammates was left open. A router tested with one model version and one reviewer population needs reassessment when either changes.

Confidence is not relative advantage

Suppose an AI says it is confident. Even a well-calibrated estimate of its own accuracy does not tell us how a particular human would do on the same case. Both may find the case difficult. Or the person may have access to a fact missing from the model's input. The relevant comparison includes that difference, not just the model's score.

And a confident sentence is not automatically a calibrated probability. A proposed routing rule needs evidence from independently checked cases. A numerical cutoff copied from another workflow would look precise while telling us very little.

Here is an independent hypothetical example. A museum wants help drafting catalogue descriptions. The assistant can extract dates and materials from supplied records. A curator can inspect an object and consult provenance documents that are not in the input. Those are different information sources, not a blanket claim that people are better at judgment.

Try the routing decision: two catalogue entries

Entry A: the task is to reformat a verified accession date into the museum's display format. A deterministic formatter may be enough. Adding a language model and a curator could create more opportunities for mistakes than the task requires.

Entry B: a donor's attribution conflicts with a conservation report. The assistant can show the conflict and cite the records. It should not silently choose a provenance claim because its generated confidence is high. A qualified curator needs the underlying evidence and an explicit opportunity to defer publication.

These are proposed decisions, not observed museum outcomes. If the curator has no better evidence either, escalation has uncovered an unresolved question, not manufactured an answer.

In that design, the first boundary is whether the action is allowed at all. The second is whether there is enough information to act. Only then does it make sense to compare which permitted route is likely to work better. The chess experiment isolates recommendation selection. A real workflow also has permissions, delay and a cost of being wrong.

Test the router, not just the partners

Start with cases that can be reviewed independently. Compare the current process, a fixed division of work, and the proposed adaptive router. Where an AI-only route is lawful and meaningful, include it. Hold the inputs and outcome definition steady. Time spent gathering evidence and resolving disagreements belongs in the combined workflow's cost.

Record the cases where routing changed the outcome, especially when it made the result worse. Did it send a hard case to an unprepared reviewer? Did a routine exception accumulate in a queue? Did both routes use the same mistaken source? A low escalation rate is compatible with a router that confidently sends every difficult case the wrong way.

Do not claim to know the unchosen route's outcome just because the selected route succeeded. Estimating that counterfactual needs a suitable evaluation design, such as independent assessments on held-out cases, and it may still be limited by what reviewers can observe retrospectively.

My design preference is to start with a simple allocation and make the router earn its complexity. If a fixed rule or deterministic check works as well at lower cost, use it. The chess result gives a reason to investigate routing. It does not justify spending more on routing than on model quality in every system.

The centaur advantage begins with a difference between partners. It becomes an advantage only when the workflow can use that difference reliably.

Source and revision note

Shoresh and Loewenstein, Modeling the Centaur: Human-Machine Synergy in Sequential Decision Making, March 2025 revision, AAMAS 2025. See sections 2.1, 2.5, 2.6, 3.1, 3.2 and 7.2. The chart reproduces four point estimates from the symmetric-team experiment; the museum example is hypothetical. This revision removes unsupported claims of universal team superiority and operational anecdotes that are not public evidence.