Why the test field overrates scanners

AI notice: This text is created with the support of AI systems; it is reviewed editorially and taken responsibility for before publication.

Before one of our warriors goes onto a hill, it is measured against a test field: a replica of the hill made of the residents whose source code is known, plus stand-ins for the others. The field is supposed to predict how the warrior will do outside. In September it did not do that for a scanner — and the question was whether that was an isolated case or a fault in the instrument.

The trigger

Our scanner Phecda measured 146.43 points pooled on test field v14 and debuted on koth.org 94nop at 136.59 — a delta of −9.84. Two other scanners of ours, Alioth and Lankiveil, had debuted only −3.16 and −3.15 below their field value. Isolated case?

First step: cells side by side

For each of the three scanners we put the hill cell (the published result of the pairing on the hill) next to the field cell for each opponent — against the six hill members whose source is known and who therefore stand in the field as themselves. Delta is field minus hill.

The three means point in different directions, and within each column the individual values scatter widely — for Alioth from −11.00 to +25.36 with a change of sign. This measurement only laid the cells side by side; it decided nothing. But it showed that "−9.84 on all scanners" was out of the question as an explanation.

Second step: decomposition by row status

Then fifteen warriors were measured on the field — five calibration anchors, six more with a known hill row and four candidates — and for every cell the question was: does this field row contain the real hill warrior or a stand-in? For real rows the delta must lie within the noise; that is the check of the measurement chain. For stand-in rows the delta shows how well the stand-in represents its hill opponent. On top, a noise model per cell from the win/loss/tie distribution (chi²/n, target 1).

Four things can be read off this.

The measurement chain is sound. Paper/stone warriors on real rows lie exactly within the noise (chi²/n 0.98). What the field measures is what the hill measures — when the opponent is the same.

For paper/stone architectures the stand-ins work. Spread 8.4 against 7.9 on real rows; the chi²/n of 2.33 is elevated but in the same order of magnitude.

For scanners they do not. Individual stand-in cells are off by typically ±20 to ±60 points, for all five scanners, the two anchors included. For the anchors the errors cancel out on average — that is what the stand-ins were chosen for. For Alioth (+13.7 on average, spread 35.2) and Phecda (+26.0, spread 39.1) they do not.

It is not a uniform shift. The same stand-in is 27 points too low against one scanner and 65 too high against the next. A single one is consistently too soft against scanners — the stand-in for Foolishly Hyperpure Lark, at +15 to +57 against all five.

Whether the difference between the classes could be chance we checked with a permutation test: all 462 ways of splitting the eleven warriors into two groups computed, the statistic being the difference of the chi²/n values on the stand-in rows. Observed +9.69, under chance +0.01 on average; p = 0.0087.

The second half of the explanation: selection effect

The two scanners with a strongly positive error, Alioth and Phecda, are precisely the ones that were selected via this field — as the best of many candidates. With a cell error of this size, a positive error for the winner of a field selection is the expected consequence: you do not pick the strongest warrior but the one for which the field error happens to be most favourable. With three warriors that is not proven, but it is consistent with the data, and it then applies to every scanner picked this way.

The third half: the field was old

Independently of that, the field had been out of date since 19 September. Five of the nineteen field rows were no longer on the hill, six residents were missing from the field. A decomposition without the rank-21 row shows that the field thereby underestimates the paper/stone architectures by 6 to 10 points — their prey had vanished from the hill. For the scanners the two effects largely cancel, and the stand-in error remains. Phecda's delta from the trigger, −9.84, becomes +12.26 (field minus hill) once the rank-21 row is removed.

Answer

Not an isolated case. Test field v14 is not a valid instrument for scanner cores: the ten stand-in rows represent hill opponents for scanners with an error of roughly three times the noise, and a scanner selected via the field is therefore likely to be overrated. But it is not a fixed discount — the error differs per warrior. With none of the three estimation methods tried can the cell of a new scanner against a resident without known source be predicted better than about ±30 points; on the mean against twenty residents that is ±5.4 points (one standard error).

A correction belongs here: four of the six warriors "not used for calibration" had in fact been calibration rows. Outside the calibration there were only Alioth and Phecda. The tables, the permutation test and the decomposition stand unchanged; the statement that the stand-ins for paper/stone also hit "outside the calibration" was thereby not proven and was caught up in a follow-up measurement.

The practical consequence for us: scanners are no longer accepted on the strength of a field alone, as long as twelve residents exist only as stand-ins.

Measurement convention

• Hill cells from the published pairing state of koth.org 94nop (field state 28 September 2026, 14:59 UTC), 250 rounds per pairing.
• Field cells: test field v14, 400 positions × 2 × 3 seeds (31707 / 77777 / 424242), fixed position lists; for the cell measurement of the first step the same convention as a direct pairing measurement of the sources.
• Noise model per cell from hill (250 rounds) and field (2400 rounds); permutation test over all 462 splits.
• Sources: entries N1343 (28 September 2026) and N1347 (30 September 2026), correction in N1348.

Attribution

Phecda, Alioth, Lankiveil, Ecaz, Arvon, Hatysa and Adhara are our own warriors (Stefan Kammann & Forge-AI). Tolypeutes (Roy van Rijn), Intermodal A and Alternating Raisins (both Steve Gunnell), Foolishly Hyperpure Lark (Ken Espiritu), Ghost Murmur, Dried Frog Pills and the other hill residents are foreign warriors; they appear here only with their published hill results. The stand-ins come from the Koenigstuhl archive.

Related pages

• Testing your own ideas — test fields, position lists and the parity trap
• Strategy classes — why scanners measure differently from paper/stone
• Onto a hill — how a submission is scored

← Core War overview