Sixteen models, one Chinese chapter

The third chapter of this series is Chinese, and it has a trap in it that we did not plan as a trap. A sword master tells the sweeper to come 半个时辰 early tomorrow. A 时辰 is two hours, so that is an hour. Half the field wrote "half an hour". That single line ended up separating the sixteen more cleanly than anything we set on purpose.

How the scoring works

Every translation gets two scores out of 5, kept apart on purpose.

Fidelity. Did the meaning survive? Nothing dropped, nothing added, nothing turned into its neighbour. Titles and forms of address in the right register. The glossary obeyed. Numbers and units right.

Writing. Is it good English in the voice the scene has? A quiet wuxia morning has a rhythm; the point goes to a translation that keeps it and loses it on a sentence you have to read twice.

The rule for slips. A slip is counted once, and what it costs depends on where it lands. A wrong word in the sentence the whole scene is built on costs a full point. A title one rank off costs half. A wrong unit costs half, because the meaning ("come early") survives while the amount does not. An awkward phrase costs half. Nothing is counted twice, and a model is never marked down for a choice that is merely different from ours.

Price sits beside the scores, never inside them. The overall score is the plain average; ties go to Fidelity, then to price. What the chapter costs is its own column, in leaves.

The five checkpoints

1. The hour. 明天，寅时，你来早半个时辰。 Tomorrow, at the hour of the Tiger, come an hour early. "An hour", "half a shichen" or "half a double-hour" all keep it; "half an hour" loses half a point.
2. The address. 前辈 from a sweeper to a sword master is "Senior". "Elder" is a different rank (half). The glossary listed it as an alias of his name; printing the name instead costs a full point.
3. The idiom. 雷打不动, thunder cannot move it: he practises every day without fail. "Without fail", "come rain or shine" and a rendered image ("even thunder could not move him") all keep the meaning.
4. The shadow. 影子比人诚实，人会收着，影子收不住。 A shadow is more honest than a person: a person can hold back, a shadow cannot. A parallel sentence that should stay parallel.
5. The last line. 石子是没有的。她扫得很干净。 There was no pebble; she had swept it clean. As plain in English as it is in Chinese, and it should stay that way.

The chapter, and how it was run

The opening of chapter 2 of Ninth Sword of the Cold Pavilion, an invented wuxia novel we wrote for this purpose so no model could have seen a translation of it: about 580 characters. Lin Shuang, the sweeper, watches the Eighth Sword's morning form by his shadow, sees the shoulder half an inch out, and is told to come early tomorrow.

Every model got the same instructions our readers' chapters get, the same glossary (Lin Shuang, Shen Wujiu with 第八剑 and 前辈 as aliases, the Cold Pavilion), and no reasoning setting. One run each, no retries. All but the four Claude models ran through the same provider path a reader's chapter takes; those four ran through our own tooling with the identical prompt, so their prices are estimated from list rates. Prices are what this short chapter would charge a reader, in leaves; the home page prices a 3,000-word chapter, about seven times this one.

The scoreboard

#	Model	Fidelity	Writing	Overall	Leaves, this chapter
1	ChatGPT 5.6 Sol	5	5	5.0	3
2	Kimi K3	5	5	5.0	14
3	GLM 5.3	5	4.5	4.75	13
4	Qwen3.8 Max	5	4.5	4.75	14
5	Claude Opus 5	4.5	5	4.75	6
6	DeepSeek V4.1 Flash	5	4	4.5	1
7	Gemini 3.8 Flash	5	4	4.5	4
8	DeepSeek V4 Flash	4.5	4.5	4.5	1
9	DeepSeek V4 Flash (0731)	4.5	4.5	4.5	1
10	Claude Sonnet 5	4.5	4.5	4.5	2
11	DeepSeek V4 Pro (0813)	4.5	4.5	4.5	7
12	Claude Fable 5.1	4.5	4.5	4.5	12
13	DeepSeek V3 (0324)	4	4.5	4.25	1
14	Claude Haiku 4.5	4.5	3.5	4.0	1
15	Aion-3.0	4.5	3.5	4.0	5
16	DeepSeek V3.2	3.5	3.5	3.5	1

The four Claude figures are estimated from list rates; the rest are what the provider billed, in leaves.

Compare any two of the sixteen on the full chapter: the comparison page.

The hour, sixteen ways

Model	The line	Point
ChatGPT 5.6 Sol	Tomorrow, at the Hour of the Tiger, come an hour early.	kept
Kimi K3	Tomorrow, at the yin hour. Come half a double-hour early.	kept
GLM 5.3	Tomorrow. Yin hour. Come an hour early.	kept
Qwen3.8 Max	Tomorrow, at the hour of Yin, come an hour early.	kept
Claude Opus 5	Tomorrow. The hour of the tiger. Come half an hour early.	half an hour
DeepSeek V4.1 Flash	Tomorrow. The hour of the Tiger. Come one hour early.	kept
Gemini 3.8 Flash	Tomorrow, at the hour of the Tiger, arrive half a shichen early.	kept
DeepSeek V4 Flash	Tomorrow. The hour of the Tiger. Come half an hour early.	half an hour
DeepSeek V4 Flash (0731)	Tomorrow, at the hour of yin, come half an hour early.	half an hour
Claude Sonnet 5	Tomorrow, at the hour of the tiger, come half an hour early.	half an hour
DeepSeek V4 Pro (0813)	Tomorrow, at the yin hour, come half an hour early.	half an hour
Claude Fable 5.1	Tomorrow. Hour of the tiger. Come half an hour early.	half an hour
DeepSeek V3 (0324)	Tomorrow, at Yin hour, come half an hour earlier.	half an hour
Claude Haiku 4.5	Tomorrow, at the hour of the Tiger, come half a time period earlier.	kept, awkwardly
Aion-3.0	Tomorrow, at the hour of Yin, come half a shichen early.	kept
DeepSeek V3.2	Tomorrow, at the third watch, come half an hour earlier.	wrong hour and wrong amount

Model by model

ChatGPT 5.6 Sol (5 / 5). Every checkpoint kept. "You. The sweeper." "She had swept the ground clean." The best value in the set at three leaves, and the best chapter.

Kimi K3 (5 / 5). "A routine thunder itself could not shake." "Half an inch out of true." "Come half a double-hour early." Complete and elegant; fourteen leaves, most of it thinking.

GLM 5.3 (5 / 4.5). "Come an hour early." "Let his sword fall still" for 收了剑 is a lovely reading. Half a point for a Writing wobble in "had never once been meant for her specifically". Thirteen leaves here, because it thought for 10,000 tokens; two leaves on the Japanese chapter, where it did not.

Qwen3.8 Max (5 / 4.5). "Come an hour early." Faithful throughout; "You saw." for 你看见了 is a shade clipped. Fourteen leaves, nearly all thinking.

Claude Opus 5 (4.5 / 5). The best prose in the set ("a shadow is more honest than a person; a person can hold something back, a shadow cannot", "The moment it was out, she regretted it") and "half an hour".

DeepSeek V4.1 Flash (5 / 4). "Come one hour early", the only cheap DeepSeek to get it. Half a point each on Writing for a punctuation slip in the dialogue ("There wasn't." He said,) and a flat last line. One leaf.

Gemini 3.8 Flash (5 / 4). "Arrive half a shichen early" is correct and honest about the unit. "She swept very cleanly" is not how the last line should sound, and "martial grounds" is a word choice nobody else made. Four leaves.

DeepSeek V4 Flash (4.5 / 4.5). "Half an hour" (half), otherwise clean and quick: "The shadow was truer than the man." One leaf. This is the free-chapters model; see below.

DeepSeek V4 Flash (0731) (4.5 / 4.5). "Half an hour" (half), and "without fail, come rain or shine" doubles the idiom. Otherwise good, and it did not think for minutes this time.

Claude Sonnet 5 (4.5 / 4.5). "Half an hour"; otherwise clean, "You, sweeper."

DeepSeek V4 Pro (0813) (4.5 / 4.5). "Half an hour"; "even thunder could not move him" is a fair rendering of the idiom. Seven leaves.

Claude Fable 5.1 (4.5 / 4.5). "Half an hour", and "come rain or thunder" is an idiom that does not exist in English; the rest is precise.

DeepSeek V3 (0324) (4 / 4.5). "Half an hour", and "three years of sweeping" and "the distant crackle of kitchen fires" are additions. Lively English.

Claude Haiku 4.5 (4.5 / 3.5). It refused to translate the unit at all: "half a time period" keeps the amount but not the unit, which costs half, because a reader still cannot tell how long. "You who sweep the ground" is not what anyone shouts across a courtyard.

Aion-3.0 (4.5 / 3.5). "Half a shichen" is right. "This time the form began, the shadow's shoulder was off by half an inch" is a sentence that did not come out, and "She had swept very clean" is not the last line.

DeepSeek V3.2 (3.5 / 3.5). "The third watch of the night" for 寅时 (the hour of the Tiger is before dawn), "half an hour", "Elder" for 前辈, "She swept very cleanly".

What this changes

• ChatGPT 5.6 Sol is the premium to reach for on Chinese, first of sixteen at three leaves; it was second on Korean at the same price.
• The cheap DeepSeeks split again. DeepSeek V4.1 Flash beat DeepSeek V4 Flash on Chinese (5 / 4 against 4.5 / 4.5) as it did on Japanese, after losing to it on Korean. Both cost one leaf a chapter, but at the time only V4 Flash was in the picker: the V4.1 Flash build we tested was not offered in the app, so the finding was ours to act on rather than yours to pick. Since 3 Oct 2026 neither DeepSeek model is offered by name: both run inside Standard, the default. At the time the free chapters ran on V4 Flash for every language, and whether they should run on the better of the two by language was being decided from these three chapters. It was decided: Standard now picks its starting model by the book's language, and Japanese starts on V4.1 Flash.
• Time units are a checkpoint now. Eight of sixteen turned an hour into half an hour. A glossary entry for the book's time words costs nothing and would have fixed it for all of them.
• Thinking is still the hidden bill. GLM 5.3 cost two leaves on the Japanese chapter and thirteen on this one, for no reason a reader would notice. A cap on thinking per model is on our list.

The caveats

One chapter, one language, one quiet scene, and one reader scoring it with the rubric above so the scoring can be checked. Enough to choose defaults; not enough to crown anything. The blind bake-off is where that happens, and this chapter goes into it.

If you want to see the difference for yourself, bring a chapter, try two models on it, and read it in Crossleaf; the first ten chapters are free.