99 cases3 runs5 reposrun Oct 12 · 16:38 UTC
How often the composer gets it right
99 cases: 60 golden, 20 paraphrases, 10 off-template, 3 cannot-answer and 6 follow-ups. Each case ran three times per model, through the composer itself.
Template accuracy100%no spread · 99 cases
Bindings pass rate100%no spread · 55 cases
Median time to done6.2sp90 8.3 s · 291 composed answers
Cost per answer$0.0021$0.62 for 291 composed answers
Time to done
90% by 8.3 s · from 291 composed answers · 3 runs
| Time to done | Answers |
|---|---|
| under 2 s | 0 |
| 2 to 4 s | 17 |
| 4 to 6 s | 113 |
| 6 to 8 s | 126 |
| 8 to 10 s | 30 |
| 10 to 15 s | 4 |
| over 15 s | 1 |
Against Sonnet alone
- Cost, 10,000 answersin US dollars21 in production, 138 in sonnet.−85%better
- Median time to donein milliseconds6,155 in production, 4,813 in sonnet.+28%worse
- ComponentsF1 at least 0.85100% pass · 0.99 mean F1 in production, 25% pass · 0.75 mean F1 in sonnet.+75ptsbetter
- Rulesno piece dropped100% in production, 91% in sonnet.+9ptsbetter
from 291 composed answers · 291 by Sonnet3 better · 1 worse
| Measure | sonnet | production | Change |
|---|---|---|---|
| Cost, 10,000 answers | 138 | 21 | −85%, better |
| Median time to done | 4,813 | 6,155 | +28%, worse |
| Components | 25% pass · 0.75 mean F1 | 100% pass · 0.99 mean F1 | +75 pts, better |
| Rules | 91% | 100% | +9 pts, better |
Production
| Check | Passed | Spread | Cases |
|---|---|---|---|
| Templateexpected template | 100% | no spread | 99 |
| ComponentsF1 at least 0.85 · no spread | 100% pass · 0.99 mean F1 | 83 | |
| Hot metricsexact set | 100% | no spread | 10 |
| Schemaevery piece valid | 100% | no spread | 97 |
| Bindingsprops match data | 100% | no spread | 55 |
| Numbersno prose mismatch | 100% | no spread | 97 |
| Rulesno piece dropped | 100% | no spread | 97 |
| Measure | Typical | Tail or total | |
|---|---|---|---|
| Time to donemedian and p90 | 6.2 s | p90 8.3 s | |
| Time to first piecemedian and p90 | 6.0 s | p90 7.7 s | |
| Costper answer and in total | $0.0021 | $0.62 total | |
Sonnet alone
| Check | Passed | Spread | Cases |
|---|---|---|---|
| Templateexpected template | not scored · no routing | ||
| ComponentsF1 at least 0.85 · 24 to 25% | 25% pass · 0.75 mean F1 | 83 | |
| Hot metricsexact set | 30% | no spread | 10 |
| Schemaevery piece valid | 89% | no spread | 97 |
| Bindingsprops match data | not scored · no routing | ||
| Numbersno prose mismatch | 100% | no spread | 97 |
| Rulesno piece dropped | 91% | 91 to 92% | 97 |
| Measure | Typical | Tail or total | |
|---|---|---|---|
| Time to donemedian and p90 | 4.8 s | p90 7.3 s | |
| Time to first piecemedian and p90 | 2.6 s | p90 5.4 s | |
| Costper answer and in total | $0.0138 | $4.01 total | |
Haiku alone
| Check | Passed | Spread | Cases |
|---|---|---|---|
| Templateexpected template | 100% | no spread | 99 |
| ComponentsF1 at least 0.85 · no spread | 100% pass · 0.99 mean F1 | 83 | |
| Hot metricsexact set | 100% | no spread | 10 |
| Schemaevery piece valid | 99% | 98 to 100% | 97 |
| Bindingsprops match data | 100% | no spread | 55 |
| Numbersno prose mismatch | 100% | no spread | 97 |
| Rulesno piece dropped | 100% | 99 to 100% | 97 |
| Measure | Typical | Tail or total | |
|---|---|---|---|
| Time to donemedian and p90 | 6.2 s | p90 8.6 s | |
| Time to first piecemedian and p90 | 6.2 s | p90 8.2 s | |
| Costper answer and in total | $0.0007 | $0.20 total | |
Haiku-only matches production on template, bindings and numbers at about a third of the cost; its misses are off-template copy-length and layout rules; Sonnet earns its cost only on off-template rule compliance.
The Sonnet run composes every question off-template on purpose, so it skips routing, and template and bindings are not scored for it.
Cases that failed a check
Every case passed every check