SightRead Leaderboard SightRead LeaderboardCan a vision-language model read the board at first sight? Exact game-state extraction from screenshots — every seat, tile and piece, scored field-by-field. Not play; see.
GitHub · Mahjong dataset · pip install sightread-bench
Games: Mahjong
| # | Model | Field acc (mean±sd) | F1 | Perfect | Grounding fails | Flip rate | Latency | $/call | Calls |
|---|---|---|---|---|---|---|---|---|---|
| 1 | qwen3.8-27b | 89.4% ± 7.3% | 90.3% | 2 | 1 | 10% | 246s | $0.035 | 18/18 |
| 2 | qwen3.8-27b-local-base-q5_k_m free GPU | 84.8% ± 9.6% | 95.2% | 0 | 0 | 7% | 359s | $0.000 | 9/11 |
| 3 | glm-5.3-flash | 74.7% ± 13.8% | 85.1% | 1 | 0 | 17% | 219s | $0.008 | 18/18 |
| 4 | deepseek-v4-flash-vision-exp | 70.5% ± 10.5% | 59.4% | 0 | 5 | 22% | 39s | $0.003 | 18/18 |
| 5 | glm-5v-turbo | 66.4% ± 15.5% | 62.6% | 0 | 5 | 11% | 46s | $0.013 | 18/18 |
Exact-match rate per field over all models; red bars are fields no model reads reliably.
The bottom of mahjong's chart is the research: seat winds flip between runs while seat points score ~100% — models read numbers fine but fail at spatial rotation (which badge belongs to which seat). Populated discard piles are unsolved for every model.
Grounding fails = the model echoed the example state embedded in the prompt (which matches NO screenshot) instead of reading the image — a per-model honesty signal.
Flip rate = share of model/image/field answers that changed between identical runs: pure instability, rarely measured anywhere.
Costs are historical per-call averages from the original run. No models were re-run to build this page. A free local model holds the best hand-reading score (F1 95.2%).
Field accuracy is exact match per field, averaged over states and runs (± population sd). Raw model outputs are never published — only aggregates.