OCF 2026 AI RESEARCH INTERNSHIP · CASE STUDY

開源 VLM 繁體中文 OCR 與文件解析評測

Benchmarking Open-Source VLMs on Traditional Chinese OCR and Document Parsing

這份報告記錄一個為期約 9 週的 AI 研究實習專案：從兩套獨立評測流程出發，追查同模型為何可能得到不同分數，透過消融實驗驗證影響來源，再把結論落實為可重現、可稽核、可長時間無人值守的 VLM 評測系統。

This report documents a roughly nine-week AI research internship project. Starting from two independently built evaluation pipelines, we investigated why one and the same model could end up with different scores, used an ablation study to verify where those differences came from, and then turned the conclusions into a VLM evaluation system that is reproducible, auditable, and able to run unattended for long stretches.

GitHub · Project Repository ↗ 研究 × 評測工程 × 可重現性 Research × Evaluation Engineering × Reproducibility
9 weeks
2026/07/01–08/30 實習期間
Internship period, 2026/07/01–08/30
18,530
消融實驗推論次數 · API error 0
Ablation inference calls · 0 API errors
5,357
TC_STR 3,706 + OmniDocBench 1,651
TC_STR 3,706 + OmniDocBench 1,651
16 VLM
跨五階段評測或設定的開源模型
Open-source models evaluated or configured across five stages
01 / PROJECT OVERVIEW

這個分數，值得相信嗎？

Is this score worth believing?

本實習屬於 OCF 2026 AI 研究實習計畫的評測路線，目標是建立一套能回答實務問題的評測方法：當 VLM 被用來處理繁體中文場景文字與整頁文件時，我們要如何控制 prompt、推論參數、後處理、評分器與資料版本，讓不同模型的結果可以被公平比較，也能被後續研究者重現與稽核。

This internship sits on the evaluation track of the OCF 2026 AI Research Internship programme. The goal was to build an evaluation methodology that answers a practical question: when VLMs are used on Traditional Chinese scene text and full-page documents, how should we control the prompt, the generation parameters, the post-processing, the scorer and the dataset version so that results from different models can be compared fairly — and later reproduced and audited by other researchers?

FLAGSHIP FINDING
同一模型、同一資料集，只改評測設定，Exact Match 可從 0% 變成 64.84%。
Same model, same dataset — change only the evaluation settings and Exact Match moves from 0% to 64.84%.

這個結果讓專案重心從「累積排行榜分數」轉為「把設定、來源與例外納入評測系統本身」。後續的 run signature、model digest、dataset fingerprint、evaluator commit 與 raw response 保存，都是由這個方法論問題延伸出的工程設計。

This result shifted the project's centre of gravity from "accumulating leaderboard numbers" to "making configuration, provenance and exceptions part of the evaluation system itself." Everything that followed — run signatures, model digests, dataset fingerprints, evaluator commits and retained raw responses — is engineering that grew out of this one methodological problem.

WHY IT MATTERS
政府公報、判決書、歷史報刊等開放資料常以掃描影像存在；本地可執行的開源 VLM 若能被可靠評估，可降低外部 API 成本與資料外流風險。
Open data such as government gazettes, court judgments and historical newspapers often exists only as scanned images. If locally runnable open-source VLMs can be evaluated reliably, they cut both external API costs and the risk of data leaving the organisation.
RESEARCH QUESTION
VLM OCR 的自由文字輸出不像選擇題有唯一評分方式；標點、空白、後處理與額外解釋都可能改變分數，因此必須先界定「怎麼量」。
Free-form text output from VLM OCR has no single obvious way to score it, unlike multiple choice. Punctuation, whitespace, post-processing and unsolicited explanations can all move the number, so "how we measure" has to be defined first.
PROJECT VALUE
最後的主要交付是一套可重跑、可追溯來源、失敗可見、且能交接給後續研究者的 evaluation infrastructure；排行榜只是附帶產物。
The main deliverable is evaluation infrastructure that can be re-run, traced back to its sources, keeps failures visible, and can be handed over to the next researcher. The leaderboard is a by-product.
02 / TEAM & JOURNEY

兩位評測實習生，從兩套獨立流程走向同一套方法論

Two evaluation interns, two independent pipelines, one shared methodology

本報告主要呈現游聿堂與黃以信共同執行的 VLM 評測研究。兩人的起點不同：一方先從「測量如何被建構」追查分數差異，一方從「來源與版本如何被追溯」建立可重現的基礎設施；這兩條工作線最後在消融實驗與正式 benchmark 中合流。

This report mainly presents the VLM evaluation research carried out jointly by Yu Yu-Tang and Huang Yi-Hsin. The two started from different angles: one traced the score gap by asking how the measurement itself was constructed, the other built reproducible infrastructure by asking how provenance and versions can be traced. The two lines of work finally converged in the ablation study and the formal benchmark runs.

EVALUATION METHOD
游聿堂
  • 建立 eval/：最早的 Ollama VLM 評測流程
  • 設計後處理規則與 EM／CM／ANLS／F1 四項指標
  • 製作互動式 HTML 報告與長時間無人值守執行腳本
  • 首先提出「同模型同資料，為何不同流程會得到不同分數？」
  • Built eval/, the earliest Ollama VLM evaluation pipeline
  • Designed the post-processing rules and the four metrics: EM / CM / ANLS / F1
  • Produced the interactive HTML report and the long-running unattended execution scripts
  • First raised the question: same model, same data — why do different pipelines produce different scores?
REPRODUCIBILITY & INFRA
黃以信
  • 建立第二套獨立流程 Sixhuang/，形成可比較的對照
  • 完成 7 組 OFAT 消融實驗，共 18,530 次推論
  • 建立 benchmark_suite/ 與 TC-STR_and_OminDoc/ 長時間評測系統
  • 把 model digest、dataset fingerprint、prompt hash、evaluator commit 納入來源追溯
  • Built a second, independent pipeline Sixhuang/ as a comparable control
  • Completed 7 OFAT ablation variants totalling 18,530 inference calls
  • Built the long-running evaluation systems benchmark_suite/ and TC-STR_and_OminDoc/
  • Brought model digest, dataset fingerprint, prompt hash and evaluator commit into provenance tracking
PROJECT TIMELINE · FIVE STAGES
07/15
提出問題
Raise the question
eval/
建立第一套流程，發現跨流程分數可能不一致。
eval/
First pipeline built; cross-pipeline scores turn out to disagree.
07/16
獨立重做
Rebuild independently
Sixhuang/
第二套流程形成對照，差異成為後續變因。
Sixhuang/
A second pipeline becomes the control; its differences become the variables to test.
07/21
實驗驗證
Verify by experiment
ablation_experiment/
7 組 OFAT，18,530 次推論回答分數差異來源。
ablation_experiment/
7 OFAT variants, 18,530 inference calls, answering where the score gap comes from.
07/28
制度化
Institutionalise
benchmark_suite/
把結論寫進無人值守、append-only、可 resume 的系統。
benchmark_suite/
The conclusions are written into an unattended, append-only, resumable system.
07/30–08/14
擴大驗證
Scale up validation
TC-STR_and_OminDoc/
8 模型 OCR、1,651 頁官方文件 evaluator 與比較報告。
TC-STR_and_OminDoc/
OCR across 8 models, the official evaluator over 1,651 document pages, plus comparison reports.
03 / ABLATION EXPERIMENT

消融實驗：分數落差從何而來

Ablation study: where does the score gap come from?

eval/ 與 Sixhuang/ 兩套獨立流程，同一模型、同一資料卻算出不同分數，讓「模型能力」與「評測流程效果」混在一起，難以判斷分數真正代表什麼。為了拆開兩者，我們把兩套流程對齊成同一個 baseline，再用OFAT 消融實驗（每次只改一項設定）逐一量出各變因的影響。

The two independent pipelines eval/ and Sixhuang/ computed different scores for the same model on the same data, mixing "model capability" together with "evaluation-pipeline effects" and making it hard to say what a score actually meant. To separate the two, we aligned both pipelines onto a single shared baseline and then used an OFAT ablation study (one setting changed per run) to measure each factor's influence in turn.

GLM-OCR Q8_0
單一模型 × TC-STR 3,706 筆測試資料
One model × 3,706 TC-STR test samples
7 組
variant，5 組需要重新推論、2 組離線重算
variants — 5 required fresh inference, 2 were recomputed offline
18,530
次 Ollama request，API error 數為 0
Ollama requests, with 0 API errors
0% – 64.84%
同模型同資料，EM 隨評測設定變動的區間
Range over which EM moves with evaluation settings, same model and data

每組僅相對 baseline 更動一項設定。EM（完全相符）、CM（是否包含正解）、ANLS（編輯距離正規化相似度）、F1（字元級精確與召回的綜合分數）四項指標的完整定義見 08 章。

Each variant changes exactly one setting relative to the baseline. Full definitions of the four metrics — EM (exact match), CM (whether the ground truth is contained in the output), ANLS (normalised similarity based on edit distance) and F1 (the combined character-level precision and recall score) — are given in section 08.

實驗 相對 baseline 的唯一改動 EM↑ CM↑ ANLS↑ F1↑ 平均延遲
VARIANT THE ONE CHANGE VS BASELINE EM↑ CM↑ ANLS↑ F1↑ MEAN LATENCY
Aligned baseline /api/generate、長 prompt、repeat_penalty = 1.6、num_predict = 80 47.06% 76.52% 54.54% 59.17% 0.602 s
Aligned baseline /api/generate, long prompt, repeat_penalty = 1.6, num_predict = 80 47.06% 76.52% 54.54% 59.17% 0.602 s
Endpoint：/api/chat 改用 /api/chat，其餘不變 47.06% 76.52% 54.54% 59.17% 0.582 s
Endpoint: /api/chat Switch to /api/chat, everything else unchanged 47.06% 76.52% 54.54% 59.17% 0.582 s
Sixhuang short prompt 171 字長 prompt 改成 39 字短 prompt 0.00% 79.06% 0.00% 13.56% 0.594 s
Sixhuang short prompt 171-character long prompt replaced by a 39-character short prompt 0.00% 79.06% 0.00% 13.56% 0.594 s
不指定 repeat_penalty = 1.6 回到 Ollama／模型預設值，不再顯式傳入 1.6 64.84% 82.46% 72.08% 76.62% 0.583 s
Do not set repeat_penalty = 1.6 Fall back to the Ollama / model default instead of passing 1.6 explicitly 64.84% 82.46% 72.08% 76.62% 0.583 s
num_predict = 25 輸出長度上限由 80 降至 25 46.03% 76.34% 53.29% 65.07% 0.293 s
num_predict = 25 Output length cap lowered from 80 to 25 46.03% 76.34% 53.29% 65.07% 0.293 s
Sixhuang minimal postprocess baseline raw response 改用最小後處理，離線重算 0.00% 76.79% 0.00% 6.88% 離線重算
Sixhuang minimal postprocess The baseline raw responses re-scored with minimal post-processing, computed offline 0.00% 76.79% 0.00% 6.88% offline recompute
Sixhuang scorer baseline prediction 不變，改用另一套 scorer 計分 47.87% 77.63% 55.02% 59.96% 離線重算
Sixhuang scorer Baseline predictions unchanged, scored with the other scorer instead 47.87% 77.63% 55.02% 59.96% offline recompute
KEY 01
repeat_penalty 是最大單一效益
repeat_penalty is the single largest lever
不指定 repeat_penalty = 1.6、回到預設值後，EM 增加 17.78 個百分點；779 題由錯轉對、120 題由對轉錯，淨增 659 題完全正確。1.6 對這個模型可能過高。
Dropping the explicit repeat_penalty = 1.6 and reverting to the default raised EM by 17.78 percentage points: 779 items flipped from wrong to right and 120 from right to wrong, a net gain of 659 fully correct answers. 1.6 is probably too high for this model.
KEY 02
EM 歸零 ≠ 完全沒辨識到文字
EM at zero ≠ the text was never recognised
短 prompt 讓 EM／ANLS 歸零，但 CM 反而升到 79.06%（全實驗組最高）：答案仍藏在輸出裡，只是後面接了 Markdown、解釋、英文分析與 prompt 複誦，導致嚴格比對失敗。
The short prompt drove EM and ANLS to zero, yet CM rose to 79.06% — the highest of any variant. The answer was still there in the output; it was just followed by Markdown, explanations, English commentary and a recitation of the prompt, which broke strict matching.
KEY 03
後處理決定了分數能不能看
Post-processing decides whether the score is readable at all
aligned 後處理修改了 3,704／3,706 筆 raw response；同一批輸出換成最小後處理，EM 立刻歸零。目前分數衡量的是「模型＋後處理」的系統表現，不是模型原始輸出。
The aligned post-processing modified 3,704 of 3,706 raw responses; feed the same outputs through minimal post-processing and EM drops straight to zero. What the score measures today is the performance of "model plus post-processing" as a system, not the model's raw output.
KEY 04
endpoint 可從嫌疑名單中排除
The endpoint can be struck off the suspect list
/api/generate 與 /api/chat 的 3,706 筆 processed prediction 逐字相同，四項指標無差異，0.02 秒的延遲差不足以判定孰快。
All 3,706 processed predictions from /api/generate and /api/chat were character-for-character identical, with no difference across the four metrics. A 0.02 s latency gap is not enough to call one faster.
KEY 05
換 scorer，分數就會不一樣
Change the scorer and the score changes
prediction 完全沒變，只換 scorer，EM 就多判 30 題正確（+0.81pp）。差異來自空白／標點正規化與 CM 單向或雙向判定，屬報告層次差異，不是模型變強——跨團隊比分前應先對齊 scorer。
With predictions completely unchanged, swapping the scorer alone marked 30 more items correct (+0.81 pp). The difference comes from whitespace and punctuation normalisation and from whether CM is judged in one direction or both — a reporting-layer difference, not a stronger model. Teams should align scorers before comparing numbers.
KEY 06
縮短輸出：快 51%，但有隱藏代價
Shorter output: 51% faster, with a hidden cost
num_predict 由 80 降到 25，平均延遲降 51%，但 EM 降 1.03 個百分點；部分 prompt 複誦被截成不完整片段，反而躲過了針對完整片語設計的後處理規則。
Lowering num_predict from 80 to 25 cut mean latency by 51% but also cost 1.03 percentage points of EM: some prompt recitations were truncated into incomplete fragments, which then slipped past post-processing rules written for complete phrases.
本次建議設定
Recommended configuration from this study
—aligned 長 prompt、num_predict = 80、不顯式指定 repeat_penalty = 1.6、aligned 後處理、團隊統一 scorer；/api/generate 或 /api/chat 皆可，但同一場評測要固定同一種。
—此組合得到 EM 64.84%、CM 82.46%、ANLS 72.08%、F1 76.62%，是本次已測組合中最佳結果，也是後續 benchmark_suite/ 固定推論設定的依據之一。
—The aligned long prompt, num_predict = 80, no explicit repeat_penalty = 1.6, the aligned post-processing, and one scorer agreed across the team. Either /api/generate or /api/chat is fine, but a single evaluation run must stick to one of them.
—This combination yielded EM 64.84%, CM 82.46%, ANLS 72.08% and F1 76.62% — the best result among the combinations tested here, and one of the reasons the later benchmark_suite/ fixed its generation settings the way it did.
解讀限制：這是 OFAT 單變因主效應分析，各列 delta 不能直接相加，且只代表 GLM-OCR Q8、TC-STR 與當時的 Ollama 環境，未必可推廣至其他模型。CM 只檢查 prediction 是否包含 ground truth，答案藏在大量錯誤內容中仍可能得分，不應單獨用 CM 判斷 OCR 品質；temperature = 0 也不保證跨 Ollama 版本、推論後端或硬體逐字一致。
Limits on interpretation: this is an OFAT single-factor main-effect analysis, so the deltas in each row cannot simply be added together. It also only describes GLM-OCR Q8, TC-STR and the Ollama environment of the time, and does not necessarily generalise to other models. CM only checks whether the prediction contains the ground truth, so an answer buried inside a mass of wrong content can still score — CM alone should not be used to judge OCR quality. And temperature = 0 does not guarantee character-identical output across Ollama versions, inference backends or hardware.
查看完整結果摘要 ↗ 查看逐題案例互動報告 ↗
Read the full results summary ↗ Open the per-item interactive report ↗
04 / EVALUATION TASKS

評測任務與資料

Evaluation tasks and data

這是一套可重現（reproducible，任何人重跑都能得到相同結果）的評測工具，比較多個VLM在兩件任務上的表現。兩套工具都用Ollama依序載入不同模型、餵同一批圖片並記分。

This is a reproducible evaluation toolkit — anyone re-running it should get the same result — that compares several VLMs on two tasks. Both pipelines use Ollama to load models one after another, feed them the same set of images, and record the scores.

TASK A · TC_STR
繁體中文短文字辨識
Traditional Chinese short-text recognition

辨識圖片中的繁體中文短文字，類似傳統 OCR（光學文字辨識）。資料集為 esun-ai/traditional-chinese-text-recogn-dataset 官方 test split。

Recognise short Traditional Chinese text in an image, much like classic OCR (optical character recognition). The dataset is the official test split of esun-ai/traditional-chinese-text-recogn-dataset.

TASK B · OMNIDOCBENCH
整頁文件解析
Full-page document parsing

把整頁文件（標題、表格、公式、多欄排版）轉成結構化 Markdown。原始筆記寫作 OminDocBench，正確名稱為 OmniDocBench。

Convert a whole document page — headings, tables, formulas, multi-column layout — into structured Markdown. Our original notes spelled it OminDocBench; the correct name is OmniDocBench.

核心堅持是「公平比較」：所有模型看到的圖片、提示詞、參數完全一致，不會為表現差的模型偷偷調整。
The core commitment is fair comparison: every model sees exactly the same images, prompts and parameters, and nothing is quietly tuned for a model that is doing badly.
05 / PRINCIPLES

設計原則

Design principles

R1
同題同料
Same items, same inputs
同一場評測，每個模型看到的圖片、提示詞、參數都一樣。
Within one evaluation run, every model sees the same images, prompts and parameters.
R2
一次一張圖
One image per request
每次請求只送一張圖，不批次塞多張。
Each request carries a single image; images are never batched together.
R3
依序換模型
Models run one at a time
測完一個才卸載、換下一個，避免 GPU 記憶體被多模型同時占用。
A model is unloaded before the next is loaded, so GPU memory is never shared between models.
R4
暖機不計分
Warm-up is not scored
模型剛載入時速度不穩，這段 warm-up 的結果不算分。
A freshly loaded model has unstable throughput, so warm-up results are excluded from the score.
R5
保留爛表現
Bad output stays in
答錯、答空白、被截斷都算模型自己的實力，不事後清理答案。
Wrong answers, blank answers and truncated answers all count as the model's own performance; nothing is cleaned up after the fact.
R6
原始回應永久保留
Raw responses are kept forever
不論好壞都存起來，方便日後查證。
Good or bad, every response is stored so it can be checked later.

這樣可以確保分數低是模型能力問題，而不是測試流程出錯造成的誤判。

Together these ensure that a low score reflects model capability rather than a misjudgement caused by a broken test pipeline.

06 / PIPELINE

整體運算流程

The end-to-end pipeline

TC_STR 與 OmniDocBench 都走同一條流程，中間有兩道人工／自動關卡。

TC_STR and OmniDocBench follow the same pipeline, with two gates — one automatic, one human — along the way.

01
準備工作
Preparation
下載資料集、確認模型清單，並用SHA-256記錄每張圖的指紋。
Download the dataset, confirm the model list, and record a SHA-256 fingerprint for every image.
02
Preflight 事前檢查
Preflight checks
確認模型版本、顯示卡是否 100% 由本次評測占用。
Verify the model version and that the GPU is 100% dedicated to this run.
失敗 →
on failure →
BLOCKED
標記後中止，不繼續往下跑
Flagged and aborted; the run does not continue
03
Smoke 小規模測試
Smoke test
TC_STR 取 20 題／OmniDocBench 取 20 頁先跑一次。
A first pass over 20 TC_STR items or 20 OmniDocBench pages.
04
人工檢視結果
Human review of the results
HUMAN GATE
流程不會自動跑到底：smoke 結果須經人工確認沒問題，才進入正式全量評測。
The pipeline does not run all the way through by itself: a person must confirm the smoke results look right before the full evaluation begins.
有問題 ↺
if wrong ↺
退回步驟 02 重新檢查
Return to step 02 and check again
05
操作者手動啟動正式全量評測
The operator starts the full run manually
避免無人監督下產生一堆錯誤結果。
This avoids generating piles of bad results with nobody watching.
06
逐題／逐頁推論
Per-item / per-page inference
寫入 SQLite checkpoint，中斷可續跑。
Progress is written to a SQLite checkpoint, so an interrupted run can resume.
07
正式評分
Official scoring
TC_STR 自行計算；OmniDocBench 使用官方 Docker 評分程式。
TC_STR is scored by our own code; OmniDocBench uses the official Docker evaluator.
08
產出報告
Report generation
HTML 互動報告、CSV、JSON。
Interactive HTML report, CSV and JSON.
簽章比對（signature）
Signature matching
資料版本、模型版本、提示詞、參數任一改變，系統就判定是「不同的一次評測」，新舊結果不會混在一起計分。
If the dataset version, model version, prompt or parameters change in any way, the system treats it as a different evaluation run, and old and new results are never scored together.
07 / ENVIRONMENT

系統環境

System environment

雲端主機
Cloud host
AWS EC2
作業系統
Operating system
Ubuntu 26.04
顯示卡
GPU
NVIDIA L40S
≈45GB VRAM
模型執行工具
Model runtime
Ollama 0.31.1
容器化工具
Containerisation
Docker
鎖定版本
version-pinned
可重建暫存區
Rebuildable scratch storage
關機會消失，但能重新下載，例如資料集本身。
Lost on shutdown but re-downloadable — the dataset itself, for example.
永久保存區
Persistent storage
存放檢查點、原始回應與報告，不因關機消失。
Holds checkpoints, raw responses and reports, and survives shutdown.
08 / TC_STR

TC_STR：繁體中文短文字辨識

TC_STR: Traditional Chinese short-text recognition

資料來源為 esun-ai/traditional-chinese-text-recogn-dataset 的官方 test split，共 3,706 筆。提示詞要求模型「只輸出圖片中看到的文字」，所有模型逐字相同；temperature = 0、num_predict = 80、逾時 180 秒。

The data is the official test split of esun-ai/traditional-chinese-text-recogn-dataset, 3,706 items in total. The prompt tells the model to output only the text it sees in the image, and is character-for-character identical across all models. Settings: temperature = 0, num_predict = 80, 180-second timeout.

5.1 參與比較的 8 個模型

5.1 The 8 models compared

卡片內為參數量與量化方式；標記 MoE 者為混合專家模型。

Each card shows the parameter count and the quantisation format. Models marked MoE are mixture-of-experts models.

Gemma 4 E2B
4.6B
Q4_0
Gemma 4 E4B
7.5B
Q4_0
Gemma 4 12B
11.9B
Q4_0
Gemma 4 26B A4B
25.2B
實際啟用 ≈3.8B
≈3.8B active
Q4_0MoE
Gemma 4 31B
30.7B
Q4_0
GLM OCR BF16
1.1B
F16計分例外scoring exception
GLM-4.6V-Flash 9B
9.4B
Q4_K_M
Kimi-VL-A3B-Instruct
16B
實際啟用 ≈3B
≈3B active
Q4_K_MMoE

5.2 評分指標（皆為越高越好）

5.2 Scoring metrics (higher is better for all four)

Exact Match
答案跟正確答案（ground truth, GT）一字不差才算對。
Counted correct only if the answer matches the ground truth (GT) character for character.
Containment Match
正確答案是否完整出現在模型答案裡（允許多輸出）。
Whether the ground truth appears in full inside the model's answer (extra output is allowed).
ANLS
用編輯距離算相似度，低於 0.5 直接以 0 分計。
A similarity derived from edit distance; anything below 0.5 is scored as 0.
Character F1
拆成單字比對，答對多少、又多打多少，綜合算分。
Compares character by character, combining how much was right with how much extra was produced.
計分例外：GLM OCR BF16 有時回傳正常文字，卻同時標記 done=false 且缺統計欄位。人工確認後，只有這個模型放寬規則：有回傳文字就納入計分，但註記「無法判斷是否截斷」，效率數據不能與其他模型比較。
Scoring exception: GLM OCR BF16 sometimes returned perfectly normal text while also reporting done=false and omitting its statistics fields. After manual review, the rule was relaxed for this model only: any returned text is scored, but flagged as "truncation cannot be determined", and its efficiency figures are not comparable with the other models.
09 / OMNIDOCBENCH

OmniDocBench：整頁文件解析

OmniDocBench: full-page document parsing

使用 Hugging Face 的 opendatalab/OmniDocBench，鎖定 v1.6、共 1,651 頁，涵蓋多語言、多欄排版、表格、公式、模糊掃描等情境。提示詞要求輸出整頁 Markdown：保留閱讀順序、表格轉成保留合併儲存格的 HTML table、公式以 LaTeX 表示。

We use opendatalab/OmniDocBench from Hugging Face, pinned to v1.6, 1,651 pages in total, covering multilingual content, multi-column layouts, tables, formulas and blurry scans. The prompt asks for the whole page as Markdown: reading order preserved, tables converted to HTML tables that keep merged cells, and formulas expressed in LaTeX.

Gemma 4 系列
Gemma 4 family
E2B · E4B · 12B
26B A4B · 31B
Qwen3-VL
4B · 32B
InternVL3.5
4B · 38B
社群轉換 GGUF 版
community-converted GGUF build

6.1 評分方式：官方 evaluator

6.1 Scoring method: the official evaluator

與 TC_STR 自行算分不同，這裡用 OmniDocBench 官方釘死版本的評分程式，跑在 Docker 容器裡，確保評分邏輯不因主機環境而異。

Unlike TC_STR, which we score ourselves, this task uses OmniDocBench's official scoring program at a pinned version, run inside a Docker container so the scoring logic cannot drift with the host environment.

↓ 越低越好
↓ lower is better
Text 編輯距離
Text edit distance
純文字與正確答案差多少（正規化後）。
How far the plain text is from the ground truth, after normalisation.
↑ 越高越好
↑ higher is better
Formula CDM
Formula CDM
公式結構與內容是否比對正確。
Whether formula structure and content match correctly.
↑ 越高越好
↑ higher is better
Table TEDS
Table TEDS
表格樹狀結構的相似度。
Similarity of the table's tree structure.
↓ 越低越好
↓ lower is better
Reading Order
閱讀順序是否與正確答案一致。
Whether reading order matches the ground truth.
官方綜合
official composite
Overall
((1−text edit)×100
+ TEDS + CDM) ÷ 3

另保留與 TC_STR 相同的 4 個補充診斷指標，但標註「非官方排行榜指標」，避免與正式分數混淆。

We also keep the same four supplementary diagnostic metrics used for TC_STR, but label them "not official leaderboard metrics" so they are not confused with the official scores.

DEBUG LOG
復讀迴圈讓官方 evaluator 卡死
Repetition loops hang the official evaluator

用參數量較小的模型（如 Gemma 4 E2B）跑完整測試集時，約 30% 的輸出會出現「答案不斷復讀」。這類輸出流入官方 matching 流程後，連內建用來處理長輸入的chunked Hungarian algorithm備援機制也會一起卡死、不回傳結果。

Running the full test set with smaller models such as Gemma 4 E2B, roughly 30% of the outputs fall into endless repetition of the answer. Once such output reaches the official matching stage, even the built-in chunked Hungarian algorithm fallback for long inputs hangs and never returns a result.

match_workers: 4 → 1
把 evaluator 的平行工作行程數調降即可避免卡死（官方 issue #228）。
Lowering the evaluator's worker-process count avoids the hang (upstream issue #228).
temperature: 1.0
temperature 設太低容易誘發復讀，故 OmniDocBench 固定 1.0（TC_STR 為 0）。
Too low a temperature readily triggers repetition, so OmniDocBench is fixed at 1.0 (TC_STR uses 0).
10 / RESULTS

研究結果

Results

以下圖表寬度均依表格數值等比例縮放，與數字完全一致。

Every bar below is scaled in direct proportion to the value in the corresponding table, so the graphics and the numbers agree exactly.

10.1 TC_STR 全量跑分結果

10.1 TC_STR full-set results

全量 3,706 筆測試集，依 EM 由高到低。數值取自 run tcstr_20260729T162125139376Z 的 primary leaderboard。

The complete 3,706-item test set, ordered by EM from high to low. Values are taken from the primary leaderboard of run tcstr_20260729T162125139376Z.

Exact Match 排行
Exact Match ranking
0–100% · 越高越好
0–100% · higher is better
GLM-4.6V-Flash 9B
88.4
Gemma 4 26B A4B
82.0
Gemma 4 31B
80.6
Gemma 4 12B
58.9
Gemma 4 E4B
49.0
Gemma 4 E2B
43.2
Kimi-VL-A3B-Instruct
38.5
GLM OCR BF16
36.7
GLM-4.6V-Flash Gemma 4（越深＝模型越大） Kimi-VL-A3B GLM OCR BF16（截斷率最高）
GLM-4.6V-Flash Gemma 4 (darker = larger model) Kimi-VL-A3B GLM OCR BF16 (highest truncation rate)
事後驗證
POST-HOC CHECK
GLM-4.6V-Flash（9.4B）為何贏過 Gemma 4 31B（30.7B）？
Why does GLM-4.6V-Flash (9.4B) beat Gemma 4 31B (30.7B)?
針對這個只用約三分之一參數量卻全面勝出的現象，我們額外做了事後查證：先排除「計分方式不公平」的可能——兩模型使用的 prompt、生成參數、後處理規則與 scorer 完全相同，且 GLM-4.6V-Flash 的 EM 與 CM 幾乎相等（落差僅 0.08 個百分點，Gemma 31B 則是 0.54 個百分點），代表領先是真實辨識能力的差距，不是評分方式造成的假象。
A model with roughly a third of the parameters winning across the board deserved a closer look, so we ran a post-hoc check. First we ruled out unfair scoring: both models used identical prompts, generation parameters, post-processing rules and scorer, and GLM-4.6V-Flash's EM and CM are nearly equal (a gap of just 0.08 percentage points, against 0.54 for Gemma 31B). The lead therefore reflects a real difference in recognition ability, not an artefact of how we scored.
不過，兩個模型都不在專案保留的逐題原始紀錄裡，無法進一步拆解「贏在哪類題目」。我們推測，比較可能的原因是訓練資料的針對性——GLM 系列的產品定位本來就偏向 OCR／文件理解，很可能在預訓練或微調階段用了大量結構化短答案文字辨識語料；Gemma 4 則是通用型多模態模型，OCR 只是它眾多能力之一。但這純粹是根據兩個模型的產品定位做的推論，目前沒有訓練資料組成的直接證據可以證實。
Neither model, however, appears in the per-item raw records the project retained, so we cannot break down exactly which kinds of items it won on. Our conjecture is that the likeliest cause is how targeted the training data was: the GLM series is positioned as an OCR / document-understanding product line and very plausibly saw large amounts of structured short-answer text-recognition corpora during pre-training or fine-tuning, whereas Gemma 4 is a general-purpose multimodal model for which OCR is one capability among many. This is purely an inference from the two models' product positioning; we have no direct evidence about training-data composition to confirm it.
事後驗證
POST-HOC CHECK
Gemma 4 26B A4B（MoE，實際啟用≈3.8B）以些微差距超過 31B 稠密模型
Gemma 4 26B A4B (MoE, ≈3.8B active) edges past the 31B dense model
這裡出現一個反直覺的現象：26B A4B 標示的參數量比 31B 少，且是 MoE 架構，實際啟用參數只有約 3.8B，卻在這個短文字 OCR 任務上以 82.0% 些微超過稠密架構的 31B（80.6%）。我們推測，短文字 OCR 本質上是「認出一小段字串」的局部匹配型任務，MoE 只要有一兩個「專家」剛好擅長處理這類短字串辨識，就足以把整體表現拉起來，不太需要動用模型的全部容量。
A counter-intuitive result: 26B A4B lists fewer parameters than 31B, is a MoE architecture, and activates only about 3.8B parameters — yet on this short-text OCR task it edges past the dense 31B model, 82.0% to 80.6%. Our conjecture is that short-text OCR is essentially a local-matching task — recognise one short string — so a MoE only needs one or two experts that happen to be good at short-string recognition to lift overall performance, without drawing on the model's full capacity.
但這個優勢並不穩定——在 10.2 節的 OmniDocBench 整頁文件解析任務中，同樣這兩個模型的名次完全反過來，26B A4B 反而輸給 31B 8.5 分（詳見下方對應段落）。這提醒我們，「MoE 用更少參數打贏稠密模型」比較像是任務依賴的個案，而不是一個穩定的架構優勢。
The advantage is not stable, though: on the OmniDocBench full-page parsing task in section 10.2 the two models swap places entirely, with 26B A4B losing to 31B by 8.5 points (see the corresponding passage below). That suggests "MoE beats dense with fewer parameters" is a task-dependent one-off rather than a durable architectural advantage.
EM 對比 CM
EM versus CM
0–100% · 越高越好
0–100% · higher is better
EM 完全相符 CM 包含比對
EM exact match CM containment match
GLM-4.6V-Flash 9B
88.4
88.5
Gemma 4 26B A4B
82.0
82.5
Gemma 4 31B
80.6
81.2
Gemma 4 12B
58.9
60.4
Gemma 4 E4B
49.0
49.2
Gemma 4 E2B
43.2
43.8
Kimi-VL-A3B-Instruct
38.5
39.6
GLM OCR BF16
36.7
77.2
GLM OCR BF16 是唯一 EM 遠低於 CM 的模型，落差達 40.5 個百分點。
GLM OCR BF16 is the only model whose EM falls far below its CM — a gap of 40.5 percentage points.
事後驗證
POST-HOC CHECK
GLM OCR BF16 的 EM／CM 落差：復讀，還是別的原因？
The EM/CM gap for GLM OCR BF16: repetition, or something else?
我們對這 40.5 個百分點的落差做了更深入的事後檢視。由於 BF16、tcstr_ 這次執行的逐題原始資料沒有保留下來，改用同模型家族唯一可取得的逐題資料（GLM-OCR Q8_0，3,706 筆）做替代分析：約 19.8%（734 筆）的輸出出現「格式失控」（含多餘的 markdown 標記或片語重複迴圈），這個子集的 EM 是 0%、CM 卻高達 76%，落差型態跟這裡看到的現象一致。
We examined this 40.5-point gap more closely. Because the per-item raw data for the BF16 tcstr_ run was not retained, we substituted the only per-item data available from the same model family (GLM-OCR Q8_0, 3,706 items). There, about 19.8% of outputs (734 items) show "format breakdown" — stray markdown markers or phrase-repetition loops. That subset scores 0% EM but as much as 76% CM, the same gap pattern seen here.
但值得注意的是，如果嚴格定義成「同一個片語連續重複」，符合的只有 0.24%（9 筆）——多數案例其實是模型答對之後，接著用簡體中文自言自語一大段，而不是單純卡在同一句話跳針。我們推測，這種「答對後切換語域碎念」的現象，可能跟訓練語料裡混雜了簡體與繁體 OCR 資料有關：模型在輸出信心開始下降、進入某種不確定狀態時，反而會切換到訓練資料裡出現頻率更高的簡體語域繼續生成，而不是單純卡在同一個 token 迴圈裡。這只是根據現象做的合理推測，目前沒有訓練語料組成的資料可以驗證。另外我們也發現，答案越長，格式失控的比例越高（從 1 個字的 12.85% 一路上升到 7–10 個字的 34.99%），顯示這個問題可能跟輸出長度直接相關。
Notably, under the strict definition of "the same phrase repeated consecutively", only 0.24% (9 items) qualify. In most cases the model actually gets the answer right and then rambles on at length in Simplified Chinese, rather than merely skipping like a stuck record. Our conjecture is that this "correct answer, then a drift into another register" behaviour may relate to Simplified and Traditional OCR data being mixed in the training corpus: as output confidence starts to fall and the model enters some uncertain state, it switches to the Simplified register that appears more frequently in training data and keeps generating, instead of getting caught in a single-token loop. This is a plausible reading of the symptom only; we have no data on corpus composition to verify it. We also found that the longer the answer, the higher the rate of format breakdown — rising steadily from 12.85% for single-character answers to 34.99% for 7–10 character answers — which suggests the problem is directly tied to output length.
平均延遲
Mean latency
秒／題 · 越低越好
seconds per item · lower is better
GLM-4.6V-Flash 9B
0.41s
Gemma 4 26B A4B
1.18s
Gemma 4 31B
1.20s
Gemma 4 12B
1.27s
Gemma 4 E4B
1.22s
Gemma 4 E2B
1.16s
Kimi-VL-A3B-Instruct
0.33s
GLM OCR BF16
0.27s
截斷樣本比例
Share of truncated samples
佔 3,706 筆 · 越低越好
out of 3,706 items · lower is better
GLM-4.6V-Flash 9B
0.0%
Gemma 4 26B A4B
0.2%
Gemma 4 31B
0.0%
Gemma 4 12B
0.1%
Gemma 4 E4B
2.9%
Gemma 4 E2B
3.4%
Kimi-VL-A3B-Instruct
12.2%
GLM OCR BF16
54.9%
事後驗證
POST-HOC CHECK
Kimi-VL-A3B 的高截斷率，其實可能與 MoE 無關
Kimi-VL-A3B's high truncation rate may have nothing to do with MoE
同樣是「實際啟用參數很小」的 MoE 模型，Kimi-VL-A3B 的截斷率（12.2%）卻是 Gemma 4 26B A4B（0.2%）的 60 倍。我們原本猜測「MoE 本身不保證穩定性，要看個別模型的訓練與量化品質」，事後查證發現一個更具體的線索：Kimi-VL 用的是第三方（mradermacher）自行製作的 Q4_K_M 靜態量化版本、非官方 Moonshot tag；而 Gemma 26B A4B 是 Google 官方做 QAT（量化感知訓練）後直接透過 Ollama 官方管道釋出。量化來源的「官方程度」落差方向，剛好跟兩者穩定性落差的方向一致。
Both are MoE models with small active-parameter counts, yet Kimi-VL-A3B's truncation rate (12.2%) is 60 times that of Gemma 4 26B A4B (0.2%). Our initial guess was simply that "MoE guarantees no stability; it depends on each model's training and quantisation quality." The post-hoc check turned up a more concrete lead: Kimi-VL used a third-party (mradermacher) Q4_K_M static quantisation rather than an official Moonshot tag, whereas Gemma 26B A4B was released by Google through the official Ollama channel after QAT (quantisation-aware training). The gap in how "official" the quantisation source is points in exactly the same direction as the gap in stability.
我們推測，官方做 QAT 時，訓練過程本身就會針對量化後的數值分布做校正，讓模型在低精度下依然能準確判斷「什麼時候該停止生成」；第三方事後靜態量化沒有這道校正手續，可能特別容易讓這類對數值精度敏感的停止訊號失準，導致生成失控、一路輸出到被截斷。這只是根據量化流程原理做的推論——由於兩模型的逐題資料都沒有保留，無法比對兩者「在哪類輸入上特別容易截斷」，也無法驗證是不是特定專家路由出了問題（Ollama 的 GGUF 推論本身不會輸出路由資訊）。
Our conjecture is that official QAT calibrates for the post-quantisation numeric distribution during training itself, so the model still judges accurately when to stop generating even at low precision. Third-party post-hoc static quantisation skips that calibration step, which may make precision-sensitive stop signals particularly prone to going astray — generation runs away and keeps going until it is truncated. This is an inference from how the quantisation processes work: because per-item data was retained for neither model, we cannot compare which kinds of input each is most likely to truncate on, nor verify whether specific expert routing is at fault (Ollama's GGUF inference does not expose routing information at all).
# MODEL EM↑ CM↑ ANLS↑ F1↑ 延遲 截斷
# MODEL EM↑ CM↑ ANLS↑ F1↑ LATENCY TRUNCATED
1 GLM-4.6V-Flash 9B 88.37% 88.45% 94.07% 94.30% 0.41s 0 / 3,706
2 Gemma 4 26B A4B 81.98% 82.46% 88.71% 89.58% 1.18s 7 / 3,706
3 Gemma 4 31B 80.63% 81.17% 87.82% 88.55% 1.20s 0 / 3,706
4 Gemma 4 12B 58.90% 60.39% 72.89% 75.06% 1.27s 4 / 3,706
5 Gemma 4 E4B 48.95% 49.19% 69.99% 72.87% 1.22s 109 / 3,706
6 Gemma 4 E2B 43.20% 43.82% 66.52% 70.09% 1.16s 126 / 3,706
7 Kimi-VL-A3B-Instruct 38.51% 39.64% 49.86% 53.69% 0.33s 453 / 3,706
8 GLM OCR BF16 36.70% 77.23% 40.72% 45.41% 0.27s 2,034 / 3,706
KEY 01
GLM-4.6V-Flash 9B 全面最強
GLM-4.6V-Flash 9B is strongest across the board
EM／CM／ANLS／F1 四項全部最高，零截斷、零異常、成功率 100%，速度也是數一數二快（0.41 秒／題）。
Highest on all four of EM, CM, ANLS and F1, with zero truncations, zero anomalies and a 100% success rate — and among the fastest at 0.41 s per item.
KEY 02
Gemma 越大越準，但非線性
Bigger Gemma is more accurate, but not linearly
26B A4B（MoE，實際啟用 ≈3.8B）以 82.0 些微超過參數更大的 31B 稠密模型（80.6），與 7.2 節「嚴格隨參數量遞增」不同。
26B A4B (MoE, ≈3.8B active) edges past the larger dense 31B model, 82.0 to 80.6 — unlike section 7.2, where scores rise strictly with parameter count.
KEY 03
GLM OCR BF16 的 EM／CM 落差
The EM/CM gap for GLM OCR BF16
54.9%（2,034／3,706）樣本被標記截斷：答案常「藏」在輸出裡（CM 高），但格式失控而在嚴格比對下失分（EM 低）。
54.9% of samples (2,034 of 3,706) are flagged as truncated. The answer is often buried inside the output — hence the high CM — but format breakdown loses the points under strict matching, hence the low EM.
延伸觀察
Further observations
—推論速度跟架構的關係比參數量更直接：三個非 Gemma 模型延遲都在 0.27～0.41 秒，遠快於所有 Gemma 4（1.15～1.27 秒），可能與量化方式（F16／Q4_K_M vs Q4_0）或推論架構有關。
—Kimi-VL-A3B-Instruct 雖是 MoE、實際啟用參數小，異常率卻不低（453 筆截斷），EM 僅 38.5%：MoE 本身不保證穩定性，仍要看個別模型的訓練與量化品質。
—Inference speed tracks architecture more directly than parameter count: the three non-Gemma models all sit at 0.27–0.41 s, far faster than every Gemma 4 (1.15–1.27 s). Quantisation format (F16 / Q4_K_M vs Q4_0) or inference architecture may be responsible.
—Kimi-VL-A3B-Instruct is a MoE with few active parameters, yet its anomaly rate is far from low (453 truncations) and its EM only 38.5%. MoE alone guarantees no stability; it still comes down to each model's training and quantisation quality.

10.2 OmniDocBench 全語料庫跑分

10.2 OmniDocBench full-corpus results

官方 evaluator 在全語料庫 1,651 頁上的正式分數，依 Overall 由高到低。Overall 只吃 text、table、formula 三項。

Official evaluator scores over the full 1,651-page corpus, ordered by Overall from high to low. Overall draws on just three components: text, table and formula.

Overall 總分排行
Overall score ranking
0–100 · 越高越好
0–100 · higher is better
Qwen3-VL 32B
85.3
Qwen3-VL 4B
75.6
Gemma 4 31B
71.9
InternVL3.5 38B
70.5
InternVL3.5 4B
69.6
Gemma 4 26B A4B
63.4
Gemma 4 12B
37.5
Gemma 4 E4B
26.9
Gemma 4 E2B
17.3
Qwen3-VL Gemma 4 InternVL3.5
事後驗證
POST-HOC CHECK
10.1 節「MoE 以小勝大」的優勢，在這裡完全消失
The "MoE punches above its weight" advantage from 10.1 vanishes entirely here
10.1 節的 TC_STR 短文字 OCR 測驗裡，Gemma 4 26B A4B（MoE，實際啟用≈3.8B）以些微差距贏過稠密架構的 Gemma 4 31B。但換到這裡的整頁文件解析任務，兩者名次完全反過來：26B A4B 的 Overall 只有 63.4，31B 卻拿到 71.9，反而輸了 8.5 分。
In the TC_STR short-text OCR test in section 10.1, Gemma 4 26B A4B (MoE, ≈3.8B active) narrowly beat the dense Gemma 4 31B. On the full-page document parsing task here, the order reverses completely: 26B A4B manages an Overall of just 63.4 against 71.9 for the 31B, losing by 8.5 points.
我們推測，整頁文件解析同時需要處理版面順序、跨區塊表格、公式辨識等多種能力，比較吃模型的「整體容量」，不像短文字 OCR 那樣只需要局部字串匹配；MoE 架構在推論時每次只啟用一小部分參數，遇到需要綜合多種能力的任務時，可能就沒辦法像稠密模型那樣「每一層都動用全部參數」來處理複雜的版面關係。這個解釋合理，但目前沒有機制層級的證據（例如逐頁的專家路由紀錄）可以證實，只能說「MoE 打贏稠密模型」看起來是任務依賴的個案，不是穩定的架構優勢。
Our conjecture is that full-page parsing simultaneously demands layout ordering, tables that span blocks, formula recognition and more, which draws on the model's overall capacity — unlike short-text OCR, which only needs local string matching. A MoE activates a small share of its parameters per token, so on a task requiring several capabilities at once it may be unable to bring "all parameters at every layer" to bear on complex layout relationships the way a dense model can. The explanation is plausible, but we have no mechanism-level evidence such as per-page expert-routing records to confirm it. All we can say is that "MoE beats dense" looks like a task-dependent one-off rather than a durable architectural advantage.
四維能力雷達
Capability radar
全部換算為 0–100，越外圈越好
all rescaled to 0–100; further out is better
Text TEDS CDM Order Overall
Text／Order 已由編輯距離換算為準確度（1 − edit）。點選右側模型即可疊圖比較。
Text and Order have been converted from edit distance to accuracy (1 − edit). Select models on the right to overlay them.
選擇要疊圖的模型
Choose models to overlay
Qwen3-VL 32B
Text
88.9
TEDS
78.5
CDM
88.5
Order
79.4
Overall
85.3
Gemma 4 31B
Text
71.4
TEDS
63.2
CDM
81.3
Order
72.4
Overall
71.9
Gemma 4 E2B
Text
19.9
TEDS
6.8
CDM
25.2
Order
37.3
Overall
17.3
# MODEL OVERALL↑ TEXT EDIT↓ TEDS↑ CDM↑ ORDER↓
1 Qwen3-VL 32B 85.31 0.111 0.785 0.885 0.206
2 Qwen3-VL 4B 75.62 0.149 0.676 0.742 0.243
3 Gemma 4 31B 71.93 0.286 0.632 0.813 0.276
4 InternVL3.5 38B 70.47 0.255 0.579 0.790 0.282
5 InternVL3.5 4B 69.64 0.196 0.593 0.693 0.258
6 Gemma 4 26B A4B 63.43 0.374 0.553 0.724 0.351
7 Gemma 4 12B 37.46 0.644 0.301 0.467 0.502
8 Gemma 4 E4B 26.89 0.714 0.204 0.317 0.549
9 Gemma 4 E2B 17.32 0.801 0.068 0.252 0.627
KEY 01
Qwen3-VL 32B 全面領先
Qwen3-VL 32B leads across the board
Overall 把第二名拉開近 10 分，表格 TEDS 與公式 CDM 都是九個模型中最好的。
Its Overall opens a near-10-point lead over second place, and it posts the best table TEDS and formula CDM of all nine models.
KEY 02
Gemma 4 幾乎與參數量成正比
Gemma 4 scales almost in proportion to parameter count
從 E2B（17.3）一路爬到 31B（71.9），沒有中途停滯，顯示文件解析能力很吃模型規模。
It climbs steadily from E2B (17.3) all the way to 31B (71.9) with no plateau along the way, showing that document parsing depends heavily on model scale.
KEY 03
小模型不一定輸大模型
Small models do not necessarily lose to large ones
Qwen3-VL 4B（75.6）打贏 Gemma 4 31B；InternVL3.5 從 4B（69.6）放大到 38B（70.5）也只好一點點。
Qwen3-VL 4B (75.6) beats Gemma 4 31B, and scaling InternVL3.5 from 4B (69.6) to 38B (70.5) buys only a marginal gain.
延伸觀察
Further observations
—Gemma 4 E2B 的 table TEDS 只有 0.068，幾乎等於「看不懂表格」；同系列放大到 31B 才追到 0.632，表格結構理解對小模型特別吃力。
—formula CDM 普遍是九個模型裡相對最好的一項，即使最小的 E2B 也有 0.252，推測與 LaTeX 符號結構相對固定有關。
—Qwen3-VL 4B 與 InternVL3.5 4B 的 Overall 都超過「號稱較大」的 Gemma 4 26B A4B（63.4）：架構與訓練資料品質，有時比帳面參數量更關鍵。
—Gemma 4 E2B's table TEDS is only 0.068 — effectively "cannot read tables at all". Scaling the same family up to 31B only reaches 0.632, so table-structure understanding is especially hard for small models.
—Formula CDM is generally the relatively strongest metric across all nine models; even the smallest, E2B, reaches 0.252. We suspect this relates to the comparatively fixed structure of LaTeX notation.
—Both Qwen3-VL 4B and InternVL3.5 4B post a higher Overall than the nominally larger Gemma 4 26B A4B (63.4): architecture and training-data quality sometimes matter more than the parameter count on paper.
事後驗證
POST-HOC CHECK
Gemma 4 E2B 表格分數崩到 0.068，真的是「版面類型」造成的嗎？
Gemma 4 E2B's table score collapses to 0.068 — is layout type really the cause?
我們原本猜測是特定版面類型（跨頁表格、合併儲存格、多欄排版）造成 E2B 的表格分數特別低，但事後查證發現這個假說目前無法驗證：OmniDocBench 官方標註雖然定義了版面子分類欄位，但實際的 665 筆表格標註全部是空的，官方自己都沒有填入分類資訊。改用可取得的替代欄位（頁面版面類型、表格數量、bbox 大小）分層後，還發現一個違反直覺的現象——含表格的頁面反而比全體頁面更常是單欄排版（65.5% 對 53.7%），多欄佔比更低，跟「多欄排版是主因」的直覺方向相反。
We initially guessed that particular layout types — tables spanning pages, merged cells, multi-column text — were behind E2B's especially low table score. The post-hoc check found this hypothesis currently untestable: although the official OmniDocBench annotations define layout sub-category fields, all 665 actual table annotations are empty, so even upstream never filled the classification in. Stratifying instead by the substitute fields available — page layout type, table count, bbox size — produced a counter-intuitive finding: pages containing tables are in fact more often single-column than pages overall (65.5% versus 53.7%), with a smaller multi-column share, the opposite direction from the intuition that multi-column layout is the main cause.
我們反而在原始報告的除錯紀錄裡找到另一條線索：小模型（含 E2B）約有三成輸出會出現復讀，導致官方 matching 演算法連同備援機制一起卡死。我們推測，表格輸出需要模型同時維持大量 HTML 標籤（如 <td>、<tr>、colspan）的巢狀結構，token 密度和語法複雜度都比連續段落文字高出許多；如果小模型本來就有「長輸出容易復讀」的傾向，這種結構化、重複性標籤特別多的輸出格式，可能比一般段落文字更容易誘發復讀迴圈，進而拖累 TEDS 分數。不過這個解釋把兩條線索接在一起，目前沒有逐頁資料能證實表格輸出的復讀率是否真的比其他內容類型更高，仍屬推論。
Instead we found another lead in the debug log of the original report: roughly 30% of small-model outputs (E2B included) fall into repetition, hanging the official matching algorithm along with its fallback. Our conjecture is that table output requires the model to maintain a large nested structure of HTML tags — <td>, <tr>, colspan and so on — at far higher token density and syntactic complexity than continuous prose. If small models already tend to repeat on long outputs, this heavily structured, tag-repetitive format may trigger repetition loops more readily than ordinary paragraphs, dragging the TEDS score down with it. This explanation does splice two leads together, though, and with no per-page data we cannot confirm whether table output really does repeat more often than other content types. It remains a conjecture.
事後驗證
POST-HOC CHECK
公式 CDM 對規模「相對」不敏感，但原因跟原本想的不完全一樣
Formula CDM is "relatively" insensitive to scale — but not quite for the reason we assumed
原始報告推測是「LaTeX 符號結構相對固定」讓公式分數比較不受模型規模影響。我們先驗證了一個競爭假說——會不會其實是測試集裡的公式本來就偏簡單，造成天花板效應？結果用官方標註的 2,066 條公式做複雜度分層後發現並非如此：簡單公式只佔 27.9%，超過 42% 含矩陣結構或高密度 LaTeX 指令，測試集本身並不簡單，天花板假說站不住腳。
The original report suggested the relatively fixed structure of LaTeX notation made formula scores less sensitive to model scale. We first tested a competing hypothesis: might the formulas in the test set simply be easy, producing a ceiling effect? Stratifying the 2,066 officially annotated formulas by complexity showed otherwise — only 27.9% are simple, while over 42% contain matrix structures or dense LaTeX commands. The test set is not easy, so the ceiling hypothesis does not hold.
量化比較後也確認 CDM 確實比表格 TEDS「相對不敏感」：CDM 的最小／最大模型分數比值是 28.5%（E2B 的 0.252／Qwen3-VL 32B 的 0.885），明顯高於表格 TEDS 的 8.7%（0.068／0.785）。但 CDM 仍然隨模型規模單調上升，並不是一條真正的平坦線，所以「不受規模影響」這個說法本身其實略為誇大。我們推測，既然測試集不是偏簡單，比較站得住腳的解釋方向，還是回到「LaTeX 符號本身有高度局部化、重複出現的視覺模式」（例如分數線、根號、上下標的固定寫法）——小模型即使整體語意理解能力弱，只要預訓練階段看過足夠多的數學符號 OCR 語料，也能靠局部模式匹配辨認出來；相較之下，表格需要模型在較大範圍內追蹤欄位對齊與跨儲存格關係，這種長距離空間推理能力，明顯更依賴模型容量。這仍然是我們根據任務性質做的推論，還需要逐公式、依複雜度分層的分數才能真正證實。
Quantifying the comparison did confirm that CDM is relatively less sensitive than table TEDS: the smallest-to-largest model score ratio for CDM is 28.5% (E2B's 0.252 over Qwen3-VL 32B's 0.885), clearly above the 8.7% for table TEDS (0.068 over 0.785). But CDM still rises monotonically with model scale rather than tracing a genuinely flat line, so the claim that it is "unaffected by scale" is itself somewhat overstated. Our conjecture is that, since the test set is not easy, the more defensible explanation still returns to LaTeX notation having highly localised, recurring visual patterns — the fixed way fraction bars, radicals and sub/superscripts are written. Even a small model with weak overall semantic understanding can recognise these by local pattern matching, provided it saw enough mathematical-symbol OCR data in pre-training. Tables, by contrast, require the model to track column alignment and cross-cell relationships over a much wider span, and that kind of long-range spatial reasoning clearly depends far more on model capacity. This too is an inference from the nature of the task; per-formula scores stratified by complexity would be needed to really confirm it.

10.3 結果呈現方式

10.3 How results are presented

README 特別強調不把「正在跑」的分數寫死在文件裡，實際數字以每次評測輸出資料夾裡的報告為準。工具本身還會產出：

The README deliberately avoids hard-coding in-progress scores into the documentation; the authoritative numbers are the reports in each run's output directory. The tooling also produces:

互動式比較報告（HTML）
An interactive comparison report (HTML)
可依任何指標排序、篩選「表現最好／最差／隨機抽樣」的頁面，快速找出模型的弱點情境。
Sortable by any metric, with filters for best / worst / random-sample pages, so a model's weak scenarios surface quickly.
詳細統計
Detailed statistics
成功率、API 失敗次數、重試次數、空白輸出次數、平均延遲，區分「模型能力問題」與「系統工程問題」。
Success rate, API failure count, retry count, blank-output count and mean latency — separating model-capability problems from systems-engineering problems.
逐頁／逐題原始紀錄
Per-page / per-item raw records
模型原始回答完整保留，供之後追查或人工複核。
The model's raw answers are retained in full for later investigation or manual review.
11 / ENGINEERING CHALLENGES

工程挑戰的核心：如何讓分數可信

The core engineering challenge: making a score trustworthy

以下三個案例最能呈現這份實習的研究工程性質：這三個案例的根源都在實驗設計，會直接影響分數能不能被解讀，不只是程式寫錯這麼簡單。

The three cases below best capture the research-engineering character of this internship. All three are rooted in experimental design and directly affect whether a score can be interpreted at all — they are not merely coding mistakes.

CHALLENGE 01 · MEASUREMENT
兩套程式跑同一模型，為什麼會得到不同分數？
Two codebases, one model — why different scores?
→ OFAT ABLATION（詳見 03 章）→ OFAT ABLATION (see section 03)
Problem
短 prompt、後處理與 scorer 定義不同，讓「模型能力」和「評測流程效果」混在一起。
Differing short prompts, post-processing and scorer definitions mixed "model capability" together with "evaluation-pipeline effects".
Action
先對齊共同 baseline，再一次只改一個變因；7 組 variant，其中 5 組重新推論。
Align both onto a shared baseline first, then change one factor at a time: 7 variants, 5 of which required fresh inference.
Outcome
證實單一設定即可讓 EM 產生巨大變化，後續系統因此把 effective config 當成實驗身分的一部分。
Proved that a single setting can swing EM enormously, which is why later systems treat the effective config as part of an experiment's identity.
CHALLENGE 02 · OFFICIAL EVALUATOR
模型復讀讓 OmniDocBench matching 階段卡死
Model repetition hangs the OmniDocBench matching stage
→ DEBUG UPSTREAM
Problem
小模型約有三成輸出出現長篇復讀，官方 matching 流程連長輸入備援機制也可能一起卡住。
Roughly 30% of small-model outputs run into long repetitions, and the official matching stage can hang along with its long-input fallback.
Action
將參數 temperature 從 0 調整成 1，發現可大幅度減少長篇復讀的問題。同時保存 raw output、對照官方 issue #228，將 match_workers 由 4 降至 1；並以 smoke test 驗證生成設定。
Raising temperature from 0 to 1 substantially reduced the long-repetition problem. We also retained the raw output, followed upstream issue #228 to lower match_workers from 4 to 1, and validated the generation settings with a smoke test.
Outcome
evaluator 得以穩定完成；模型復讀仍保留為模型失敗證據，而不是用事後清洗把問題藏掉。
The evaluator now completes reliably, and model repetition is kept as evidence of model failure rather than scrubbed away after the fact.
CHALLENGE 03 · PROVENANCE
同名模型、不同版本與第三方量化，結果還能算同一個模型嗎？
Same model name, different version, third-party quantisation — is it still the same model?
→ TRACE EVERYTHING
Problem
模型 tag、資料 revision 或 evaluator commit 只要悄悄改變，舊結果和新結果就可能被錯誤混用。
If a model tag, dataset revision or evaluator commit changes quietly, old and new results can end up wrongly mixed together.
Action
resume key / run signature 納入 model digest、dataset fingerprint、prompt hash 與 dependency lock；第三方 GGUF 另存來源與風險。
The resume key and run signature incorporate model digest, dataset fingerprint, prompt hash and dependency lock; third-party GGUF builds have their source and risks recorded separately.
Outcome
任一關鍵設定不同即視為新實驗，不與舊資料混跑；每一個分數都能沿 metadata 回溯到實際執行條件。
Any difference in a key setting counts as a new experiment and never runs against old data, and every score can be traced back through metadata to the conditions it was actually produced under.
12 / DELIVERABLES & REFLECTION

實習結束後留下的成果

What the internship leaves behind

01 · EVALUATION PIPELINES
可重跑的評測工具
Re-runnable evaluation tooling
eval/、ablation_experiment/、benchmark_suite/、TC-STR_and_OminDoc/，涵蓋短文字 OCR、整頁解析與多模型比較。
eval/, ablation_experiment/, benchmark_suite/ and TC-STR_and_OminDoc/, covering short-text OCR, full-page parsing and multi-model comparison.
02 · REPRODUCIBILITY METADATA
讓結果具備來源證明
Results that carry proof of provenance
run manifest、dataset manifest、model digest、prompt hash、version pinning、protocol exceptions，避免「同名但不同實驗」被混為一談。
Run manifests, dataset manifests, model digests, prompt hashes, version pinning and protocol exceptions, so that experiments sharing a name but differing in substance are never conflated.
03 · EVIDENCE & REPORTS
原始證據與互動式成果
Raw evidence and interactive output
raw response、逐筆 CSV／JSON／SQLite、消融結果與自包含 HTML 報告，讓結論可以被人工複核，而不是只剩最後一個百分比。
Raw responses, per-item CSV / JSON / SQLite, ablation results and self-contained HTML reports, so conclusions can be checked by a human instead of collapsing into one final percentage.
04 · HANDOFF DOCUMENTATION
可交接的研究文件
Handover-ready research documentation
方法論、指標、版本釘選、模型來源、執行環境與例外紀錄被寫成文件，降低研究只存在於原作者記憶中的風險。
Methodology, metrics, version pins, model provenance, runtime environment and exception records are all written down, reducing the risk that the research exists only in its original authors' memories.
WHAT THIS INTERNSHIP TAUGHT US
Benchmark 分數是整條 pipeline 的產物
模型、prompt、generation options、後處理與 scorer 都可能改變結果。比較模型之前，先確認測量方式是否對齊。
可重現性不是報告附錄
長時間 benchmark 必須從一開始就設計 resume、版本身分、異常紀錄與原始輸出保存，而不是跑完再補文件。
失敗案例本身就是研究資料
復讀、截斷、metadata 缺失與 evaluator hang 不該被靜默清除；留下它們，才能知道系統真正的使用邊界。
A benchmark score is the product of the whole pipeline
Model, prompt, generation options, post-processing and scorer can all move the result. Before comparing models, confirm that the measurement is aligned.
Reproducibility is not an appendix
A long-running benchmark has to be designed from the start with resume, version identity, anomaly logging and raw-output retention — not documented after the run is over.
Failure cases are research data in their own right
Repetition, truncation, missing metadata and evaluator hangs should not be silently cleared away. Keeping them is how you learn where the system's real limits are.
後續事後驗證：六個假說的結果（詳見 10.1／10.2 各圖表下方的「事後驗證」區塊）
Follow-up post-hoc checks: results for six hypotheses (see the "post-hoc check" blocks below the charts in 10.1 and 10.2)
—六項假說沒有一項能被完整證實：根本原因是 TC_STR bench 與 OmniDocBench 的逐題／逐頁原始輸出從未提交到 repo，只留下彙總分數——這也印證了上方「可重現性不是報告附錄」這一課。
—最重要的修正：Gemma 26B A4B「以小勝大」只在 TC-STR 短文字任務成立，換到 OmniDocBench 整頁解析反而輸 31B 8.5 分，不是穩定的 MoE 優勢，而是任務依賴的個案。
—Kimi-VL 的不穩定原本被歸咎於 MoE 架構，反向證據推翻了這個猜測，改指向第三方非官方量化版本這條更有根據的線索。
—GLM-4.6V-Flash 勝過 Gemma 31B 已排除計分不公平造成假象，但「為什麼」在架構或訓練資料層面仍是未解之謎，留待補齊逐題資料後再驗證。
—Not one of the six hypotheses could be fully confirmed. The root cause is that the per-item and per-page raw outputs of the TC_STR bench and OmniDocBench were never committed to the repository — only the aggregate scores were. Which rather proves the lesson above: reproducibility is not an appendix.
—The most important correction: Gemma 26B A4B's "small beats large" holds only on the TC-STR short-text task. On OmniDocBench full-page parsing it loses to the 31B by 8.5 points. This is a task-dependent one-off, not a durable MoE advantage.
—Kimi-VL's instability was originally blamed on the MoE architecture. Contrary evidence overturned that guess and pointed instead to the better-grounded lead of a third-party, unofficial quantisation.
—GLM-4.6V-Flash beating Gemma 31B is not a scoring artefact — that much is ruled out — but why it wins, at the level of architecture or training data, remains unsolved, pending the per-item data needed to check it.
查看完整 GitHub 專案 ↗ 查看完整分工與工作紀錄 ↗
Browse the full GitHub project ↗ See the full division of work and work log ↗
13 / SUMMARY

研究限制、下一步與結語

Limitations, next steps and closing remarks

這份專案的核心價值，是建立一套嚴謹、可重現、可稽核的評測方法；「哪個模型比較強」只是過程中的副產品。做法是：固定資料、提示詞與推論設定，搭配雜湊指紋、版本鎖定、檢查點與例外紀錄，使每一次分數都能回溯到具體條件。這無法消除所有 benchmark 偏差，但能讓本次模型比較中的非模型因素更透明，也讓後續研究者知道哪些結果可以比較、哪些必須保留限制。

The core value of this project is a rigorous, reproducible, auditable evaluation method; which model is stronger is a by-product along the way. The approach: fix the data, the prompts and the generation settings, and pair them with hash fingerprints, version pinning, checkpoints and exception records, so that every score can be traced back to the specific conditions that produced it. This cannot eliminate every benchmark bias, but it does make the non-model factors in this comparison more transparent, and it tells later researchers which results are comparable and which come with caveats attached.

研究限制
Limitations
GLM OCR BF16 需搭配例外規則解讀
因 completion metadata 缺失採用放寬計分，EM／CM 落差與截斷率並非與其他 7 個模型完全同等條件。
部分模型為社群轉換版本
InternVL3.5（4B／38B）是社群自行轉換的 GGUF 量化版，效能可能與官方原生版本有落差。
MoE 的「實際啟用參數」是估算值
Gemma 4 26B A4B、Kimi-VL-A3B-Instruct 標注的啟用參數為概念性估算，非官方精確數字。
抽樣報告與全語料庫結果不可互換
互動式比較報告只挑最佳／最差／隨機各 5 頁（共 15 頁）；本頁採用全語料庫 1,651 頁官方分數。
跨環境重現性有其極限
即使固定 temperature=0、鎖定 digest 與資料版本，換了 Ollama 版本、驅動或硬體仍不保證逐字元一致。
GLM OCR BF16 must be read alongside its exception rule
Because completion metadata was missing, relaxed scoring was applied, so its EM/CM gap and truncation rate are not on exactly the same footing as the other seven models.
Some models are community conversions
InternVL3.5 (4B / 38B) are community-converted GGUF quantisations, whose performance may differ from the official native builds.
MoE "active parameter" figures are estimates
The active-parameter counts given for Gemma 4 26B A4B and Kimi-VL-A3B-Instruct are conceptual estimates, not official exact figures.
Sampled reports and full-corpus results are not interchangeable
The interactive comparison report takes only 5 best, 5 worst and 5 random pages (15 in total); this page uses the official scores over the full 1,651-page corpus.
Cross-environment reproducibility has limits
Even with temperature=0 fixed and digests and dataset versions pinned, a change of Ollama version, driver or hardware still does not guarantee character-identical output.
未來展望
Future work
GLM OCR BF16 異常樣本錯誤分析
逐題檢視 2,034 筆截斷樣本，釐清哪類文字（長句、密集小字、特殊符號）容易誘發復讀。
弱項子指標細部分析
針對小模型的 table TEDS，找出是哪類版面（跨頁、合併儲存格、多欄）造成失分。
repeat_penalty 細部 sweep 與交互作用驗證
03 章的消融實驗已證明 1.6 明顯有害，但尚未定位最佳值；也尚未檢驗 prompt、後處理與 num_predict 之間的交互作用，OFAT 無法捕捉這些交互效應。
整合成可持續更新的 dashboard
把互動式比較報告與靜態圖表整合，新模型跑完就能自動併入比較。
Error analysis of GLM OCR BF16's anomalous samples
Review the 2,034 truncated samples item by item to establish which kinds of text — long sentences, dense small print, unusual symbols — most readily trigger repetition.
Detailed analysis of the weakest sub-metrics
For small models' table TEDS, identify which layouts (page-spanning, merged cells, multi-column) are losing the points.
A finer repeat_penalty sweep and interaction testing
The ablation in section 03 showed 1.6 to be clearly harmful but did not locate the optimum, and the interactions among prompt, post-processing and num_predict remain untested — OFAT cannot capture those interaction effects.
Consolidation into a continuously updated dashboard
Merge the interactive comparison report with the static charts, so a newly evaluated model folds into the comparison automatically.
14 / ACKNOWLEDGEMENTS

致謝

Acknowledgements

［佔位符：致謝內容待補— 感謝在這次實習期間提供指導、協助與資源支持的師長、夥伴與單位。］

[Placeholder: acknowledgements to be completed — with thanks to the mentors, colleagues and organisations who provided guidance, help and resources over the course of this internship.]

This English edition was translated from the Traditional Chinese original by Claude Opus 5 (High reasoning effort). Numbers, model names, identifiers and code paths are carried over verbatim; where any wording diverges, the Traditional Chinese version is authoritative.
OCF 2026 AI 研究實習 · VLM OCR 評測成果 OCF 2026 AI Research Internship · VLM OCR benchmark results ocftw/2026-OCF-Intern ↗