Dory1.0-4B, measured from the outside

Each row pools independent runs with fresh seeds; ± is the sampling standard error. GPQA pools every Dory run at that loop count: experiment 2, the showdown and, at 2 loops, three control runs. IFBench (NVIDIA's metric, the mean fraction of instructions followed, strict) and AA-LCR are experiment 2's runs.

Qwen3.5-9B. Dory ahead on: competition math 88.8 vs 79.7 (+9.1 ± 2.3), IFBench 79.1 vs 66.7 (+12.4 ± 2.4), IFNovel 93.3 vs 59.3 (+34.0 ± 2.9), AA-LCR 74.0 vs 63.0 (+11.0 ± 5.3). Behind on: MMLU-Pro 77.7 vs 83.1 (-5.4 ± 0.7), GPQA 73.4 vs 82.2 (-8.8 ± 2.1), SimpleQA 3.4 vs 8.1 (-4.7 ± 0.4), MRCR 39.0 vs 96.3 (-57.2 ± 3.0). Not resolved by these sets: LiveCodeBench 68.4 vs 63.2 (+5.2 ± 2.6), IFEval 93.3 vs 90.6 (+2.8 ± 1.4).

Gemma 4 26B-A4B. Dory ahead on: competition math 88.8 vs 83.1 (+5.7 ± 2.3), IFNovel 93.3 vs 85.0 (+8.3 ± 2.3), AA-LCR 74.0 vs 63.0 (+11.0 ± 5.3). Behind on: MMLU-Pro 77.7 vs 85.8 (-8.1 ± 0.7), GPQA 73.4 vs 79.7 (-6.3 ± 2.2), SimpleQA 3.4 vs 7.5 (-4.1 ± 0.4), LiveCodeBench 68.4 vs 74.7 (-6.3 ± 2.5), MRCR 39.0 vs 52.0 (-13.0 ± 3.7). Not resolved by these sets: IFEval 93.3 vs 94.6 (-1.3 ± 1.2), IFBench 79.1 vs 79.1 (+0.0 ± 2.1).

In brackets: the difference, paired by question, ± one standard error; a difference beyond two standard errors counts. Scores are this run's (Dory's GPQA here is one run; every run pooled reads 73.2 at 4 loops).

All at Dory's 4-loop setting; each model at its maker's recommended sampling with reasoning on.

The showdown's raw IFBench answers, graded with AllenAI's verifiers at commit 1c40f0c1 and NVIDIA's metric. ‘As returned’ keeps the whitespace that follows the reasoning; ‘stripped’ removes it, as this harness does.

• vLLM: the Dory pull request, vllm-project/vllm#60149, at 55076950, installed over the precompiled cu130 wheel of main 153a1716 (0.32.0rc1.dev319), torch 2.14.0.
• Image: experiment 2 ran on Modal's debian_slim Python 3.12 image; the showdown on nvidia/cuda:13.0.1-devel-ubuntu24.04 with Python 3.12, because nvcc is needed on B200, where vLLM JIT-compiles FlashInfer's attention. Both set VLLM_USE_FLASHINFER_SAMPLER=0; the vLLM build is the same.
• Hardware: one GPU per job on Modal. Dense models ran on B200, falling back to H200 or H100. The MoE models (gpt-oss-20b, Qwen3.6-35B-A3B, Nemotron 3.5 Lightning, LFM2.5-24B-A2B, GLM-4.7-Flash) ran on H200 with moe_backend="triton", because their B200 start-ups failed in this build. Gemma 4 26B-A4B ran on B200 and H200 with Triton MoE. Each configuration's GPUs are in the settings table.
• Weights: Dory nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 at b366b87f; every other model pinned to its Hugging Face main commit of Tue 20 Oct 2026 (table below).

git clone https://github.com/vllm-project/vllm && cd vllm
git fetch origin pull/60149/head && git checkout 55076950b6c155f384d6766d63563c3bc20a0e3f
VLLM_USE_PRECOMPILED=1 \
VLLM_PRECOMPILED_WHEEL_COMMIT=153a17166e5d07bc38184dedeea542d2e8885e55 \
VLLM_PRECOMPILED_WHEEL_VARIANT=cu130 pip install .

from vllm import LLM, SamplingParams
llm = LLM(model=DORY_PATH, max_model_len=73728,   # GPQA here; 69,632 in experiment 2, up to 196,608 for AA-LCR
          enable_prefix_caching=True, max_num_seqs=256,      # down to 4 for AA-LCR at 4 loops
          seed=zlib.crc32(run_tag.encode()) % 2**31,          # a fresh seed per launch = an independent draw
          hf_overrides={"n_recurrent_loops": k,              # 1, 2 or 4
                        "recurrent_kv_cache_mode": "per_loop"})
sp = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=65536)   # the card's sampling
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
text = llm.generate([prompt], sp)[0].outputs[0].text
answer = text.rsplit("</think>", 1)[1].strip() if "</think>" in text else ""   # unclosed = no answer = wrong

Output budget: 64K tokens for GPQA, competition math, LiveCodeBench and AA-LCR (AA-LCR up to what each model's context window leaves after the ~100K-token prompt); 32K for MMLU-Pro, SimpleQA, IFEval, IFBench, IFNovel and MRCR. A reply cut at the budget, or with its reasoning never closed, scores 0. Every model ran with reasoning on, at its maker's recommended sampling.

The Dory card names NeMo Gym as its harness. Gym has no Dory recipe, so experiment 2 ported its Super 3.5 recipes at the commits their published numbers were made with (Gym ae8d388d for GPQA and IFBench, 5a7dd6fe for AA-LCR) and ran them beside this harness:

• GPQA with Gym's prompt (aai/mcq-4choices) and its lenient extractor: 66.4 / 73.2 / 72.5 at 1 / 2 / 4 loops (2 repeats each, one fixed answer order per question, so position bias is in it too), against the card's 69.19 / 72.41 / 72.16.
• IFBench: the card's number is Gym's reward, the mean over prompts of the strict fraction of instructions followed, graded by AllenAI's verifiers on the final answer. This page reports the same metric on the same 300 prompts.
• AA-LCR: Gym's judge (Qwen3-235B-A22B-Instruct-2507, FP8) and our gpt-oss-120b judge, re-grading the same 12 complete answer sets: Gym's judge reads between 1 point lower and 2 points higher than ours (mean +0.4).

A score is the mean over questions of each question's mean over its repeats. Two error bars appear. Against the card, which measured the same questions, the yardstick is the sampling standard error: the run-to-run noise of repeats on those questions. Between loop counts or models, differences are paired by question: the spread of the per-question differences divided by √questions. Each launch is seeded from its run name, so repeats are independent draws; byte-identical texts are checked for before pooling. Long-context columns (AA-LCR, MRCR) and IFNovel are reported beside the index, not in it.

The same Dory job (198 GPQA prompts × 4,096 tokens, 1 loop, batch 256) on eight Modal GPU types. Decode-bound evaluation favours memory bandwidth, so the cheapest card per hour is the most expensive per token.

• IFBench metric: the card's number is NVIDIA's mean strict fraction, not IFBench's prompt-level loose; experiment 1 compared the wrong one.
• Repeats that weren't: vLLM seeded once per engine made separately launched AA-LCR “repeats” byte-identical copies; every launch now gets its own seed.
• Math answers: the boxed answer is chosen as MathArena's parser does (an exact box beats a later decimal approximation), and math-verify gets sums written out (an exact HMMT answer left as a double sum was graded wrong).
• LiveCodeBench: four test workers per 8-CPU grader (oversubscription had pushed correct solutions past the 6 s limit), time-limited records re-run alone, and a finished reply without a reasoning marker graded on its last code block as lcb_runner does.
• IFEval: Google's verifiers draw a random letter for two malformed rows and langdetect samples; both are now seeded, so re-grades agree.
• Equal budgets: MRCR kept at 32K for every model (grouping it with AA-LCR had given five models 64K), and context limits taken from each repo's config.

Seven review passes checked each benchmark's prompt, extraction and grading against the reference code and real outputs, and a sceptic tried to refute each finding; 17 of 37 held, grouped into the six fixes above.

• No agentic tasks. The card's Terminal-Bench, τ³-Bench, GDPval and SWE-bench columns are not run here, so this does not re-measure its Artificial Analysis index estimate.
• Humanity's Last Exam (a gated dataset), AA-Omniscience and SciCode are not run either.
• One draw of AA-LCR and MRCR per configuration in the showdown; AA-LCR draws of the same model differ by up to 11 points (experiment 2).
• Context windows decide long-context budgets. A 131K-context model has much less room to think about a ~100K-token AA-LCR prompt than a 262K one; Olmo 3 (65K) cannot run it at all.
• Some models exceed the output budgets: Ornith 1.0 9B on up to 68% of a benchmark's prompts; MiMo-V2.6 Distill 9B on up to 56% of a benchmark's prompts; Qwen3.5-4B on up to 43% of a benchmark's prompts; Qwen3.5-9B on up to 32% of a benchmark's prompts; Qwen3.6-35B-A3B on up to 29% of a benchmark's prompts; LFM2.5-2.6B on up to 28% of a benchmark's prompts; GLM-4.7-Flash on up to 24% of a benchmark's prompts; Gemma 4 26B-A4B on up to 24% of a benchmark's prompts; gpt-oss-20b (high) on up to 23% of a benchmark's prompts; Jamba2-3B on up to 18% of a benchmark's prompts; LFM2.5-24B-A2B on up to 15% of a benchmark's prompts; Olmo 3 7B Think on up to 12% of a benchmark's prompts. A cut answer scores 0, so their scores there partly measure that.
• Vendor sampling differs: Liquid recommends temperature 0.1 for its LFM2.5 models, which ran that way; a control of LFM2.5-2.6B at temperature 1.0 is in the full report.