Dory1.0-4B: What a Looped 4B Model Actually Costs Per Token

NVIDIA put Nemotron-Labs Dory1.0-4B on Hugging Face on October 16 under the OpenMDW-1.1 license. It is a looped, or latent-recurrent, language model: a block of 20 layers runs one to four times per token, and you pick the loop count at serve time. NVIDIA's own table has the four-loop setting scoring 61.8 on SWE-bench Verified, next to 61 for Qwen3.5-9B under the same harness.

The headline reads like "a 9B model's results from a 4B model." For anyone who has to serve it, that framing hides where the cost goes. I read the model card, config.json, and the safetensors header to work out what each loop count costs in FLOPs, weight traffic, and KV cache. The short version is that looping saves memory capacity, not compute, and the default serve command gives back part of that saving.

What's in the checkpoint

From the model card, the config, and the tensor shapes in model.safetensors:

The recurrent block holds 1.84B of the 3.98B parameters. The input and output stacks are 0.73B each, and the embedding table and LM head are 0.34B each.

The backbone is a normalized transformer (nGPT), which keeps hidden states on the unit hypersphere. Reapplying the same 20 layers to a state over and over is a good way to blow up activations, so a normalized backbone is a sensible choice for a looped model. The re-injection of the input before each pass follows the recurrent-depth design from Geiping et al. last year. What's new here is a production-trained, RL-tuned checkpoint with quantized variants on day one, which takes the idea past the research-prototype stage.

Compute and weight traffic per token

Each loop adds 20 blocks to the path. One loop runs 36 blocks per token, two loops run 56, and four loops run 96. Using the usual estimate of two FLOPs per weight per token, and counting the LM head but not the attention-score math:

These are my estimates from the tensor shapes, not NVIDIA's numbers.

Four loops cost about as much arithmetic as a dense 9B model, which is about 18 GFLOP per token by the same estimate. So if Qwen3.5-9B is dense, the SWE-bench comparison with it is much closer to like-for-like compute than the "4B" label suggests. Looping didn't make the model cheaper to run at that quality level. It made it smaller to store.

At batch 1, decode is bound by memory bandwidth, and the shared weights don't help there either. The recurrent block is 3.67 GB in BF16, and about 1 GB even in NVFP4 at roughly 4.5 bits per weight. Neither fits in any GPU's L2, so every loop streams the block from DRAM again. On a single stream, four loops move about 2.5 times the bytes of one loop.

The measured numbers on the card fit this picture. On DGX Spark at batch 1, with 2,560-in/300-out and 128-in/1,024-out requests, NVFP4 runs 2.17 to 2.44 times faster than BF16, and MXFP8 1.69 to 1.84 times faster. That is what you'd expect when the job is moving weights. The card doesn't say which loop count those runs used, so I can't turn them into tokens per second per loop.

Where looping does pay off is capacity. The whole model is 7.95 GB in BF16, and the NVFP4 version is much smaller. That leaves more of an 80 GB H100, a 24 GB consumer card, or DGX Spark's 128 GB of unified memory for KV cache and batch. On a device where a 9B model barely fits, Dory with four loops fits easily and spends the saved memory on context.

The KV cache depends on a flag

This is the part I'd check before deploying. Each full-attention layer stores 4 KiB of K and V per token in BF16 (2 × 4 KV heads × 256 × 2 bytes). Of the 9 full-attention blocks, 2 are in the input stack, 2 in the output stack, and 5 inside the recurrent block.

Dory has two cache modes:

• per_loop keeps a separate cache for each pass through the recurrent block, so the 5 recurrent full-attention layers are paid for once per loop.
• last_loop keeps one cache per recurrent layer, and each loop overwrites the previous one, so the cache is the same size as one loop's cache no matter how many loops you run.

Here is what that means per token and at full context. This is my arithmetic and covers only the full-attention layers. The sliding-window layers add a fixed 108 to 288 MiB per sequence depending on mode and loop count.

For one 256K session, the cache is larger than the model, even in last_loop, so the KV setting matters more than the weight format for capacity planning.

The vLLM quick start on the model card uses "n_recurrent_loops": 2 with "recurrent_kv_cache_mode": "per_loop" and --max-num-seqs 128. Take 32K-token sessions on one 80 GB GPU with roughly 65 GiB left for cache after weights and workspace (my assumption). last_loop fits about 57 such sessions, two-loop per_loop about 37, and four-loop per_loop about 21. So the default costs about a third of your concurrency compared with last_loop, and four loops in per_loop mode costs nearly two-thirds.

NVIDIA says accuracy with last_loop "stays close to the per-loop mode with nearly unchanged output lengths," but it doesn't publish those numbers. Under last_loop, a later loop's query attends to earlier tokens' final-loop keys and values rather than the matching loop's, so it's an approximation. I'd run my own evals in both modes before switching a fleet over.

What looping does and doesn't buy on the benchmarks

All the scores below are NVIDIA's own, measured under one harness, and I haven't seen independent reproductions yet.

• Coding and agentic tasks gain the most. SWE-bench Verified goes from 50.3 at one loop to 59.5 at two and 61.8 at four. SciCode goes from 27.3 to 34.94 to 35.11. Terminal-Bench 2.1 goes from 12.5 to 22.2, which is still below the 9B's 24.
• Most of the gain arrives with the second loop. GPQA Diamond goes from 69.19 to 72.41 at two loops and then slips to 72.16 at four. HLE does the same, going from 8.25 to 10.89 and then down to 10.43.
• Knowledge doesn't move. AA-Omniscience accuracy is 10.03, 10.08, and 10.03 across the three settings, below Qwen3.5-4B's 15.02. That makes sense, because looping adds compute per token, not stored facts. If your workload is recall-heavy, more loops won't help and a bigger model will.
• Instruction following and long context look strong. IFBench reaches 79 and AA-LCR reaches 73 at four loops, the highest in the table on both.

The card also claims looping cuts the number of generated tokens, because more reasoning happens inside the loops. It shows this as a chart without a table. If it holds on your tasks, the end-to-end cost of four loops can be less than the 2.5 times per-token figure. Measure that before you dismiss the higher loop counts.

Serving notes

• Loop count is set per deployment, not per request. In vLLM it goes in --hf-overrides at launch, and in llama.cpp it's dory.num_loops in --override-kv. "One checkpoint, several compute budgets" means one file on disk with separate pools today, unless the engine adds per-request control.
• Prefix caches won't carry across loop counts or cache modes. The cached keys and values differ between settings, so a one-loop pool and a four-loop pool can't share cached prompts. That's my reading of the design, not something the card states.
• Support isn't upstream yet. vLLM support lives in vllm-project/vllm#60149, open since October 6. vLLM 0.32.0 shipped overnight without it, so for now you install from the PR branch. llama.cpp support is in ggml-org/llama.cpp#30306, also unmerged. The card's Mac recipe pins a fork commit and a Q5_K_M GGUF of 3,673,288,864 bytes.
• Prefill scales with loops too. Long prompts go through every loop, so time to first token on a 100K-token prompt grows roughly with blocks per token. That's 1.5 times the one-loop cost at two loops and 2.5 times at four, by the FLOP estimate above.

Where I'd use it

Dory makes sense where memory is the constraint and compute isn't, such as a local coding agent on a 24 GB card or DGX Spark, or a GPU you're filling with long-context sessions rather than raw throughput. In those places, two loops with last_loop caching is the setting I'd start from. It gets most of the accuracy gain for 1.5 times the one-loop compute and no extra KV cost.

Where compute is the constraint, as on a busy H100 pool serving short requests at high concurrency, the math is less kind. Four loops cost about as much as a dense 9B per token, and you'd want the token-reduction claim proven on your own traffic before choosing it over a model that just has more parameters.

The more interesting long-term result might be the training note. Pre-training the first ~4.5T tokens natively in NVFP4 and still shipping a competitive model is a useful data point for anyone budgeting Blackwell training runs, whether or not they care about looping.

Related reading: Nemotron 3.5 Super VL and Bolt, Ling-3.1-flash and the hybrid KV budget, Granite Switch 4.1 and shared KV, RTX Spark N1X.

Anpoo Sivanadi, Staff Software Engineer