Ling-3.1-flash: What 63 Linear-Attention Layers Do to the KV Budget

inclusionAI, the Ant Group team behind the Ling, Ring and Ming model families, put the weights for Ling-3.1-flash on Hugging Face this week under the MIT license. It is a 560B-parameter mixture-of-experts model with about 25B parameters active per token. That size class is crowded now. What made me open the config file is the attention stack: 63 of its 72 layers do not keep a KV cache at all.

For long-running agent sessions, most of the GPU memory that isn't weights goes to KV cache. So this post is mostly arithmetic from the published config.json, plus what the day-zero SGLang and vLLM recipes say about running it.

The layer layout

The model is built from nine groups of eight layers. In each group, seven layers use Kimi Delta Attention (KDA), the gated delta-rule linear attention Moonshot introduced with Kimi Linear, and one layer uses gated Multi-head Latent Attention (MLA), the compressed attention DeepSeek made common. That gives 63 KDA layers and 9 MLA layers.

The rest of the specs from the model card and config:

Two numbers on that list disagree with each other, and both matter for capacity planning. The card says 1M tokens of context, but the shipped config sets max_position_embeddings to 262,144 with no RoPE scaling, and the SGLang cookbook and vLLM recipe both use 262,144. I would plan around 262K until inclusionAI ships a config that says otherwise. The cookbook also lists about 17B active parameters against the card's 25B. My own rough count from the config lands closer to 25B once you include the KDA projections and the embedding and output matrices; the KDA layers carry full 4,096-to-8,192 projections for queries, keys, values and gates, and with 63 of them, attention weights are a large share of what each token touches.

The memory math

Here is the part I care about. An MLA layer caches a compressed latent per token instead of full keys and values: 512 values for the latent plus 64 for the RoPE key, so 576 values. At BF16 that is 1,152 bytes per token per layer.

With 9 MLA layers, Ling-3.1-flash grows by 10,368 bytes per token, about 10 KiB. Per sequence:

• 32K tokens: about 0.32 GiB
• 128K tokens: about 1.27 GiB
• 262K tokens: about 2.53 GiB
• 1M tokens, if the longer context ever ships: about 10.1 GiB

If all 72 layers were MLA, those numbers would be eight times larger: about 20 GiB per sequence at 262K and 81 GiB at 1M. That is the headline benefit, and it is real. An FP8 KV cache would halve both columns.

The KDA layers are not free, though. Each one keeps a fixed recurrent state per sequence: for 64 heads with 128-dimensional keys and values, that is 64 × 128 × 128, or about 1.05M values per layer. Across 63 layers that is 252 MiB per sequence if the state is held in FP32, which is common for numerical stability in delta-rule kernels, or 126 MiB in BF16. The short convolution in front of each KDA layer (kernel size 4) adds under 10 MiB more. These are my estimates from the config, not figures inclusionAI published, and the exact state dtype depends on the engine.

That fixed cost changes how you think about the hybrid. Compared with a hypothetical version where those 63 layers were MLA too, the KDA state only comes out ahead once a sequence passes roughly 3,600 tokens at FP32 state (about 1,800 at BF16). For short chat requests, the hybrid holds more memory per sequence than a pure-MLA model would. For agent sessions that sit at 50K to 200K tokens of accumulated transcript, it holds far less. This model is clearly tuned for the second case, which matches how it is marketed and how its benchmarks were run.

One more wrinkle for tensor-parallel deployments. With plain TP, MLA's shared latent cache is typically replicated on every rank, because every head reads the same latent. KDA state, by contrast, splits by head, and the cookbook notes that BailingKDA shards its projections on attention TP. So the part of the cache that grows with context is also the part that does not shrink when you add GPUs, unless you turn on data-parallel attention.

Fitting it on a node

The checkpoint is about 572 GB of FP8 weights, and the vLLM recipe gives a loading estimate of about 689 GB of aggregate VRAM before any cache. The reference deployments are eight H200s (8 × 141 GB = 1,128 GB) with TP8 and EP8, or four GB300s (4 × 288 GB = 1,152 GB) with TP4 and EP4. That leaves on the order of 400 GB per node for cache, state, activations and CUDA graphs. At roughly 2.8 GiB per 262K-token session including KDA state, the memory side of a long-context agent workload is unlikely to be what limits you; compute and scheduling will be.

The measured numbers in the SGLang cookbook are batch-1 only:

• 8 × H200, TP8 + EP8, NEXTN speculation plus the two Hopper decode flags (SGLANG_ENABLE_FP8_BLOCK_GEMV=1, SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=1): 251.2 output tokens/s
• 4 × GB300: 180.0 tokens/s, TPOT 5.14 ms, acceptance 2.59

MTP acceptance length was 2.61 on random tokens and 2.13 on 30 real-text prompts. I would size against the real-text number. There are no published concurrency curves yet beyond a reproduction command for a 32-concurrent run, so anyone planning a multi-tenant deployment will need to measure.

Where the hybrid makes serving harder

A plain KV cache is easy to prefix-cache: you hash blocks of tokens and reuse matching blocks. A recurrent state does not work that way. It is a single summary of everything before it, so you can only reuse it at positions where you saved a snapshot. Both engines handle this with hybrid-specific settings:

• vLLM runs the model from a branch with --mamba-cache-mode align, alongside --enable-prefix-caching, so state checkpoints line up with cache block boundaries.
• SGLang's multi-turn preset combines --mamba-radix-cache-strategy extra_buffer, --chunked-prefill-size 4096 and SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK=1, which caches the state both at the shared prefix fork and at the tail of each turn. In the cookbook's measured workload that saved about 576 tokens of prefill per later turn.
• SGLang's HiCache with Mooncake as an L3 store restored 5,952 of 6,000 tokens on every TP rank in their test, using a setting called hicache_storage_pass_prefix_keys for hybrid KDA prefix reuse.

There are also explicit gaps. Prefill context parallelism is disabled because BailingKDA does not implement the sequence gather and recurrent-state handling it needs. Prefill/decode disaggregation is exposed in SGLang's playground with Mooncake transfer, and the runtime has hybrid state transfer support, but the cookbook says this model's PD deployment has not been verified. If your long-context plan depends on splitting a 200K-token prefill across nodes, that is not available today.

My practical advice for anyone bringing it up: send a long prompt twice and confirm the second request reports cached tokens, then evict the cache, repeat the prompt, and confirm the answer is still correct. Hybrid state caching bugs tend to show up as quietly wrong output, not crashes.

The benchmarks, briefly

inclusionAI reports Terminal-Bench 2.1 at 81.18, SWE-bench Pro at 68.74, SWE-bench Multilingual at 88.5, SkillsBench v1.1 at 68.7, HLE at 37.44 and Terminal-Bench 4.0 at 46.03 on the model card. IndieRouter's write-up walks through the comparison chart against GPT-5.6 Sol, Claude Opus 5, Kimi K3, GLM 5.3 and DeepSeek-V4.1-Flash, where Ling lands first on HealthBench Professional and FrontierSWE and second on most of the rest. These are vendor numbers, and some agentic scores were run through LingCode, inclusionAI's own unreleased harness built on the open-source agent Pi. That makes those panels a comparison of model plus harness. I would wait for independent runs before ranking it.

Why I think this one matters

The interesting part of Ling-3.1-flash is not that it scores near the closed frontier at 25B active. It is that a 560B open model under an MIT license commits seven of every eight layers to linear attention, ships FP8 weights that fit one eight-GPU node, and arrives with day-zero engine support that already handles the hard part of caching recurrent state. The remaining gaps are specific and documented: context-parallel prefill, verified PD disaggregation, and the mismatch between a 1M-token card and a 262K config.

If you are running long agent sessions on your own hardware, this is the model I would benchmark against a pure-attention MoE of similar size, at your real context lengths, with the cache settings above turned on.

Related reading: Muse Spark Open: latent MoE and hybrid attention, Nemotron 3.5 Super VL and Bolt, ReplaySSM and hybrid-model decode, DeepSeek V4.1 Flash

Anpoo Sivanadi, Staff Software Engineer