The vLLM team published GLM-5.3 Optimizations, Part 2 on October 23, 2026. Part 1 kept long contexts decoding on one 8x H200 node. Part 2 goes to two nodes, splits prefill from decode, and stacks three techniques: prefill context parallelism (PCP), decode context parallelism (DCP), and Hybrid HiSparse on the decode side.

The headline is a 1M-token context served on H200s, which is a generation-old part with 141 GB of HBM per GPU. More useful to me is why the usual knobs fail here, since it applies to any MLA model with a sparse indexer. Every number below is from the post unless I say it's my arithmetic.

1. Why tensor parallelism is the wrong split for sparse MLA

GLM-5.3 uses multi-head latent attention (MLA) with a sparse top-k: an indexer scores every earlier token and sparse MLA attends to the 2,048 it picks. The post walks through the two standard ways to run prefill on an 8x H200 node.

  • DP8 puts each prompt on one GPU. The other seven don't help with that prompt's attention, so time to first token (TTFT) on long prompts suffers, and the whole prompt's KV has to fit on that one GPU.
  • TP8 splits the 64 attention heads eight ways. But MLA's latent KV is shared by all 64 heads, so every GPU stores the full cache and reads it to serve only 8 heads. The indexer isn't partitioned at all, so its quadratic cost repeats on every GPU.

That second point is the one I'd underline for anyone who reaches for TP by habit. With MLA, TP gives you no extra KV room and wastes memory bandwidth. The post's footnote puts numbers on it: each selected KV row is 660 bytes (656 bytes of FP8 KV in FlashMLA's fp8_ds_mla format plus a 4-byte index), and serving 8 heads per row gives about 26 FLOPs per byte moved. Serving all 64 heads per row gives about 191. The H200 SXM's ridge point, using the dense 989 BF16 TFLOP/s and 4.8 TB/s, is about 206 FLOPs per byte. So TP8 leaves sparse MLA deep in memory-bound territory, while serving every head per row gets it close to compute-bound.

2. PCP: split the prompt by token, and balance the causal mask

PCP gives each GPU one eighth of the prompt's query tokens with all 64 heads. A naive contiguous split would be lopsided, because under causal attention the GPU holding the end of the prompt does far more work than the one holding the start. vLLM cuts the prompt into 16 chunks and gives GPU g chunk g and chunk 15 minus g, so each GPU gets one cheap chunk and one expensive one. That's the load-balanced sharding from the Context Parallelism for Scalable Million-Token Inference paper. New KV is all-gathered each layer so every query still sees the full context.

Two things fall out of this. Each KV row now serves 64 heads, which is where the 26 to 191 FLOPs-per-byte jump comes from. And each GPU indexes only its own tokens, so the indexer stops being replicated. The post counts prefill FLOPs and finds the indexer overtakes sparse MLA under TP8 once a prompt reaches about 32K tokens. Under PCP8 that crossover moves out to about 256K.

3. DCP: PCP alone still replicates the cache

PCP has a catch. Because new KV is all-gathered every layer, every GPU still ends up holding the full KV cache. That replicated cache, not compute, caps context length.

The post's KV math is worth copying into your own capacity sheet. Per token, GLM-5.3 stores 78 layers at 656 bytes of MLA KV, 21 indexer layers at 132 bytes (128 FP8 values plus a 4-byte scale), and 788 bytes for the MTP layer, for 54,728 bytes total. At 1,048,576 tokens that's 53.4 GiB of FP8 KV for one request. After weights, each H200 has room for about 15 GiB of KV.

My arithmetic from the same per-token figure: a 256K-token request needs about 13.4 GiB, which is right at that 15 GiB budget. That matches the post's statement that past roughly 250K tokens, PCP8 alone can't hold the KV cache.

DCP fixes this by dealing tokens round-robin across the eight GPUs, so token i lives on GPU i mod 8. The 1M-token request drops to 6.7 GiB per GPU. In this prefill engine it works the other way: each layer, every GPU all-gathers the shards into a temporary full-context copy, attends for its own queries, then frees the copy. Only the persistent cache is divided by eight.

The post is direct that this isn't free. On one PCP8 node, running a 2K-token chunk with and without DCP8 at equal cached context, DCP8 adds about 6% per step up to 64K tokens of context and about 15% at 248K. The team says it's working on overlapping each layer's all-gather with the previous layer's compute. Until that lands, I'd treat DCP as something you turn on when your context limit requires it, not by default for short-context traffic.

4. The deployment, spelled out

Two 8x H200 nodes, both with FP8 KV cache and MTP speculative decoding at three draft tokens (MTP3):

  • Prefill: one PCP8 x DCP8 x EP8 engine with 512 GiB of CPU KV offload, so later turns reuse their prefix.
  • Decode: eight DP ranks with EP8 and Hybrid HiSparse, each with a 48 GiB host pool plus 16 GiB of offload, so decode concurrency is bounded by host memory rather than HBM.

A prefix-aware llm-d router sends each request to prefill, the NIXL connector moves its KV to a decode rank, and the decode rank streams tokens back. Baselines use the same 16 GPUs, 512 GiB of host memory and vLLM build: DP8 prefill, and plain CPU offload instead of HiSparse on decode.

The less glamorous part is the KV handoff. Prefill holds each request's KV sharded by token across eight GPUs, while each decode rank needs the whole thing, including the MTP layer's KV, in its own paged layout. NIXL does that reassembly between nodes, and prefix reuse from CPU offload has to keep working alongside it. If you run a different connector, this is the piece I'd expect to break first.

5. The results, and how much weight to put on them

The benchmark is SemiAnalysis AgentX: Claude Code traces replayed with prefix reuse, a fresh server per concurrency, one-hour runs. Throughput is logical tokens per second per GPU including prefix-cached input. Interactivity is AgentX's P90 end-to-end-normalized output tokens per second per user.

  • Up to concurrency 32 the deployment is mostly prefill-bound. With HiSparse decode on both sides, PCP8 x DCP8 prefill gives 1.7 to 3.2 times the interactivity of DP8 at the same throughput, and median TTFT drops from 1.5 to 5.2 seconds down to 0.8 to 0.9 seconds.
  • At c8 and c16 the decoder choice barely matters, and offload-only decode is slightly ahead at c8.
  • From c32 the deployment becomes decode-bound. At c64, HiSparse decode reaches 8,340 logical tok/s/GPU at 29.6 tok/s/user, while every other configuration drops to 1 to 2 tok/s/user.
  • At 1M context, only the PCP8 x DCP8 plus HiSparse combination works. A DP8 prefill rank can't hold one 1M-token request, and neither can an offload-only decode rank.

The caveat I'd keep in front of anyone quoting these numbers: MTP3 ran with a synthetic acceptance length fixed at 2.99, so every configuration got the same speculative speedup, and your real acceptance rate will move the absolute numbers. Throughput also counts cached prefix tokens, which flatters any workload with heavy prefix reuse. Agentic coding traces are exactly that kind of workload, so I'd compare these numbers only against other AgentX runs.

6. Trying it

The post says everything runs on vLLM v0.32.0 or later (vllm/vllm-openai:v0.32.0). Results were measured on vLLM main at 9e38b4111a, and the reproduction gist pins 2537629eb2. To match the numbers, use the gist's commit. I covered what else changed in that release in my vLLM 0.32 upgrade notes, and the router side in llm-d 1.0 upgrade notes.

The prefill engine's key flags:

--prefill-context-parallel-size 8 \
--decode-context-parallel-size 8 \
--dcp-comm-backend ag_rs \
--enable-expert-parallel \
--attention-config '{"sparse_mla_force_mqa":true}'

Don't drop the last one. Without sparse_mla_force_mqa, prefill batches whose prompts fit within the 2,048-token top-k take the dense MHA path. The flag keeps them on the sparse-MLA (MQA) kernel, so every KV row serves all heads.

What I'd take from this

  1. For MLA models, budget KV per request from the per-token byte count, not from head count. TP won't give you more room.
  2. If prompts regularly pass 32K tokens, profile the indexer on its own. Under TP it repeats on every GPU.
  3. Turn on DCP when your context limit needs it, and expect 6% to 15% extra per prefill step until the all-gather overlap ships.
  4. Size decode host memory deliberately. With HiSparse, host RAM is what caps decode concurrency.
  5. Re-run with your own MTP acceptance rate before you promise anyone these per-user speeds.

Sources

Anpoo Sivanadi, Staff Software Engineer