vLLM's Two New KV Transfer Modes, and Why You Pick One

Two posts on the vLLM blog this week change how KV cache moves between prefill and decode workers in a disaggregated deployment. Both describe work that started on AWS Trainium clusters, was contributed to vLLM's NIXL connector, and was then measured on H200s. Both target time to first token (TTFT). They attack different parts of it, and for now the push post points you back to pull mode if you want bi-directional transfer.

• Push-based KV transfer (Oct 14) has the prefill side write KV into memory the decode side reserved in advance, instead of the decode side reading it after prefill finishes.
• Bi-directional KV transfer (Oct 12) lets the decode node hand a conversation's KV back to a prefill node on the next turn, so the prefill node stops recomputing history it already paid for.

I run GPU clusters, so I read both with one question: where does the time actually go? The short answer is that the wire is almost never the problem.

The test rig

Both posts use the same setup, and it's worth knowing before reading any of the numbers:

That is one prefill node feeding one decode node. Real fleets run many-to-many, and that matters for both features, as I'll get to below.

Push mode: the decode loop was the bottleneck

In the default pull mode, the sequence after prefill is serial. The prefill node finishes, the router passes block IDs to the decode node, the decode node allocates destination blocks, builds NIXL descriptors and issues an RDMA READ. Only then can decoding start.

Push mode reorders that. When the decode node allocates blocks for a request, it sends their locations to the prefill node right away, while prefill is still running. Whichever of the two events lands second, prefill finishing or the registration arriving, triggers an RDMA WRITE straight into the decode node's memory. Descriptor setup and submission run on a background thread on the prefill worker, off the engine's forward-pass loop. The two sides can use different block sizes, because each side maps blocks to its own layout at transfer time.

The numbers, on Qwen3-32B with prefix caching off on the decode side so every request does a full transfer:

At low load and short prompts the two modes are nearly identical. The gap only opens as QPS and prompt length rise.

To see why, I worked out how much data is actually moving. This is my own arithmetic from Qwen3-32B's config: 64 layers, 8 KV heads, head dim 128, BF16. That's 2 × 64 × 8 × 128 × 2 bytes, or 256 KiB of KV per token. A 16,384-token prompt is 4 GiB. With TP 8 and 8 KV heads, each GPU holds one head's worth, about 512 MiB. Each GPU has two 200 Gb/s rails, about 50 GB/s, so the pure wire time is roughly 11 ms.

So the pull-mode TTFT at 8 QPS is more than a second worse than push for a transfer that should take about 11 ms on the wire. The post says where that time goes: in pull mode the decode worker posts NIXL descriptors inline in its own engine loop, so setup for incoming requests stalls token generation for everything already running. That also explains why TPOT gets worse in pull mode under load, not just TTFT. The fix is about the scheduling, not the RDMA path.

The tradeoff is routing. Pull mode lets the router pick a decode instance after prefill finishes, when it knows the most about load. Push mode commits to a decode instance up front so registration can overlap with prefill. In a 1P1D benchmark that costs nothing. In a fleet with several decode pools and uneven request lengths, choosing early means choosing with less information. There's an open vLLM PR (#45211) that moves the background-thread approach into pull mode, which would get some of the win without giving up late routing.

To try it, both sides switch kv_connector from NixlConnector to NixlPushConnector. If the connector doesn't match the mode on both sides, the troubleshooting notes say the server crashes right after the first request.

Bi-directional mode: stop recomputing the conversation

Disaggregated serving normally moves KV one way, from prefill to decode. In a multi-turn agent session that leaves two problems. The decode node ends each turn holding KV for the prompt plus the response it just generated, but the next turn goes to a prefill node that never saw the response and has to recompute it. And because only the decode node keeps the session warm, the prefill node is the one more likely to have evicted it, in which case it recomputes the entire history.

Bi-directional mode keeps those decode-side blocks alive after generation for a fixed TTL (decoder_kv_blocks_ttl, default 480 seconds) and returns their location in the last chunk of the streaming response. A small stateful router stores that metadata keyed on a client-supplied conversation_id and attaches it to the next turn's prefill request. The prefill node checks its own prefix cache first, then reads the rest from the decode node over RDMA and computes only the new suffix. KV tensors go peer to peer. The router only holds block locations.

With prefix caching off on the prefill node, to mimic eviction or landing on a different prefill node:

That's about 1.6x at the longest Qwen prompt and 3.3x for GLM-5-FP8. The bigger model wins more because each token it doesn't recompute costs more.

Two things stood out to me when I looked closer.

First, the transfer is much slower than the link. If most of a 24K-token Qwen context comes back at 256 KiB per token, that's about 6.3 GB total, or about 790 MB per GPU. Moving that in 130 ms is about 6 GB/s per GPU, roughly an eighth of what two 200 Gb/s rails can carry. That's my estimate, since the post doesn't give the exact token count moved per turn, but the direction is clear: per-transfer overhead dominates, not bandwidth.

Second, that overhead has a floor of about 30 ms even at short prompts. The kv_recompute_threshold setting decides when reuse is worth it, and its default is 64 tokens. On this rig, recomputing 64 tokens of Qwen3-32B on eight H200s takes a few milliseconds at most, far below a 30 ms transfer. I'd treat the default as a starting point and tune it by measuring the transfer floor on your own fabric, probably into the hundreds of tokens.

What I'd watch before turning either one on

Memory held by the TTL. Retained blocks sit in decode-node HBM for the full TTL whether the conversation comes back or not, and the TTL isn't renewed. A 20K-token Qwen3-32B conversation is about 5 GB of KV. By my rough count, after weights, an 8×H200 decode node has room for something like 170 of those parked sessions at once. Set the TTL to your real gap between turns, not the default. The post says a heartbeat-renewed lease on the decode side is the natural next step.

Router state is in process. The reference proxy keeps its conversation_id map in memory. Restart it and every session falls back to a full recompute. Nothing breaks, but TTFT jumps across the board. In Kubernetes, that means a router pod reschedule shows up as a latency spike, and you should keep the proxy's cache TTL below the decode TTL, as the troubleshooting notes say.

Silent TCP fallback. If fi_info -p efa comes back empty, NIXL can fail to initialize or quietly run over TCP. Both posts suggest launching with NIXL_LOG_LEVEL=DEBUG and grepping for the rail mapping lines to confirm every GPU stayed on its PCIe-local rails. I'd make that a readiness check, not a one-time manual step.

Version drift. The bi-directional post says vLLM 0.21.0 or later, but its reproduction uses 0.29.0 plus a patch from PR #40186 so streaming responses carry kv_transfer_params. Without that patch every turn is a cache miss. Check that your build includes it before you benchmark.

You choose one. The push post says the pull connector is still the right one for workloads that need bi-directional transfer. If your traffic is mostly single-shot requests with long prompts at high QPS, push is the one to test. If it's long agent sessions that keep resending a growing history, bi-directional is. Many fleets have both kinds of traffic, which is an argument for separate pools rather than one setting.

Both posts include full reproduction scripts, and the setup is two GPU nodes plus a proxy, so it's cheap to rerun on your own fabric before trusting anyone's chart, mine included. If you're building the benchmark more formally, the IETF draft on benchmarking AI inference fabrics covers RDMA KV transfer between prefill and decode workers, and comparisons across RoCEv2 and Ultra Ethernet transports.

Related reading on this site: vLLM 0.31 fast restart, Lightbits Inferra KV prefetch, Astera Leo X fabric-attached KV memory, and Ling-3.1-flash's KV budget.

Anpoo Sivanadi, Staff Software Engineer