The slowest part of most of my inference rollouts is the gap between a pod getting scheduled and the engine reporting ready: load weights, shard them, run quantization post-processing, compile, warm up, capture graphs. Multiply that by every config change and every crash loop, and you get a lot of expensive GPUs doing nothing.

vLLM v0.31.0 was published at 06:44 UTC on October 5, 2026 (11:44 PM PT on October 4). The release notes count 717 commits from 307 contributors, 96 of them new. There is plenty for kernel people, including more SM100 work for DeepSeek-V4.1-Flash. The part I care about is the one the release calls Fast restart: a vllm preload command that keeps post-quantized weights resident in GPU memory across engine restarts, plus experimental initialized-engine snapshots built on CRIU. They are separate mechanisms that I’d deploy differently.

Mechanism 1: vllm preload and the ipc_cache loader

The design is laid out in the Fast Start tracking issue and the original weight daemon PR. A persistent daemon per GPU loads its rank’s shard once, runs quantization post-processing, and exports every parameter and buffer as a CUDA IPC handle. An engine started with --load-format ipc_cache builds its model on the meta device and asks the daemon on its GPU for the tensors over a Unix domain socket. The CLI PR wrapped this as one command: 

vllm preload --model meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4
vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4 --load-format ipc_cache

According to the preload docs, restart time drops “from minutes (disk load + quantization processing) to seconds.” That is the project’s claim; I found no benchmark table for the daemon path, so I won’t put a number on it. The 0.31.0 additions are data-parallel support, MTP draft models, a /health endpoint and a readiness wait.

From the docs, before you build on it:

  • Two modes. zero_copy (the default) maps the daemon’s allocations directly. The daemon has to outlive the engine, and each GPU holds one copy of the weights. copy clones every tensor into the engine and then releases the daemon’s cache.
  • Sleep mode conflicts. In zero_copy, the docs warn against sleep mode (CuMemAllocator weight offloading), because the weights live in the daemon’s IPC allocations.
  • Fallback is on by default. If the daemon is missing or the fingerprint doesn’t match, the engine loads from disk. connect_timeout_s defaults to 5.0 and state_timeout_s to 300.0.
  • Identity. Socket paths come from the GPU UUID, so they survive CUDA_VISIBLE_DEVICES remapping, and the engine checks that the daemon’s GPU UUID matches its own device.
  • Trust boundary. The socket protocol uses pickle. It is restricted to the same user on the same node: a 0700 directory, 0600 sockets, and SO_PEERCRED peer-UID checks on Linux. Treat it as a local-only interface.
  • Lifecycle. An exclusive lock file per GPU makes a second daemon fail fast. If any rank dies during startup, the launcher exits, and SIGTERM tears down every rank and frees the memory.

The Kubernetes wrinkle

The most useful operator detail isn’t in the release yet. PR #60006, opened on October 5 UTC and still open, rewrites the preload guide with Docker and Kubernetes patterns. Its main point: GPU device plugins assign a GPU to one container, so you can’t just run the daemon as a sidecar on the same GPU. The proposed pattern puts the daemon and a vllm serve restart loop in the same container. Startup and liveness probes point at the daemon’s health port, and readiness points at the engine. For two-container Docker setups it lists --ipc=host, --pid=host, a shared socket volume, and the same GPUs, UID and image tag. The reasoning given is that legacy CUDA IPC handles are tied to the exporting process’s PID namespace, and /dev/shm has to be shared. The author says plainly that the Docker and Kubernetes examples have not been run on GPU hardware.

That changes how I’d write the pod spec. A Deployment rollout that replaces the pod replaces the daemon too, so you’re back to a cold load. The benefit only shows up when the engine process restarts inside a pod that stays alive: an OOM or crash, a flag change pushed through a supervisor, or a canary engine on the same GPUs. You also have to account for memory. A log quoted in issue #59647 shows the engine reporting that “of the 9.48 GiB utilization budget, 1.12 GiB is externally held.” Your gpu-memory-utilization math now has to cover a second process.

Open correctness bug

Issue #59647 is the reason I’d hold off on using this in a fine-tuning loop. The checkpoint fingerprint hashes only safetensors headers, meaning tensor names, dtypes, shapes and offsets, not weight values. For Qwen3-0.6B, the reporter measured the hashed header at 35,560 bytes of a 1,503,300,328-byte file (0.0024%). If a daemon is holding checkpoint A and you start an engine on checkpoint B with the same layout (a base model and its fine-tune, or step 0 versus step 100), the engine maps A’s weights. Neither process logs a warning, and the API still reports B. A second user reproduced it on 0.30.0 and confirmed that a different layout does fall back to disk. The fix, PR #59648, adds file size and the first 1 KiB of each tensor’s data region to the fingerprint. It was still open as of October 4, and its author notes that a change limited to bytes past the first 1 KiB of every tensor still wouldn’t change the key. Until that lands, my rule is one daemon per served checkpoint. After any weight swap, I’d compare a deterministic prompt between ipc_cache and auto before trusting the result.

Mechanism 2: vllm snapshot with CRIU

PR #51360 adds vllm snapshot create | inspect | restore. Create starts the engine, records a one-token oracle, uses sleep level 2 to release weights and KV cache, and then dumps the process tree with CRIU. Restore reloads that state before binding HTTP, replays the oracle, and fails closed if the token, text or log probability doesn’t match.

The author’s measured numbers: on one H100 PCIe 80 GB with Qwen3-32B BF16 at TP1, an ordinary warm start took a median 54.70 s against 20.38 s for a snapshot restore, a 62.7% reduction (n=3 per arm). The artifact was about 5.88 GB, and the 65.54 GB model tree stayed outside it. Creating the snapshot took 78.53 s (n=1), so the author puts break-even at the third activation. An independent tester on H20 with Qwen3-8B reported 26.64 s for a warm vllm serve against 7.05 s for a minimized restore (73.6%). A full-state artifact in their test was 52.89 GB and restored in 20.53 s.

The supported envelope is narrow, and for Kubernetes it’s the part that matters:

  • Linux x86-64, one NVIDIA GPU, TP1/PP1/DP1. Arm64 images leave out the snapshot runtime, and so do CUDA 12.x images, because the checkpoint helper needs CUDA 13 headers (#59118).
  • Same host and same GPU, with matching driver, kernel, Python, PyTorch, vLLM, model revision and engine arguments. No cross-host migration and no GPU relocation.
  • Root or passwordless sudo. The H200 CI acceptance ran in privileged Docker with kernel.io_uring_disabled=2.
  • The artifact can contain process memory and secrets, so it has to stay private.
  • Creation needs a quiet process tree, so create runs with HF_HUB_OFFLINE=1.

Privileged pods, a node-level sysctl and node-pinned artifacts holding secrets would be a hard security review in most clusters I’ve worked with. Today this is a tool for single-tenant or bare-metal nodes, not a shared serving pool. The PR thread already discusses follow-ups, including cross-host restore and artifacts that break when JIT cache files rotate.

Smaller changes operators will notice

  • --max-num-active-seqs caps RUNNING admission separately from max_num_seqs.
  • KV offloading checks cgroup memory before allocating shared memory, and UniProc startup threads are bounded by the CPU quota. Both help inside containers with limits.
  • Breaking: per-request mm_processor_kwargs and media_io_kwargs are rejected unless the server starts with --trust-request-mm-kwargs. The LoRA path is now part of the prefix-cache block hash.
  • The default image is CUDA 13.0; a -cu129 tag is still published.

How I’d use it

For a TP-sharded deployment that restarts its engine often (config iteration, crash recovery, A/B on the same node), I’d run the preload daemon inside the serving container, keep fallback on, and leave room in the memory budget for the externally held weights. I’d also keep it off any pod whose checkpoint changes until #59648 merges. I’d test snapshots on a dedicated TP1 box, because the break-even math there is straightforward. Neither fixes a cold node, where weights still come from storage. Both target the restart loop, which is where a lot of my wasted GPU-minutes go.

Last night’s TLX Jagged Flash Attention piece was about making a busy GPU faster. This release is about the minutes before it does any work at all.

Create-only. Specs from vLLM’s October 5, 2026 v0.31.0 release notes, preload docs, Fast Start tracking issue, and the snapshot / Kubernetes PRs cited above. Update when the fingerprint fix merges, when the Kubernetes preload guide is GPU-validated, or when the project publishes a daemon bench table.

Sources: vLLM v0.31.0 · preload docs · Fast Start tracking issue · weight daemon PR · CLI PR · snapshot PR #51360 · issue #59647 · PR #59648 · PR #60006 · PR #59118.

Anpoo Sivanadi, Staff Software Engineer