Adding GPUs to a Live MoE Deployment: What Elastic EP in vLLM and SGLang Actually Delivers

If you serve a big Mixture-of-Experts model with data-parallel attention and expert-parallel MoE layers, you already know the ugly part of capacity planning. Your scaling unit is a whole replica. When one replica of DeepSeek-class weights spans 16 GPUs, "add a bit of capacity" means "add 16 more GPUs," or tear the deployment down and relaunch it wider.

Two posts from last week attack exactly that, and both lean on the same new piece of plumbing. On October 19 the vLLM blog published Achieving near-zero-downtime scaling in Elastic Expert Parallelism. On October 21 LMSYS published Elastic Expert Parallelism Scaling in SGLang: Growing a Live MoE Deployment with NIXL. Both use NIXL EP as the expert dispatch and combine data plane. Both are written by NVIDIA engineers working in the open projects.

I read them as an operator: what is the actual pause, what was held constant, and what do I still have to build. Every number below comes from those two posts or from the vLLM test file they point to.

1. The headline number, and what it replaced

The vLLM post is built around one benchmark: DeepSeek V4 Pro on GB200 GPUs, TP=1, scaling between DP/EP 16 and 20.

The baseline is grim. Restarting vLLM to go from 16 to 20 GB200 GPUs stopped serving for 197 seconds. vLLM's first Elastic EP release (May 2026) avoided the restart but still paused serving while new workers initialized, communication groups were rebuilt, weights loaded and CUDA graphs were captured. That pause was 100 seconds scaling up and 56 seconds scaling back down.

The new path breaks each resize into background preparation and a short commit, and the post reports three configurations side by side:

Read the bottom half of that table before you get excited about the top half. Scaling up still takes about 92 seconds end to end. What changed is that nearly all of it now happens while the old topology keeps serving. If your autoscaler reacts to a traffic spike, you still wait a minute and a half for the new GPUs to contribute. You just stop dropping traffic during that minute and a half. Configuration C is only available with NIXL EP.

The setup details matter for reproducing it: weights were pre-downloaded as safetensors on shared Lustre storage, loaded with the default --load-format auto, and the Triton, TileLang and FlashInfer kernel caches were warm. Expect cold caches or slower storage to change the total time.

2. Why the pause was long, and why NIXL EP fixes it

The interesting engineering is in why configuration B still paused for 58 seconds even with async preparation.

The EP communication backends vLLM had before NIXL EP used fixed membership. Changing the rank set meant building a new communicator instance, with its own buffers, memory registrations, IPC handles and RDMA queue pairs. Worse, switching communicators changed the buffers that the EP kernels were captured against, which invalidated every CUDA graph on every retained worker. Warmup and recapture across all the batch shapes had to happen during the commit, while serving was stopped.

NIXL EP adds and removes ranks in place, reusing the same NIXL agent, buffers and connections. Its kernels read active membership from device state that can be updated in place, and it keeps the captured dispatch and combine entries valid across reconfiguration. vLLM did matching work on its side: it reuses retained workers' fused-MoE execution objects, keeps graph-visible workspace and dispatch-output addresses stable, and updates expert-routing mappings in place so EPLB can change contents without moving tensors (PR #54985). New DP coordination and EPLB groups still get created, but their communication runs outside the captured forward pass, so they don't break graphs.

Joining workers have the opposite problem. They have no graphs yet, and capturing them normally requires real EP traffic with peers that are busy serving. NIXL EP lets each joiner temporarily mask every remote peer, so it captures graphs against only its local rank, then reactivates peers without invalidating those graphs.

After commit, asynchronous EPLB lets the new workers start on attention while expert placement is redistributed in the background.

The SLO test is the part I'd show a skeptical SRE. The team replayed the same 11,391-request scenario, ramping from about 8 to 16 requests per second, against a static 16-GPU deployment, a restart to 20 GPUs, and an Elastic EP scale to 20. The SLO was rolling p95 TTFT under 5 seconds. Elastic EP completed all 11,391 requests and stayed under the SLO the whole time, and the post says background preparation caused no visible degradation compared to the static deployment.

3. SGLang's version: same data plane, different shape

The SGLang post covers runtime scale-up, built on its earlier Elastic EP failure-recovery work. The architecture splits into three layers: SGLang owns the serving control plane, Mooncake Process Group handles elastic membership and barriers, and NIXL EP carries the expert traffic.

The knob that matters is max_ep_size. Metadata and communication buffers for the largest topology you might reach are reserved at launch, while joining ranks only start when needed. The post is direct about the cost: higher ceilings reserve more buffer memory on the active GPUs, which leaves less memory for KV cache. Pick the ceiling you actually expect to hit.

The measurement is smaller in scale than vLLM's: 8x H100, DeepSeek-V3-Lite FP8, compared against a restart. Elastic EP reached the first completed request on the new topology 5.3x to 8.2x faster than restart. Every repeated elastic transition completed and passed a post-scale GSM8K check. The post is also explicit that token delivery pauses during the topology transition, and the absolute seconds live in a chart rather than the text, so I won't quote them.

The CUDA graph story differs from vLLM's. SGLang recaptures full-decode CUDA graphs for the expanded topology before commit (PR #33723); prefill and piecewise recapture are not supported yet. You want graphs on. With 256 GSM8K examples and 50 request threads, post-scale output throughput was:

That's 3.1x to 3.3x. The post also lists what's merged as of October 4: runtime eager scale-up, EPLB after scale-up, graph recapture, and scaling at DP-attention shard granularity (TP2/CP1 shards, so DP2/EP4 grows to DP3/EP6). Scale-down and grow-back is still under review in PR #33111. Recovery after scale-up is planned.

4. What you still have to own

Neither engine is an autoscaler. Both expose an API that admits processes. You bring the GPUs.

On SGLang, the primary launches with --elastic-ep-backend mooncake, --moe-a2a-backend nixl, an initial size and --max-ep-size. Joiners launch with --elastic-ep-join-mode scale and a contiguous --elastic-ep-join-rank-offset. Then:

curl -X POST http://<PRIMARY_HOST>:21000/scale_elastic_ep \
  -H 'Content-Type: application/json' \
  -d '{"new_ep_size": 8}'
curl http://<PRIMARY_HOST>:21000/is_scaling_elastic_ep

The POST means accepted, not done. Poll until effective_ep_size hits the target and scale_phase reads serving_expanded.

On vLLM, the post points at the cuda_graphs_heavy_nixl_ep case in tests/distributed/test_elastic_ep.py, which scales DeepSeek-V2-Lite-Chat from two GPUs to four and back under load. The serve flags it uses include --enable-expert-parallel, --all2all-backend nixl_ep, --enable-elastic-ep, --elastic-ep-max-dp-size 4, --enable-eplb with --eplb-config.use_async true, and --data-parallel-backend ray. Scaling is a POST to /scale_elastic_ep with new_data_parallel_size, and /is_scaling_elastic_ep is the probe. That test checks GSM8K accuracy before, after scale-up and after scale-down, with a tolerance of 0.08. I'd copy that habit into any rollout.

Caveats I'd carry into a design review:

• One model, one shape each. vLLM's numbers are DeepSeek V4 Pro at TP=1 on GB200 between 16 and 20 GPUs. SGLang's are a small FP8 model on one 8x H100 box. Neither shows a multi-rack resize.
• No rollback in SGLang yet. Once joiners enter Mooncake WORLD, a failure during finalization can require restarting the deployment.
• Lazy connections. In SGLang, the first wider dispatch can include connection setup cost.
• Scale-up latency is unchanged in spirit. vLLM's total time to add four GPUs is still about 92 seconds. The planned fix is restoring warm workers with Dynamo Snapshot, while keeping GPU-to-GPU weight transfer from existing workers.
• Orchestrator integration is in progress. vLLM links its external load balancer work and Dynamo's Elastic EP proposal, and mentions llm-d. Until those land, the glue between your metrics and these endpoints is yours.

5. My read

The real change here is the scaling unit, more than the 0.82 seconds. With elastic EP, a 16-GPU replica can grow by four GPUs instead of sixteen, and on vLLM it can shrink back with about two seconds of interruption. That makes fractional capacity changes worth planning for on MoE serving, which wasn't true when every change cost a minute or more of dead GPUs.

If I were trying it this week, I'd start on vLLM with NIXL EP, since of the two it's the one with graph reuse and merged scale-down today. I'd keep the ceiling (--elastic-ep-max-dp-size or max_ep_size) close to the real peak, run the accuracy check after every resize, and measure total time with my own storage and cold caches before I wired it to anything automatic.

Anpoo Sivanadi, Staff Software Engineer