NVIDIA's distributed inference platform Dynamo v1.6.0 was published at 22:31 UTC on October 22, 2026, which was 3:31 PM PT. The release notes count 654 merged PRs from 148 contributors, 58 of them from outside NVIDIA. The headline features are pluggable flow control in the router, LPU-backed workers through LPX, active/standby engine failover through Dynamo Snapshot, and a Planner whose GPU budget can be changed at runtime.

This post is not about those. When I upgrade a serving stack, the risk is rarely the feature I opt into. It's the default that moved, the name that changed, and the known issue that matches my topology. Dynamo 1.6 has plenty of all three, and the notes are candid about them. Below is what I'd check before rolling it, with every number taken from the release notes. Anything that is my reading rather than theirs is labeled.

1. Check the engine versions first

Dynamo pins its engines. v1.6.0 ships SGLang v0.5.21, TensorRT-LLM v1.3.0rc29 and vLLM v0.31.0, on CUDA 13.0 for SGLang and vLLM and 13.1 for TensorRT-LLM. NIXL is v1.5.0 for the Frontend and SGLang, v1.4.1 for vLLM and v1.3.1 for TensorRT-LLM. Grove moves to v0.1.0-alpha.14 and ModelExpress to v0.7.0.

The vLLM line is the one to notice. vLLM itself shipped 0.32 on October 18 (UTC), and I wrote up its changed defaults in vLLM 0.32 Upgrade Notes. Dynamo 1.6 is still on 0.31.0, so none of those 0.32 behaviors apply inside a stock Dynamo 1.6 image. If you've already tuned clients around 0.32 semantics, such as max_tokens no longer being reserved from the context window, they don't hold here. My reading: don't mix the two upgrades in one rollout, and don't assume a Dynamo image behaves like the standalone vLLM release with the same month on it.

On TensorRT-LLM, rc29 deleted the AutoDeploy backend upstream, so backend: _autodeploy now raises an error.

2. vLLM workers have new names, and your dashboards use the old ones

vLLM worker services drop VllmWorker, VllmPrefillWorker and VllmDecodeWorker for worker, prefill and decode, matching the SGLang, TensorRT-LLM and Mocker backends (#9848). The stated reason is Grove's 45-character name limit.

The rename isn't cosmetic. Pod names, Kubernetes Service names and the dynamo_component metric label all change. Any alert that filters on dynamo_component="VllmDecodeWorker" will go quiet after the upgrade without failing, which is the worst kind of breakage. I'd grep manifests, overrides, Grafana JSON and alert rules for the old names before touching the cluster.

3. The Operator stops adopting what it doesn't own

Three Operator changes can block a rollout outright:

  • No more adoption (#13990). If a Deployment or Service already exists with the same name as an Operator-managed child but has no controller owner, or a different one, the Operator no longer adopts, updates or deletes it. The deployment reports an ownership conflict instead. Clean up leftovers from hand-applied manifests first.
  • Stricter multinode validation (#14468). multinode is accepted only on worker, prefill and decode. roles[] takes at most two entries, and each declared role needs a complete PodTemplate that owns its launch command.
  • A reserved init container name (#14927). An init container called runtime with restartPolicy: Always now switches the component into native sidecar mode.

New multinode deployments also require Grove v0.1.0-alpha.14, and the Operator now owns DYNAMO_RANK and DYNAMO_LEADER_ADDRESS. If you manage Grove yourself, upgrade it and its CRDs first. The platform chart now runs a preflight Job that blocks the Operator upgrade when Grove is incompatible or still rolling out (#16891). That Job pulls docker.io/library/python:3.12-slim by default, so air-gapped clusters need dynamo-operator.grovePreflight.image pointed at an internal registry.

Also: Grove coherent updates are now opt-in through the nvidia.com/grove-update-strategy: Coherent annotation, a frontend in deployment foo no longer discovers workers in foo-bar, and GB200 nodes report GPU SKU gb200 instead of gb200_sxm.

4. Every request now goes through the router's policy queue

Before 1.6, an uncontended request could bypass the router's queue. Now every request goes through the policy queue, which orders by deadline (#14176). A policy class with no registered workers returns a no-endpoints error, not a queue-rejected one. If your client retry logic matches on the old error, update it.

The bigger change is configuration. Scoring knobs move into the YAML passed to --router-policy-config, and that file now takes precedence over the old flags and environment variables (#15202). The six scoring flags, --load-aware and the queue-threshold flags are deprecated and slated for removal in v1.7.0. One default to watch: when shared cache is enabled through YAML, shared-cache hits now earn a multiplier of 0.5. Set shared_cache_multiplier: 0.0 if you don't want that credit. The overlap-query response also drops router_credit_blocks.

On the Planner side, throughput scaling is capped at 8 replicas per decision per component (#14296). If your autoscaling relied on jumping 20 replicas in one step during a traffic spike, raise max_throughput_scaling_replicas. The AIConfigurator options are renamed to AISimulate (--aic-* to --ais-*), and retired DYN_AIC_* variables now fail at startup.

5. Settings that used to fail quietly now fail loudly

These will bite anyone with a typo in a values file:

  • Invalid DYN_REQUEST_PLANE and DYN_EVENT_PLANE values now stop startup. They used to fall back to TCP and ZMQ silently.
  • Unbracketed IPv6 ZMQ endpoints such as tcp://2001:db8::10:5555 are rejected instead of retrying forever.
  • The documented DYN_METRICS_* histogram variables now actually change bucket boundaries. They were ignored before, so check whatever you set and forgot about.
  • The default NIXL Prometheus port moves from 19090 to 9090. If your scrape config expects 19090, set NIXL_TELEMETRY_PROMETHEUS_PORT explicitly.
  • On TensorRT-LLM, --publish-kv-events no longer enables performance metrics; add --publish-metrics.

6. Snapshot mode changes what NCCL does

Dynamo Snapshot now captures CUDA shared memory by default (#15791). The checkpoint target container must set command, and existing checkpoints get recaptured once because the compatibility hash now includes the annotation.

The easy line to miss: snapshot mode no longer forces NCCL_CUMEM_ENABLE, NCCL_NVLS_ENABLE and TLLM_NCCL_SYMMETRIC_ZERO_COPY off. If you run snapshots and your collectives were stable only because those were off, set them to 0 yourself. To keep the old capture behavior, set nvidia.com/cuda-shared-memory-support: disabled.

7. Client-facing behavior changes

A few of these reach application teams, not just the platform team:

  • Structural tags are now on by default for supported parsers. Valid tool schemas that use constructs the grammar backend rejects, such as regex lookahead, can now fail requests. DYN_ENABLE_STRUCTURAL_TAG=false restores the old behavior.
  • Non-streaming /v1/responses no longer infers tool calls from assistant text. Only structured tool calls from the backend count, so enable a backend tool-call parser.
  • /v1/models/<model>/ready returns 503 for a known but unready model, instead of 200 with ready: false. Probes that only check the status code will see the difference.

8. The known issues, sorted by how much they'd hurt

The notes list known issues per component. The ones I'd check against my own setup:

  • SGLang scheduler crash. A request whose stop_token_ids includes an ID at or above the vocabulary size can crash the scheduler when any request in the batch sets min_tokens, failing every request on that worker until restart. No workaround yet, and it also affects v1.5.0. On a public endpoint, my reading is that you should validate stop_token_ids at your gateway.
  • vLLM without max_tokens. A request with no max_tokens can generate to the full remaining context when the model sets no default. Set it on every request.
  • logit_bias is ignored. The Frontend accepts it on chat and completions and drops it before the engine. No workaround.
  • EFA silently falls back to sockets. Workers set up from the EFA on AWS guide use socket transport because the guide's LD_LIBRARY_PATH is wrong. Use /opt/amazon/ofi-nccl/lib instead of /opt/aws-ofi-nccl/lib. On EFA, I'd check this one first, since it costs bandwidth without any error.
  • Planner refuses to scale. For aggregated vLLM deployments, a regression model seeded from worker self-benchmarks can overestimate capacity and refuse to scale up under load. Use a pre-published AISimulate performance model.
  • Snapshot plus SGLang CUDA graphs. SGLang workers in Snapshot mode or with GPU Memory Service checkpointing fail at startup with NotImplementedError. The workaround is --disable-cuda-graph, which costs decode performance.
  • Smaller routing issues. Decode-only pins fail in disaggregated serving (pin both workers or neither), rolling updates can return 404s for about a second, and GlobalRouter pool requests can find no endpoints on Grove.

My upgrade order

  1. Upgrade an externally managed Grove and its CRDs to v0.1.0-alpha.14, and mirror the preflight image if you're air-gapped.
  2. Grep manifests, dashboards and alerts for VllmWorker, VllmPrefillWorker, VllmDecodeWorker, gb200_sxm and any init container named runtime.
  3. Delete or rename leftover objects that collide with Operator-managed names.
  4. Move router scoring flags into --router-policy-config, and decide on shared_cache_multiplier on purpose.
  5. Validate every DYN_* and NIXL_* variable, since bad values now stop startup.
  6. On EFA, fix the LD_LIBRARY_PATH. On Snapshot, set the NCCL variables explicitly.
  7. After rollout, watch per-component metrics under the new labels, scale-up events and the 503 rate on readiness endpoints.

Keep the vLLM 0.32 move as a separate change, after Dynamo ships an image that carries it.

Sources

Anpoo Sivanadi, Staff Software Engineer