Nemotron 3.5 Super VL and Bolt: Hybrid MoE Served From Rack to 24 GB

NVIDIA put Nemotron 3.5 Super VL and Bolt on Hugging Face on October 15, the same window RTX Spark PCs started shipping. The product blog frames it as an open agent stack from data center to desktop. The useful story for anyone who runs inference is how the same hybrid backbone is packaged for two very different memory envelopes: a 120B/12B-active multimodal MoE with a 1M context and native multi-token prediction, and a Star Elastic–compressed 74B/9B LatentMoE that lands in a 24 GB pack for local agents.

Last night I wrote about Muse Spark Open, which cuts expert width and full-attention layers to make an 816B model fit a real KV schedule. Nemotron 3.5 takes a different bet: interleave Mamba-2 with MoE and attention, ship MTP as a first-class drafter, and then surgically shrink the same parent into something that shares a laptop with your other apps.

What the cards agree on

The NVFP4 model card is explicit that vision encoder, connector, Mamba convolutions, and embeddings stay in BF16, the MoE router stays in FP32, and inference is W4A4 on Blackwell and W4A16 elsewhere. That is the kind of mixed-precision map you want before you trust a single-GPU “fits” claim.

Hybrid backbone: why Mamba-2 is in the critical path

Super VL is not a pure Transformer MoE. Layers interleave Mamba-2 state-space blocks, sparse experts, and attention, then hang a C-RADIO vision tower and an MLP connector off the language backbone. The developer deep dive spells out the serving implication: prefix caching has to preserve both attention KV and Mamba recurrent state, with extra Mamba checkpoints so speculative decode does not replay the whole prefix.

That matters for always-on agents. Every turn re-sends system prompts, tool schemas, and history. If your SSM cache is not checkpointed the same way as KV, you pay Transformer-style prefill taxes on a hybrid model that was supposed to avoid them. The vLLM and TensorRT-LLM recipes on the card turn on FlashInfer Mamba backends, shared-prefix Mamba checkpoints, and FP8 (or NVFP4-trained) KV for exactly that reason.

MTP and LiLiCorr: speculation without a second model family

Multi-token prediction is baked into the checkpoint. A shared MTP head drafts several future tokens; the main model verifies them in one pass. Super VL’s training notes describe a hidden-state-mixing warmup so later draft steps see heterogeneous states the way inference does, instead of only ground-truth step−1 states. On the card, GB200 MTP recipes use five draft tokens; H100/H200 use three; DGX Spark prefers a separate LiLiCorr drafter that scores draft-token compatibility in one shot.

NVIDIA’s developer blog claims LiLiCorr is up to 2.3× faster than no speculation on DGX Spark. Treat that as a vendor upper bound on their stack. The architectural point still stands: you get speculative decode without maintaining a separate tree of draft models for every precision and SKU, and Bolt reuses the same LiLiCorr/DFlash pattern with seven speculative tokens in the published vLLM command.

Dynamo numbers for the rack path

For multi-GPU serving, the interesting figure is not the single-stream peak. On 16 B200 GPUs at 50 tokens/s per user, Dynamo with KV-aware routing, session affinity, and KV offload is claimed to serve up to 8× more users while cutting first-token latency by 85% at the same per-user rate. Recipes cover H100 through VR200 with NVFP4 weights and FP8 KV, MTP3 or MTP5 depending on topology.

Separately, the Artificial Analysis chart NVIDIA cites puts Super VL above 600 output tokens/s on agentic tasks on a Baseten endpoint, with a “5× vs next open in class” callout. That is an endpoint measurement, not your cluster. Use it as a throughput envelope, then re-measure on your harness with tool calls and vision turns included.

Benchmarks on the NVFP4 card (same harness as BF16) land roughly: GPQA-Diamond 89.9, SWE-bench Verified 71.07, Terminal-Bench 2.1 62.78, BrowseComp 77.5, CharXiv reasoning 79.88. Quantization-aware distillation is doing real work if those gaps stay that tight.

Bolt: Star Elastic into 24 GB

Bolt is the compression story. The 24GB card and Bolt training write-up describe Star Elastic: a budget-conditioned router that heterogeneously picks routed-expert counts and MoE intermediate widths per layer while fixing top-K at 9 (down from 22 in the parent), targeting under 24 GB at batch 1 and 128K context after quantization. Architecture search runs at 8K for the first 8B tokens, recovery continues to 72B at 8K, then 48B at 128K — 120B tokens of compression training before vision adapter, SFT, RL, and IQ2-XXS quantization-aware distillation on the experts.

The packed footprint is about 22.1 GB from a ~148 GB BF16 parent path. Experts go to IQ2-XXS; Mamba layers can stay NVFP4; embeddings and output stay Q8_0. Context drops from 1M to 256K, which is the right trade for a laptop agent that still needs BrowseComp and Terminal-Bench headroom. On the published table, Bolt 24GB scores GDPVal-AA-V2 28.47, Terminal-Bench 2.1 50.98, BrowseComp 66.75, PinchBench 87.82, and GPQA Diamond 86.43 against denser and smaller MoE peers in the same memory band.

That is why today’s RTX Spark availability matters without needing another hardware post: the model that was sized for a 24 GB co-resident agent is shipping the same week as the Windows Grace-Blackwell laptops meant to run it via llama.cpp, Ollama, LM Studio, Unsloth, and vLLM.

License and customization, briefly

Both checkpoints use OpenMDW 1.1. NVIDIA is also dumping open post-training datasets and NeMo Gym environments, and the developer post lists NeMo AutoModel / NeMo RL paths for SFT, LoRA, and environment RL. If you are capacity-planning, the open weights plus Dynamo recipes matter more than the partner logos.

What I'd measure first

• Tokens/s per user and TTFT at 50 tok/s interactivity on a fixed B200 or GB200 node, with and without Dynamo KV-aware routing, using the same tool-calling harness you run in production.
• MTP acceptance length versus draft depth (3 vs 5 vs LiLiCorr-7) on DGX Spark and on a discrete Hopper node, with Mamba shared-prefix checkpointing on and off.
• Bolt 24GB concurrent sessions at 32K and 128K on a single RTX Spark / DGX Spark after FP8 KV, with vision turns enabled, to see when the 22.1 GB weight pack stops leaving room for SSM state.

Where I land

Nemotron 3.5 Super VL is the open agent checkpoint I would actually put on a NIM/Dynamo path this week: hybrid Mamba+MoE keeps long agent prefixes cheaper than a pure attention MoE of the same active size, MTP is in the weights instead of a side project, and NVFP4 QAD keeps the GPQA/SWE gaps small enough to trust. Bolt is the complementary move — Star Elastic plus 2-bit experts so the same family runs beside your desktop apps in 24 GB, timed with RTX Spark shipping. Opposite levers from Muse Spark Open’s latent experts and hybrid attention, same job: make agent context and agent concurrency fit the hardware you already bought.

Sources: NVIDIA Nemotron 3.5 product blog, developer deep dive, Super VL NVFP4 model card, Super VL BF16 model card, Bolt 74B-A9B 24GB model card, Dynamo recipes, RTX Spark PC availability. Architecture, benchmark, and serving figures are from NVIDIA’s blogs and Hugging Face cards. Footprint arithmetic that cites ~22.1 GB / ~148 GB BF16 follows NVIDIA’s Bolt compression description.

Related reading on this site: Muse Spark Open, Step 5 Preview open weights, RTX Spark N1X.

Anpoo Sivanadi, Staff Software Engineer