anpoo.

Writing

Notes

Practitioner readings of public tools and systems I care about.

  1. Humming Is Already in Your vLLM Install. Here's What the Quantized Kernel Numbers Actually Say.

    On October 21, 2026, Ant Group and the vLLM community published Announcing Humming: Fast, Flexible Kernels for Quantized Inference.

  2. vLLM and SGLang Both Caught DeepInfer on DeepSeek-V4.1-Flash. Read the Setups Before the Headlines.

    A few days after DeepSeek open-sourced DeepInfer, its own engine for DeepSeek-V4.1-Flash, both big general-purpose engines answered.

  3. Million-Token GLM-5.3 on 16 H200s: What PCP, DCP and HiSparse Actually Buy You

    The vLLM team published GLM-5.3 Optimizations, Part 2 on October 23, 2026.

  4. llm-d 1.0 Upgrade Notes: Pinned Engines, a DisaggregatedSet Prerequisite, and Quiet Dashboard Breaks

    The llm-d maintainers published llm-d v1.0 on October 20, 2026.

  5. MiniMax-M3.1-Flash on vLLM: What an NVFP4 KV Cache Changes for Capacity Planning

    MiniMax put MiniMax-M3.1-Flash on Hugging Face on October 23, and the vLLM team published a day-0 support post the same day.

  6. Dynamo 1.6 Upgrade Notes: Renamed Workers, a Mandatory Queue, and the Known Issues That Matter

    NVIDIA's distributed inference platform Dynamo v1.6.0 was published at 22:31 UTC on October 22, 2026, which was 3:31 PM PT.

  7. Raon-1 mini Preview: 8 KiB of KV per Token, and a Commit Log Worth Reading

    KRAFTON released Raon-1 mini preview on October 21 under Apache 2.0.

  8. DeepSeek Open-Sourced DeepInfer. The Deployment Q&A Is the Part to Read.

    DeepSeek put DeepInfer on GitHub this week under Apache 2.0.

  9. Reading Flint's Serving Kit Before the Benchmarks: A 40B MoE That Wants One Big GPU

    Recursive released Flint today, an Apache 2.0 mixture-of-experts coding model with 39.8B total parameters and about 2.5B active per token.

  10. The Layer Your Autotuner Never Touches: Helion Now Searches ptxas Controls

    Every kernel autotuner I've used searches the same kind of thing.

  11. TorchTPU Is Open Source: What Running PyTorch Natively on TPUs Actually Involves

    This morning I wrote about TorchNeuron and noted that TorchTPU was only a conference session with a repo on the way.

  12. TorchNeuron Public Beta: What Changes When Trainium Becomes a Normal PyTorch Device

    AWS opened up TorchNeuron today at PyTorch Conference North America in San Jose.

  13. Meta's Prometheus Numbers: What 64-of-72 Sparing and a 54K GPU-Hour MTBI Mean for NVL72 Operators

    Meta published Inside Prometheus on October 19, 2026.

  14. vLLM 0.32 Upgrade Notes: The Defaults That Changed Under You

    vLLM v0.32.0 was published at 06:50 UTC on October 18, 2026, which was 11:50 PM PT on October 17.

  15. SGLang 0.5.22 Upgrade Notes: Flags That Changed Meaning

    SGLang v0.5.22 was published at 02:55 UTC on October 19, 2026, which was 7:55 PM PT on October 18.

  16. Dory1.0-4B: What a Looped 4B Model Actually Costs Per Token

    NVIDIA put Nemotron-Labs Dory1.0-4B on Hugging Face on October 16 under the OpenMDW-1.1 license.

  17. Granite Switch 4.1: Three Models' Worth of Jobs on One KV Cache

    IBM shipped Granite Switch 4.1 this week in 8B and 30B sizes, Apache 2.0.

  18. vLLM's Two New KV Transfer Modes, and Why You Pick One

    Two posts on the vLLM blog this week change how KV cache moves between prefill and decode workers in a disaggregated deployment.

  19. Ling-3.1-flash: What 63 Linear-Attention Layers Do to the KV Budget

    inclusionAI, the Ant Group team behind the Ling, Ring and Ming model families, put the weights for Ling-3.1-flash on Hugging Face this week under the MIT license.

  20. Nemotron 3.5 Super VL and Bolt: Hybrid MoE Served From Rack to 24 GB

    NVIDIA put Nemotron 3.5 Super VL and Bolt on Hugging Face on October 15, the same window RTX Spark PCs started shipping.

  21. Muse Spark Open: Latent MoE, Hybrid Attention, and a 1M Context That Actually Fits

    Meta put the Muse Spark Open weights on Hugging Face today, October 15, the same day vLLM shipped day-0 support and NVIDIA published GB300 NVL72 serving numbers.

  22. Step 5 Preview Weights Are Out: 600B, 92 Layers, and a 74 GB Million-Token KV Cache

    StepFun put the Step 5 Preview weights on Hugging Face last night, the same day SGLang published its day-0 support post.

  23. Intel GPU D270: 480 GB on a PCIe Card, and What That Memory Is Actually For

    Intel used the OCP Global Summit this week to put final numbers and a name on Crescent Island.

  24. ReplaySSM: Hybrid Models Were Paying an HBM Tax on Every Token

    A lot of the models people serve this year are hybrids.

  25. Gemini 4 Argon: One 1M-Token Budget, a Continuation Protocol, and a Scorecard That Doesn't Win Everywhere

    Google shipped Gemini 4 Argon today, October 13, as the first model in the Gemini 4 family.

  26. Maia 300: Microsoft’s Inference ASIC Puts 288 GB of HBM4 and a 12,288-Chip Ethernet Superpod on the Table

    Microsoft used the same stage to put numbers on Azure Maia 300, the third generation of its in-house inference accelerator and the first Maia built as a multi-chiplet part.

  27. OSPA: Meta, Microsoft and Alibaba Want Every AI Fabric Switch to Boot the Same Way

    On October 8, OCP posted Open Switch Platform Architecture, a draft under OCP review initiated by Meta, Microsoft, and Alibaba: a vendor-neutral switch platform layer so AI fabric SKUs can boot, inventory, and flash the same way.

  28. Mellum2.1: A 12B MoE Coding Agent Model, and the GGUF Builds That Make It Runnable Tonight

    On October 8, JetBrains released Mellum2.1, the post-trained successor to the Mellum2 Thinking checkpoint they open-sourced in June.

  29. Liquid Open d1: Zero Output Tokens, Named Jetson Latencies, and What a Decision Model Actually Is

    On October 7, Liquid AI open-weighted two models in its d1 family: d1-3B and experimental d1-omni-600M.

  30. Vera Rubin Day Zero: What SGLang Changed in the Kernels, and What vLLM Measured on AgentX

    On October 9, two open serving stacks published the first useful Vera Rubin numbers that are not a keynote slide.

  31. RTX Spark N1X: The Windows Numbers That Matter Are TDP, Memory Ceiling, and Prefill

    On October 7, NVIDIA and Microsoft opened Windows pre-orders for RTX Spark laptops (availability October 16; compact desktops in November) and previewed DGX Station for Windows for Q4.

  32. Scality AI Inference Factory: Object Storage Over RDMA as the Shared KV Cache

    Scality launched AI Inference Factory on October 8: an open-code stack for serving open-weight models on infrastructure you own, with Scality ADI as a shared key-value (KV) cache behind the GPUs.

  33. Upscale Token Fabric: SkyFabriX at 115.2 Tbps Meets Spectrum-X Scale-Out Under One OS

    A day after Arista’s Etherlink SU-144, Upscale published Token Fabric: SkyFabriX at 115.2 Tbps, Spectrum-X scale-out, and one OS for both, with general availability aimed at early 2027.

  34. Claude Haiku 5.5: The Price Cut Is Real, but Read the 100K Tier, the Tokenizer, and the Output Bill

    Anthropic released Claude Haiku 5.5 on October 7: the small model next to Sonnet 5.5 and Opus 5.5 for high-volume work, with a real price cut that still needs the 100K tier, the tokenizer, and the output bill read first.

  35. Arista Etherlink SU-144: Ethernet Moves Inside the Scale-Up Domain, and the Rack Becomes the Unit

    Arista’s Etherlink SU-144 puts 144 accelerators in a single-hop Ethernet scale-up domain inside one rack, and up to 1,024 across racks, on OCP ESUN 1.0 — a partner-built reference design, not a SKU.

  36. openTPU: An Agent-Built LLM Accelerator on a Kintex-7 PCIe Card, and What Its Numbers Teach

    openTPU is a complete Apache 2.0 LLM inference stack on a Kintex-7 PCIe card — SystemVerilog RTL, ISA, bit-exact simulator, and host tools — with self-reported decode numbers that put memory-bound inference on one line.

  37. AICR v1.0: NVIDIA Freezes the Contract for GPU Kubernetes Recipes

    NVIDIA’s AICR v1.0.0 freezes the aicr CLI, aicrd REST API, Go SDK, and bundle schemas — a cluster configuration generator for GPU Kubernetes recipes, not a distribution or provisioner.

  38. Mistral Large 4: 1.05T Parameters, 49B Active, and What It Takes to Serve It

    Mistral’s October 6 Large 4 is a 1.05T / 49B-active open-weight multimodal MoE with a 1M-token context — public-preview API today, weights and a custom license still October 27.

  39. Reflection’s Beam: A 501B Open-Weight MoE, and the GB300 Cluster Numbers Behind It

    Reflection’s October 5 Beam is a 501B / 23B-active sparse MoE for coding, reasoning and agentic work — self-reported GB300 cluster numbers, with weights and the tech report still later this month.

  40. vLLM 0.31 Fast Restart: A GPU Weight-Cache Daemon and CRIU Engine Snapshots

    vLLM v0.31.0 Fast restart: a GPU weight-cache daemon and experimental CRIU engine snapshots — project claims of minutes-to-seconds restarts, with an open fingerprint bug and Kubernetes caveats.

  41. TLX JFA: Meta’s Blackwell Attention Kernel — ~13% Fwd / ~50% Bwd vs FA4 on Jagged

    Meta’s Oct 1 TLX Jagged Flash Attention is a Blackwell B200 kernel for packed jagged / broadcast-Q — Meta’s bf16 benches claim ~+13% forward / ~+50% backward vs May 2026 FA4 on their shapes, not an independent rematch.

  42. Dist-MoE: Meta’s Blackwell SM100 MoE Kernels — BF16 / MXFP8 Train, NVFP4 Infer

    Meta’s Oct 3 dist-moe v0.1.0 is CuTe DSL distributed MoE kernels for Blackwell SM100 — BF16 / MXFP8 E4M3 train+infer, NVFP4 infer-only — Meta’s docs, not a published throughput rematch.

  43. Kolibri-1: 78.1B / 3.46B Open MoE — Hybrid SWA, 1M Context Claim, ~78 GB FP8

    Aleph Alpha’s Oct 3 Kolibri-1 is a 78.1B / 3.46B open MoE with hybrid 4:1 SWA(512):full, native 256k validated to 1M, and ~78 GB FP8 — vendor benches and a named B200 train topology, not an independent rematch.

  44. Credo Weaver: HBM4-Class Edge Bandwidth at ~25× Capacity — OmniConnect VSR, No CoWoS

    Credo’s Oct 2 OmniConnect / Weaver post claims HBM4-class die-edge bandwidth (6.4 vs 6.1 TB/s) at ~25× capacity (4.8 TB vs 192 GB) on organic, no CoWoS — vendor architecture claims, not an independent bring-up.

  45. Lightning V2 on Tenstorrent: 4× TTS Cost Cut vs L40S — BFP8, NoC, Perceptual Fragility

    Tenstorrent and Smallest.ai put Lightning V2 on Blackhole — vendor L40S bake-off with ~3–4× modeled TTS CapEx, BFP8/LoFi co-design, and a paper on perceptual fragility — not a blanket NVIDIA cost claim.

  46. Olmo-core 3: Open MoE Training Stack Hits 1.2T on B300 — DDP, Rowwise EP, MXFP8

    Ai2 released Olmo-core 3 — open DDP+EP+PP MoE trainer on NVIDIA B300 through 1.20T / 58.4B active, 19.4K → 52K tok/s/GPU at 47B vs the old FSDP path — systems measurements, not next-Olmo weights.

  47. Everspin Hooks PERSYST MRAM to CXL — 1GB Persistent Demo Tier Between DRAM and NAND

    Everspin demonstrated CXL-connected PERSYST 1GB MRAM at SNIA SDC 2026 — memory-semantic persistence between DRAM and NAND, vendor 100× writes vs NAND, working demo not a shipping SKU.

  48. CoreWeave Puts Vera Rubin NVL72 Into Production With Cognition — 4.8× SWE-2 Tokens, Not Another Shipping BOM

    CoreWeave puts NVIDIA Vera Rubin NVL72 into limited availability with Cognition as first production customer — 4.8× SWE-2 and 3.8× RL token throughput vs GB200 (customer claims) — neocloud production, not another shipping BOM.

  49. Claude Sonnet 5.5: Everyday Coding Ladder Beside Opus 5.5 — Cost Per Task, Not Another Flagship Rematch

    Claude Sonnet 5.5: same $2/$10 list as Sonnet 5, vendor fewer-tokens plus 30%+ faster generation, near-Opus on several Max benches — everyday coding ladder beside live Opus 5.5, not another flagship rematch.

  50. Naive-N0.5-Flash: 309B / 15.5B MoE with Zero Full-Attention Layers — Native 1M via SWA+DSA

    Naive-N0.5-Flash: open-weight 309B / 15.5B MoE with zero full-attention layers, native 1M via SWA+DSA, MIT, announced $0.10/$0.40/$0.01, NaiveRT peak you should re-time — decode-cost architecture, not an AA rematch.

  51. NVIDIA Open Agent Safety: OpenShell 0.1.0 + BlueField-4 Sentry — Runtime Policy, Milliseconds Quarantine

    NVIDIA Open Agent Safety: Apache-2.0 OpenShell 0.1.0 plus optional BlueField-4 Sentry — YAML → OPA/Rego, OCSF, milliseconds quarantine as a vendor claim, no DPU required — runtime policy for agent fleets, not an AA rematch.

  52. JEV-27B: Open System 1 Decisions on a Frozen Qwen3.8-27B — KL 0.017, One B200

    AutoTrust JEV-27B: open System 1 decisions on a frozen Qwen3.8-27B — KL ≈0.017, six-bench mean 84.07, 137 ms on one B200 — vendor self-comparison vs hosted TypeSafe Jev 1.13, not a third-party bake-off.

  53. Holo4: Computer-Use Agents With Named $/Task — 27B Dense, 35B-A3B MoE, Open Trajectories

    H Company Holo4: 27B dense + 35B-A3B MoE computer-use VLMs with named OSWorld / AutomationBench $/task and open trajectories — no AA Index row, and a CC BY-NC vs Apache-2.0 license split you should write into the routing ticket.

  54. AliceAI-Foundation-80B: From-Scratch 3B Activate, KDA:GA 3:1 — and Two Open Russian Factual Benches

    Yandex AliceAI-Foundation-80B-A3B-Base: from-scratch 80B/3B hybrid MoE, KDA:GA 3:1, Apache 2.0, WikiWebFacts / HardMultiQA — no AA Index row yet, and a base you still owe SFT/RL before you compare it to Flash/Pro chat endpoints.

  55. DeepSeek-V4.1-Flash: 890 Bytes of KV per Token — and Why V4-Pro Is Being Retired

    DeepSeek-V4.1-Flash: 890 bytes KV/token, 8B prefill / 16B decode MoE, MIT weights, V4-Pro traffic steered to Flash rates — AA Index 39, with a Terminal-Bench 4.0 gap vs Opus/Sol on DeepSeek’s own table.

  56. MiMo-V2.6-Pro: Open Weights at AA 46 — and the Terminal-Bench 4.0 Asterisk

    Xiaomi MiMo-V2.6-Pro / Flash: MIT open weights, AA Intelligence Index 46, $0.435/$0.87 list (Flash cheaper), same-day foil to Grok 4.7 — with a Terminal-Bench 4.0 gap vs Opus/Astra on Xiaomi’s own table.

  57. Lessengers’ 3.2T Glass NPO: TGVs, Lens-Free DOW, and the Millimeters-from-ASIC Bet

    Lessengers’ 3.2T glass-substrate NPO: TGVs, lens-free DOW polymer waveguides, millimeters-from-ASIC placement, immersion-native claim, aimed at 51.2T/102.4T switch generations — vendor claims until your BER and yield tickets land.

  58. PicoJool’s 200G GaAs VCSEL: >37 GHz, Three Paths to 1.6T, and the Unconstrained-Supply Bet

    PicoJool’s 200G GaAs VCSEL / microVCSEL bet: >37 GHz device claims, three 1.6T recipes, WIN foundry, Series A funding qualification → early-2027 volume — vendor claims until your BER tickets land.

  59. Supermicro Is Shipping Vera Rubin NVL72: 20.7 TB HBM4 and a Cold-Plate-to-Tower BOM You Can Order

    Supermicro says Vera Rubin NVL72 racks are shipping now — 72 Rubin GPUs, 20.7 TB HBM4, and a cold-plate-to-tower DLC BOM you can order as a 16-rack Scalable Unit.

  60. EdgeCortix RAIDEN: 3.36 PFLOPS FP4 on Four Dies — Thick-Edge Physical AI as One Chiplet Stack

    EdgeCortix RAIDEN claims up to 3.36 PFLOPS FP4, 256 GB, and 1.54 TB/s die-to-die on a four-die Physical AI chiplet stack shipping samples in early 2027.

  61. WhiteFiber Continuum: 136 Tbps Across 83 km — Scale-Across When One Campus Hits the Power Wall

    WhiteFiber Continuum claims 136 Tbps / 0.9 ms RTT across 83 km of metro fiber so two campuses can act as one GPU supercluster when one site hits the power wall.

  62. Claude Opus 5.5: Fable-Class Work at Opus Prices — and the Safeguard Asterisk

    Claude Opus 5.5 — Fable-class work at $4/$20, and the fallback-safeguard asterisk.

  63. GPT-6 Sol and Luna: Half the Price, Not Always a Higher Ceiling

    GPT-6 Sol and Luna — half the GPT-5.6 promo price, cache-steerable effort, not always a higher ceiling.

  64. T-Head Zhenwu V900: 216 GB, 1,200 GB/s Chip-to-Chip, and a 500,000-Card Cluster Story

    Zhenwu V900 — 216 GB, 1,200 GB/s chip-to-chip, and a 500,000-card cluster story.

  65. Axelera Europa: 629 TOPS at 45 W — Shipping Edge 232p Math With a Dell/Supermicro Asterisk

    Europa 629 TOPS at 45 W — shipping Edge 232p math with a Dell/Supermicro asterisk.

  66. Lightbits Inferra: KV Cache Prefetch Beyond HBM — Named TTFT Tables on L40S Math

    Inferra KV-cache prefetch beyond HBM — named TTFT tables on L40S math vs recompute.

  67. Astera Leo X: Fabric-Attached DDR for KV Cache — 62% Faster Warm TTFT on H200 Math

    Leo X fabric-attached DDR for KV cache — 62% faster warm TTFT on H200 math vs CPU-DRAM offload.

  68. NewPhotonics NPG102 + Tower PH18DA: Laser-Integrated PICs Shipping for 800G–1.6T — and the CPX 6.4T On-Ramp

    NPG102 + Tower PH18DA — laser-integrated PICs shipping for 800G–1.6T, plus the CPX 6.4T on-ramp.

  69. Cornelis Active Compute Fabric + Delos Nonstop AI: Open Scale-Up Math Beyond NVLink

    Cornelis ACF + Delos Nonstop AI — 800G fabric sampling and >30 Tbps I/O dies beyond NVLink.

  70. AMD EPYC Venice vs Nvidia Vera: 2.24× SPECrate, 1.6 TB/s Stream — and the Compiler Asterisk

    EPYC Venice vs Vera — 2.24× SPECrate and 1.6 TB/s Stream, plus the compiler asterisk.

  71. Meta MTIA 450 / Arke: 18.4 TB/s HBM, FlashAttention in Silicon — and the Sep 1 Bring-Up Asterisk

    MTIA 450 / Arke — 18.4 TB/s HBM and FlashAttention in silicon, plus the Sep 1 bring-up asterisk.

  72. Lightmatter Passage L20 CPX: BiDi NPO That Halves the Fiber Plant — and the Open CPX On-Ramp

    Passage L20 CPX — BiDi NPO that halves the 512-GPU fiber plant, plus the Open CPX MSA on-ramp.

  73. AMD’s 512-GPU MI355X MLPerf: 5.75M tok/s on RoCE — and the +38% ROCm Asterisk

    512-GPU MI355X on RoCE — 5.75M tok/s, plus the +38% same-silicon ROCm asterisk.

  74. Vera Rubin NVL72’s First MLPerf: 3.7× GB300 on Qwen3-VL — and the AgentX Tokens-per-Megawatt Asterisk

    First MLPerf Inference v6.1 preview — 3.7× GB300 on Qwen3-VL, plus the AgentX tok/MW asterisk.

  75. d-Matrix Raptor Joins NVLink Fusion: 100 TB/s Cards, NVL144 Math, and the Decode-Silicon Bet

    Raptor on NVLink Fusion / MGX — 100 TB/s card math, NVL144 aggregates, and the decode bet next to LPX.

  76. Qualcomm × Amazon: Custom Inference Silicon, 1.6T Optics, and a Second ASIC Line Next to Trainium

    Custom inference silicon and 1.6T optics for AWS — a second ASIC line next to Trainium.

  77. Release Workflow Policy Tests: Catch Signing Drift Before Release Day

    Mutation-tested release-workflow policy — sole signer, OIDC scope, SHA pins, and asset-list drift.

  78. NVIDIA Groq 3 LPX: 3,431 tok/s at 100K — and the Batch-1 Caveat

    3,431 tok/s at 100K on Gemma 4 31B — and the batch-1 concurrency caveat for agent fleets.

  79. AMD Helios UALoE: Open Scale-Up on Ethernet, With the Bridge Caveats

    72× MI455X, UALink-over-Ethernet, and the open scale-up caveats next to NVLink Fusion.

  80. NVIDIA NVHBM: Moving the Memory Controller Into the Stack

    Custom HBM base die for NVLink Fusion — bandwidth, power, and die-area targets, with the caveats.

  81. OpenAI Jalapeño: Spec Sheet, InferenceX Numbers, and the Tokens-per-Megawatt Bet

    InferenceX tokens-per-megawatt numbers, HBM4 package math, and the captive ASIC caveats.

  82. AMD’s Threadripper Halo Station: Spec Sheet, HBM Math, and the DGX Station Bet It Is Making

    MI350P HBM math, the path to four cards, and AMD’s deskside bet against coherent DGX Station.

  83. K2 Horizon Ships Six Open Models — and Publishes the Reward-Hack Audit on Its Own Terminal-Bench Score

    Six Apache-2.0 models, MoVA, and a published Terminal-Bench reward-hack audit from IFM.

  84. GPT-6 Astra Is Live: Spec Sheet, Vendor Benches, and Where It Beats Fable on Cost-per-Task

    Spec sheet, vendor benches, and where GPT-6 Astra beats Fable on cost-per-task.

  85. Claude Fable 5.1 Is the New Default for Agentic Coding Fleets: Here’s the Spec Sheet

    Scores, cache pricing, and how I would route coding-agent traffic after Anthropic’s Fable 5.1 / Mythos 5.1 GA.

  86. A first look at NVIDIA Cluster Readiness Engine

    What NVCRE v0.8.0 is, how certifications compose, and how to run your first category from the official docs.