Last night’s Astera Leo X note was the fabric-attached DDR path for KV when HBM fills: a Scorpio peer-to-peer memory tier that tries to keep warm TTFT off the CPU root complex. This morning is the other half of that ladder: what happens when the next needed attention blocks still live in host DRAM or NVMe, and whether a purpose-built prefetch engine can make restores cheaper than recomputing tokens.
Lightbits Labs — the company that invented NVMe over TCP — put Inferra on the AI Infra Summit floor in Santa Clara (September 15–17, 2026) as a KV-cache orchestration engine, not a general-purpose all-flash array with an inference sticker. Primary sources: the AI Infra Summit press release, Blocks & Files’ Sep 9 write-up, Lightbits’ Inferra solution brief, and StorageNewsletter’s Sep 10 summary. Specs and TTFT multiples below are vendor claims unless a named customer quote is attached.
Spec sheet (named numbers)
| Product | Inferra — KV cache orchestration / acceleration engine |
| Vendor | Lightbits Labs (NVMe over TCP inventors) |
| Public debut window | Sep 9–15, 2026 (Blocks & Files + AI Infra Summit booth 219 / Product Spotlight Theatre Sep 15) |
| Problem statement | Long-context KV grows past GPU HBM; reactive eviction → recompute or cold restore stalls the GPU |
| Tier model | Virtualize KV across GPU HBM → host DRAM → local NVMe → networked NVMe |
| Prefetch engine | SLSAP (Sub-Linear Sparse Attention Prefetch): locality-sensitive hashing + statistical reuse patterns to rank next-needed KV blocks |
| Headline concurrency claim | Up to 16× more concurrent inference sessions on existing GPUs (vendor) |
| Headline context claim | 10M-token context windows demonstrated |
| Headline latency marketing | >300× latency improvement (Summit PR); Blocks & Files publishes the per-model TTFT table below |
| Serving integrations | Plugins for vLLM, LMCache, Dynamo |
| Hardware stance | Vendor: works on open GPU + storage stacks; 10M-token demo called out on commodity L40S |
| Customer signal | OVH CPTO Yaniv Fdida: Inferra KV tiering showed “substantial GPU utilization gains” for agent and RAG workloads (Blocks & Files) |
| Operator tool | Pod Efficiency Analyzer (vendor ROI calculator) |
Inferra is not “faster NVMe.” It is a claim that timing — prefetch before attention touches the block — beats reactive paging and beats recompute on the models they published.
The published TTFT table (and what it is not)
Blocks & Files reprints three public warm-path TTFT comparisons. Treat them as vendor benches until you reproduce them on your serving stack:
| Model / context | Baseline TTFT | With Inferra | Approx. speedup |
|---|---|---|---|
| Qwen 2.5-7B @ 410K | 72.6 s | 711 ms | ~102× |
| DeepSeek-R1-70B @ 141k | 70.8 s | 465 ms | ~152× |
| Llama-4-Scout @ 10M | 1.5 h | 13 s | ~380× |
Summit marketing rounds the family of results to >300× latency improvement and pairs it with 16× session density. The useful sentence is narrower: on these long-context restores, prefetch beat recompute by two to three orders of magnitude in the published configs. That is a different claim from “tokens/s on a saturated decode loop,” and you should keep those metrics on separate rows.
The 10M-token / L40S demo is the fleet-relevant footnote. Lightbits positions Inferra as the way to keep low-HBM or older GPUs useful for ultra-long sessions instead of buying another HBM generation just to hold KV. That is adjacent to Groq 3 LPX and Jalapeño tokenomics conversations, but it is a memory readiness product, not an XPU datasheet.
How the path is supposed to work
Blocks & Files’ mechanism sketch matches the solution brief:
- Inferra runs on the GPU server’s x86 host and watches KV block activity.
- SLSAP scores blocks likely needed next (recent tokens, semantic neighbors, RAG / multi-turn structure).
- Ranked blocks are pulled from host DRAM or NVMe and preloaded into GPU HBM over RDMA before attention misses.
- The serving layer keeps its usual shape — vLLM / LMCache / Dynamo plugins rather than a forked model runtime.
That is the opposite timing model from “wait for a miss, then stall while storage catches up.” It is also the opposite purchase from Astera Leo X: Leo X tries to put a DDR tier on the accelerator fabric; Inferra tries to make DRAM/NVMe restores land early enough that HBM still looks infinite to the scheduler. You can imagine both on one BOM. You should not collapse them into one “KV offload” line item.
Where it sits next to Leo X, NVHBM, and the optics notes
| Rung | Product | What it buys |
|---|---|---|
| Package HBM | NVHBM | Memory controller / stack economics inside the accelerator envelope |
| Fabric DDR | Astera Leo X | KV / agent state as a scale-up fabric citizen (vendor H200 / Qwen2.5-32B TTFT table) |
| Prefetch + NVMe tiers | Inferra | Proactive KV readiness across host DRAM and NVMe; published long-context TTFT ladder |
| Optical plant | Lightmatter Passage L20 CPX / NewPhotonics NPG102 | Fiber / NPO engine supply once scale-up leaves the copper tray |
| Merchant rack foil | Vera Rubin NVL72 MLPerf | What the GPU island does when HBM is enough |
Four purchase domains. Inferra does not move photons and does not invent a new CXL controller. It argues that attention scheduling + storage path is where long-context margin goes after you already bought the GPUs.
Caveats I would write into any Inferra ticket
- Vendor TTFT tables, mostly. The three-model ladder is published through trade press and Lightbits. Reproduce on your vLLM/Dynamo build, tokenizer, and concurrency before you price a NeoCloud SKU.
- L40S demo ≠ H200 / Rubin / MI355X fleet. The 10M-token existence proof is on commodity L40S. Do not transpose 102× / 152× / 380× onto a Vera Rubin NVL72 or an H200 TP=4 pod without re-measuring — same discipline as the Leo X 62% warm TTFT caveat.
- Do not confuse with Leo X. Fabric-attached DDR and NVMe prefetch solve adjacent failures. Ask which miss path your traces actually hit (CPU-DDR contention vs storage RTT vs recompute) before picking a CapEx line.
- Storage RTT still exists. Prefetch only wins if the predictor is right early enough. Wrong ranks still burn bandwidth; cold tails still look like storage.
- Beta / customer programs, not a universal CSP SKU. OVH’s utilization quote is a signal, not a published multi-tenant SLA sheet.
- Do not mix with killed Huawei OceanStor / Ascend SuperPoD KV numbers. Different vendors, media, and this desk keeps that lane closed.
How I would evaluate the claim if I ran clusters
- TTFT and p99 TTFT at fixed context lengths your customers actually buy (128k, 410k, 1M, 10M if you sell it) — with and without Inferra, same model and quant.
- Sessions per GPU under a hard TBT / TTFT SLA — the 16× claim only matters if the SLA holds.
- GPU busy fraction and tokens per watt while long-context restores are in flight — stalled GPUs still draw power.
- Hit rate of SLSAP ranks vs a naive LRU / LMCache baseline on your RAG and multi-turn traces.
- Side-by-side with a Leo X or CXL DDR tier if you are already sampling fabric memory — measure which path wins on your miss histogram, not on a slide that merges them.
Bottom line
Inferra’s public debut is the strongest KV readiness software story I have seen next to last night’s Leo X fabric-DDR note: named long-context TTFT tables, a 10M-token / L40S existence proof, and plugins aimed at the serving stacks people already run. The honest read is prefetch-vs-recompute on published configs, plus a session-density claim you must re-prove under SLA. The dishonest read is “infinite HBM” or “storage vendors fixed attention.” Until your own pod reproduces the ladder, I treat Inferra as a serious NeoCloud / managed-inference POC candidate for ultra-long context — and as the software foil every fabric-KV ticket should be measured against.
Create-only. Specs and dates from Lightbits’ AI Infra Summit debut, the Inferra solution brief, Blocks & Files’ Sep 9 TTFT table, and StorageNewsletter’s Sep 10 summary. Update when a named customer publishes a reproduced ladder, or when Lightbits posts H200/Rubin-class foils with the same TTFT methodology.
Sources: AI Infra Summit PR · Blocks & Files · Inferra solution brief · StorageNewsletter.
Anpoo Sivanadi, Staff Software Engineer