Scality AI Inference Factory: Object Storage Over RDMA as the Shared KV Cache

Scality launched AI Inference Factory on October 8: an open-code stack for serving open-weight models on infrastructure you own, with Scality ADI as a shared key-value (KV) cache behind the GPUs. The interesting claim is not the on-prem packaging. It is that object storage over RDMA is fast enough to sit in the serving path — 166 ms warm time-to-first-token on a 14K-token restore, 14× faster than recompute on that shape, and 72× on a 438,764-token GLM-5.2 session.

That sits next to two posts already on this site: Astera Leo X (fabric-attached DDR for warm KV) and Lightbits Inferra (prefetch beyond HBM with named TTFT tables). Scality’s bet is a third tier — multi-petabyte object storage, CPU off the data path, any decode GPU able to pick up any returning session.

Why the cache left the GPU

Every returning agent session is a choice: recompute the context, or read it back. Scality CTO Giorgio Regni’s companion write-up cites a 2026 agentic-workload study where 84.6% to 99.5% of input tokens were cache-reusable turn to turn, and hit rates collapsed from ~98% to 16%–26% once too many concurrent agents crowded the cache. Cloud providers price a cached input token at about a tenth of an uncached one for the same reason: recompute is the expensive path.

On your own fleet that cache is hardware you already bought — until it no longer fits. Regni’s lab measured one GLM-5.2 session at 438,764 tokens producing 189 GB of KV cache, about 430 KB per token. With the model loaded across all eight GPUs of their test server, that single session did not fit in leftover HBM. Scale that shape to thousands of concurrent coding-agent sessions and you are in petabyte territory. That is a storage problem dressed up as an inference problem.

Architecture: prefill writes, decode reads, ADI in the middle

Inference Factory packages:

Validated open-weight families (Mistral, Gemma, gpt-oss, Qwen, Kimi, GLM, DeepSeek)
A serving layer built on vLLM and Dynamo that splits prefill from decode
A control plane for auth, metering, routing, and SLA-aware scheduling
Scality ADI underneath for models, enterprise data, and inference state
The Scality AI Connector between GPUs and ADI

vLLM breaks the KV cache into fixed-size named blocks. ADI stores each block as an object. On a restore, the GPU reserves memory and tells ADI which object to place there. The TLC NVMe drive moves the object into the storage server’s RAM over PCIe, and the NIC writes it straight into GPU memory over RDMA. CPU, kernel, and the usual S3 software path stay off the data plane. Scality says that path hits 97% of network line rate between GPUs and storage.

The control plane is where they cut corners on purpose. Even S3-over-RDMA still pays for request signing, policy checks, XML, and multipart framing on every object. The AI Connector swaps that for a stripped-down control plane with single-pass authentication and no XML/multipart framing, while keeping the object model. StorageReview’s write-up walks the same path.

Split serving is the payoff. Prefill computes the KV cache for a new prompt and writes it to ADI. Decode reads it back and generates tokens. DistServe (OSDI 2024) measured up to 7.4× more requests within the same latency targets when prefill and decode no longer share GPUs. Moonshot’s Mooncake reported 75% more requests on Kimi production traffic with a shared KV pool. Scality’s version of that design uses ADI as the shared pool, so any decode GPU can resume any session and the cache is not capped by one server’s DRAM.

The numbers, with the lab caveats attached

All figures below are Scality’s self-reported lab results from the CTO post and the October 8 press release. The lab was deliberately small: one eight-GPU server, five older storage servers, and four 100 Gb/s RDMA links. Scality says the product itself runs on a 400 Gb/s fabric, and it does not name the GPU. Treat these as architecture measurements, not a bake-off against Leo X or Inferra on matched silicon.

Measurement	Result
Warm TTFT, 14K-token restore from `ADI`	166 ms (83 ms behind same context from HBM)
Recompute that 14K context	2.3 s → restore 14× faster
GLM-5.2, 438,764-token context	Recompute 529 s, read-back 7.3 s → 72×
KV cache held with 8 `vLLM` instances (1/GPU)	8 TB (~80× one GPU’s memory, ~10× the whole server)
First-token latency across 69 GB → 8 TB cache sizes	355–373 ms, flat
Concurrent sessions resumable without recompute	Up to ~1,000 (vendor claim at that capacity)
Gemma-3 27B weight load (54.86 GB) from `ADI`	1.9 s into one GPU (~10× local NVMe; ~28.9 GB/s on the four-link lab)
GPU↔storage transfer	97% of network line rate

Inter-token latency did not move with cache size: the fetch finishes before the first token, and every token after that runs from GPU memory. That is the right property if you want storage in the serving path. Regni’s line is the useful summary: restoring from ADI is the same order of magnitude as GPU memory, and storage is no longer why you keep the KV cache inside the GPU server.

Upstream fixes, not a forked stack

Two engineering notes matter more than the marketing slide.

First, Scality says vLLM’s KV offload connector was fetching about five times the cache blocks each restore needed — a tax anyone paying if they pointed that connector at an object store. They corrected the connector, left the vLLM core alone, and say they committed the fix upstream. Second, they extended the open-source ModelExpress loader to pull weights straight from object storage. The shipping stack, they claim, contains no Scality forks.

If those commits land and stay, every object store behind that connector benefits. If they only live in Scality’s packaging branch, the “open code” claim is softer than it sounds. Ask for the PR links before you treat the stack as a community contribution story.

How I would place this against Leo X and Inferra

Leo X is fabric-attached DDR next to the GPU domain — CXL/PCIe memory semantics, vendor warm-TTFT gains on H200 math. Inferra is NVMe-class prefetch with named L40S TTFT tables vs recompute. Scality is the capacity end of that spectrum: TLC → HDD → optional tape under one namespace, RDMA into HBM, designed so a returning Monday-morning agent fleet does not thrash eight GPUs’ worth of local cache.

They are not substitutes. Leo X wins when you need near-NUMA latency for a hot working set that still fits in tens of terabytes of DDR. Inferra wins when you want a storage-adjacent prefetch path with published L40S numbers. Scality wins when the working set is hundreds of gigabytes per long session and you want any decode GPU in the pool to resume any context without pinning sessions to boxes.

What is still missing for an operator ticket:

Named GPU and matched rematch. Eight unnamed GPUs on 100 Gb/s links is not a Spectrum-X / NVL72 story.
Independent TTFT curves against Leo X, Inferra, and host-DRAM offload on the same model and context lengths.
Failure modes. What happens when a storage server dies mid-restore? How is block checksumming done on the RDMA path?
Power and $/TB for the ADI tier that actually holds the 8 TB (and the 1.9 PB extrapolation).

What ships

Inference Factory is available now as a software license or fully managed service. It deploys on standard Dell, HPE, Lenovo, and Supermicro servers. Harnesses Scality names as compatible include OpenCode, Hermes, Goose, LangGraph, and Pydantic AI; the press kit also says Claude Code and Codex keep working once you retarget the endpoint. ADI already claims 12+ EB in production across 70+ countries as Scality’s existing object fleet — useful as proof they can operate storage at that scale, not as an inference SLO.

If you run disaggregated prefill/decode today and your KV offload still dies on recompute cliffs, this is the October 8 announcement to put on the whiteboard next to Leo X and Inferra — with the lab caveats written in permanent marker.

Anpoo Sivanadi, Staff Software Engineer