Running Open‑Source LLMs on NVIDIA H100 GPUs with llm‑d

When you start talking to AI agents that actually write code, debug scripts or orchestrate cloud resources, you quickly notice they’re not like the chatbots you saw a couple of years ago. Those agents keep pulling in huge codebases, re‑reading the same files over and over, and sometimes fire off a swarm of sub‑agents that all need answers at the same time. In practice the bulk of the work is not generating text – it’s reading, caching and managing gigantic contexts.

Enter llm‑d, an open‑source framework that grew out of a joint effort between IBM Research, Red Hat, Google and a handful of other industry players. The goal was simple on the surface: give companies a way to run big, open‑weight language models on the hardware they already own, without paying premium prices for commercial APIs. What they ended up with is a stack that actually understands the quirks of today’s “agentic” workloads.

To prove the point, the team took GLM‑5.2 – a mixture‑of‑experts model that carries about 753 billion parameters in total, though only roughly 39 billion are active at any given time – and slapped it onto 544 NVIDIA H100 GPUs. That sounds like a lot of GPUs, but many enterprises already have H100s or H200s in their data centers. Under a test that mimicked real‑world coding assistants – hundreds of sessions running in parallel, each reusing massive chunks of context – the deployment cranked out more than 6.6 million tokens per minute at its peak, handling up to 3 000 concurrent coding agents without a single pre‑emption.

What makes that impressive isn’t just raw speed; it’s the cost angle. At today’s cloud‑instance rates, running GLM‑5.2 on H100s with llm‑d works out to be five to ten times cheaper per token than calling the same capability via a commercial API. The savings are especially noticeable for input‑heavy traffic, which is exactly what coding assistants generate.

Why do agentic workloads behave so differently from a simple chatbot? A prior study of 219 real‑world Claude Code sessions showed that a typical request carries about 195 000 tokens but only spits out roughly 317 tokens in response. In other words, the model spends the lion’s share of its compute time just reading context – entire repositories, configuration files, dependency trees – and only a tiny fraction actually “writes”. The researchers distilled three defining traits: ultra‑long contexts, massive reuse of earlier inputs, and sudden bursts of parallel sub‑agent activity.

Armed with that insight, llm‑d layers six tricks on top of each other. First, prefix‑aware routing makes sure a request lands on the server that already holds the relevant cached context, cutting the need for redundant recomputation. On the CyberGym benchmark, switching from a coarse‑grained router to precise prefix matching boosted throughput by almost 80 % and slashed time‑to‑first‑token by two‑thirds.

Next, a tiered KV‑cache system pushes older but still‑useful prefixes out of precious GPU memory and into CPU DRAM, where they can linger without being evicted. If another node already has the exact prefix you need, a peer‑to‑peer cache‑share fetches it directly, again avoiding a costly recompute. Together, these cache‑centric ideas keep the GPUs busy with fresh work instead of re‑doing the same reads over and over.

Because a 753 billion‑parameter model can’t live on a single machine, llm‑d uses “wide expert” parallelism. Instead of replicating the whole KV‑cache across tensor‑parallel shards – which would waste memory – it distributes the expert layers across nodes while keeping the attention mechanism data‑parallel. On top of that, the system splits the inference pipeline into two disaggregated pools: one for the heavy‑weight pre‑fill (context processing) and another for the decode (token generation). These pools talk to each other via NVIDIA’s NIXL zero‑copy library, meaning data moves fast and never drops a beat.

Finally, multi‑token prediction (MTP) lets the model spit out several tokens in a single forward pass. When you combine MTP with the cache tricks and the split pools, the performance gains stack up, delivering the kind of interactive latency that developers expect from a local IDE helper.

“llm‑d isn’t just a laundry list of features,” says Carlos Costa, Distinguished Engineer at IBM Research. “It’s about all of them working together, on hardware you already have, to serve real‑world agentic traffic at scale.”

In short, the demonstration shows that enterprises don’t need to throw money at a new cloud service to run massive open models. With the right software – llm‑d – the H100 GPUs humming in their racks can handle today’s demanding AI agents, keep costs in check, and keep proprietary code where it belongs: on‑premises.