Running Open‑Source LLMs on H100 GPUs with llm‑d

When you ask a modern coding assistant to dig through an entire codebase, it isn’t just spitting out a few lines of text. It’s scanning megabytes of source, re‑using that context over and over, and sometimes spawning tiny helper agents that all fire off at once. In other words, the workload is less about “talking” and more about relentless reading and juggling of massive prompts.

That shift threw a wrench into the old inference playbook. Traditional chatbots could get away with a single GPU and a simple request‑response loop. Agentic systems – the new‑fangled AI assistants that write code, draft contracts, or even design graphics – demand a different kind of engine: one that can cache, reuse, and shuffle huge token streams without choking on latency or ballooning costs.

Enter llm‑d, an open‑source inference framework that grew out of a collaboration between IBM Research, Red Hat, Google and a handful of other AI veterans. The goal was plain: let you run gigantic, open‑weight models on the hardware you already own, and do it cheaply enough that you don’t have to lean on pricey third‑party APIs.

To prove the point, the team took GLM‑5.2 – a mixture‑of‑experts (MoE) model with about 753 billion parameters (roughly 39 billion active at any time) – and spun it across 544 NVIDIA H100 GPUs. Those H100s are the workhorse accelerators you’ll find in many enterprise data centers and public clouds, so the experiment was very much a “real‑world” test.

What happened next felt almost cinematic. With a workload that mimicked a production‑grade fleet of coding agents – hundreds of sessions, heavy context reuse, and occasional bursts of parallel sub‑agents – the deployment churned out more than 6.6 million tokens per minute at peak. That translated into serving up to 3,000 concurrent coding agents without a single pre‑emptive pause.

But the numbers that matter most to a CFO are the cost metrics. At today’s cloud‑spot rates, running GLM‑5.2 on H100s with llm‑d was between five and ten times cheaper per token than the best commercial API alternatives. The savings grew even larger on input‑heavy patterns, which is exactly what agentic workloads look like.

How did llm‑d pull this off? It’s a blend of six tightly‑coupled tricks, each tackling a piece of the “context‑re‑processing” puzzle.

1. Prefix‑aware routing. Because a lot of the work is just re‑reading the same code, the system sends each request to the GPU that already holds the relevant cached prefix. In benchmark runs, swapping to precise prefix matching boosted throughput by 79 % and slashed time‑to‑first‑token by two‑thirds.

2. Tiered KV‑cache. The key‑value cache that powers transformer attention lives not just in GPU memory but also spills over into CPU DRAM. That way, hot prefixes survive memory pressure instead of being tossed out and recomputed.

3. Peer‑to‑peer cache sharing. If the best cache lives on a sibling node, the system pulls it over the network instead of re‑doing the work locally. It’s a bit like borrowing a neighbor’s notes instead of re‑writing them.

4. Wide expert parallelism. MoE models like GLM‑5.2 are split across many experts. llm‑d spreads those experts over nodes using data‑parallel attention, sidestepping the massive KV‑cache duplication that classic tensor parallelism would force.

5. Prefill/decode disaggregation. Context loading (prefill) and token generation (decode) are pulled into separate pools that scale independently. They talk to each other via NVIDIA’s NIXL zero‑copy library – meaning no data‑moving glitches were observed during the full benchmark.

6. Multi‑token prediction (MTP). Instead of emitting one token per forward pass, the model predicts several at once, cranking up output throughput especially when many agents are active.

When you stack all these optimizations, the result is more than the sum of its parts. “llm‑d isn’t just a laundry list of features,” says Carlos Costa, distinguished engineer at IBM Research. “It’s how they all play together on hardware you already have.”

So what does this mean for enterprises? If you’ve already invested in a fleet of H100s (or even older GPUs), you now have a clear path to run cutting‑edge, open‑weight LLMs at scale, keep your proprietary data in‑house, and avoid the per‑token price tags that come with hosted services. The proof‑of‑concept is in the data – massive models can be split, cached, and served efficiently, all while staying under the radar of runaway costs.

Looking ahead, the llm‑d team plans to keep polishing the framework, adding more diagnostics and tighter integration with popular orchestration tools. For anyone watching the AI‑infra landscape, the takeaway is simple: you don’t need a brand‑new GPU rack to run today’s agentic workloads – you just need the right software stack to make the most of what’s already there.