An OpenInfer research paper — November 2026
We published The GPU Needs an Operating System, a paper presenting a path to use idle cycles on GPUs smartly.
AI chips are expensive, yet while serving large models they sit mostly idle. Serving Kimi K2 and a 397B-parameter MoE with stock vLLM, the tensor cores were busy only 6 to 11% of the time. The work is there, but it is constantly racing for memory and compute, leaving idle cycles scattered through every step.
Why the chip waits
To answer quickly, a GPU works on only a few requests at a time. For every word it generates, it reads the model’s weights from memory. At small batches it does only a few operations per byte it reads, while the chip can do hundreds. So the math units finish fast and wait.
Every chip has a fixed ratio of math it can do to bytes it can read, called its ridge. GPUs and TPUs fed from HBM or GDDR memory sit between about 240 and 560, so the gap holds however the math units are built. Mixture-of-experts models make it worse: Kimi K2 sends each token to 8 of 384 experts, so even a batch of 256 gives each expert about five tokens of work. Chips built around on-die SRAM read memory tens of times faster, and run short of capacity instead.
Our thesis: scheduling compute and memory at microsecond resolution is the key to real efficiency, not kernels that only look busy.
How fine-grained scheduling works
CPUs solved this decades ago with the operating system: it shares the processor and memory and lets urgent work go first. GPUs have pieces of it, such as SM partitioning, context priorities and green contexts, but none of them solve this problem well enough yet. Each AI program assumes it owns the whole chip, and nothing running can be asked to step aside quickly. We’re building that missing layer on three ideas:
- Price each resource separately. A GPU is not one resource but several: SM time, tensor cores, memory bandwidth, the L2 cache, memory capacity, data links and power. They trade at very different rates. Lending 60 GB of idle memory to another job cost the urgent work nothing measurable, while extra compute beside it cost a lot.
- Spend slack, not hope. Urgent work usually finishes ahead of its deadline. The scheduler spends that margin on background work, without pushing live responses past their latency target.
- Decide at the timescale where the idle time lives. Admission decisions run every few milliseconds. Placement and enforcement run every few microseconds, on the GPU itself.
When background work ran in pieces of 20 µs or less, urgent work barely noticed. At 50 µs pieces, urgent work missed every deadline. Today’s stream priorities don’t interrupt running work at all: a low-priority job slowed urgent work up to 10x.
What we measured
The test: keep a live model decoding on schedule, give background work whatever the GPU leaves idle, and measure two things. How much extra work gets done, and how much slower the live model gets.
We built the first version of our fine-grained resource scheduler, which we call an inference operating system. It runs a whole model inside one long-lived GPU kernel. Whenever the model doesn’t need part of the chip for a few microseconds, the scheduler lends that part to background work and takes it back in time.
Here’s how that compares with the best resource management available today. A fixed partition of the chip, the kind NVIDIA ships as green contexts, got about a quarter of a GPU of extra work done, but slowed the live model by 5%. That’s with background jobs written to share nicely. Ordinary jobs, like an FP8 matrix multiply or LoRA training, got less than half of that, because they fight the live model for cache and memory. On a smaller model, Llama-3.1-8B, our scheduler got the same quarter of a GPU while slowing the live model by only 0.3 to 1.6%. It also generates text for a single user 3% faster than vLLM at its best.
Why this matters
AI is moving to agents, and agents call many models at once and expect fast answers. That pushes every serving system toward small batches, and small batches are exactly where GPUs sit idle. In our measurements, a pool that only generates text kept its tensor cores busy 2.9% of the time. Large deployments already share GPUs with MIG, static partitions and green contexts. But these carve the chip into fixed slices that can’t follow idle time measured in microseconds, so most of it still goes unused.
In our tests, each GPU serving live traffic also ran 0.24 of a GPU’s worth of batch work built to share (about a quarter of a GPU), while live responses got at most 1.6% slower. Across 1,000 serving GPUs, that’s 1,000 × 0.24, the batch work of roughly 240 dedicated GPUs, done on hardware already running. Ordinary jobs that weren’t written to share got less than half of that.
That changes what one cluster can do:
- Enterprises can fine-tune and train on their own data on the same GPUs that serve their users, in the time those GPUs would otherwise sit idle.
- Sovereign and regulated clouds can run evaluation and fine-tuning for local models on the in-country GPUs already serving them, where adding cards is slow.
- Neoclouds and AI labs can fill idle time with evaluation, batch jobs and reinforcement-learning rollouts beside live traffic.
Today’s decode kernels use about half of a GPU’s memory bandwidth, while a kernel built for sharing reached 99% on just 24 of its 188 SMs. Compilers that generate kernels like that, plus three chip features (fast pausing, memory priority and partitions the hardware enforces), would free much more of every GPU.
We measured the limits on NVIDIA Blackwell and AMD MI300X. On both, the chip can’t quickly pause running work to let urgent requests through. The paper specifies that compiler support and the three chip features above, which are needed to go beyond one hand-built model, and we’d welcome working with GPU and accelerator makers on them.
The full paper, with every number and its source, is on arXiv. If you build kernels, compilers, serving systems or silicon, reach us at openinfer.io.