Three reinforcement-learning workloads can fit the same model server and still fight for GPU memory in completely different ways. One emits a very long answer, one carries a long prompt through several tool calls, and one looks balanced until concurrency rises. A router that sees only prefix locality can place each request sensibly and still push the whole serving pool into KV-cache pressure.
MISA-T is a research scheduling policy for that mixed-rollout problem. It combines admission control, workload-aware KV-capacity allocation and accounting for how long retained state remains resident. The practical buyer lesson is not that every RL cluster needs this exact scheduler. It is that rollout capacity cannot be sized from model weights and token throughput alone when RLHF, RLVR and agentic sessions share the same inference fleet.
The paper's authors report higher rollout throughput and lower training iteration time on their tested H200 and H100 systems. Those figures come from the research team, not from GPUMachines testing. The experiments also show how a placement policy with a high prefix-hit rate can fail once permissive concurrency admits too many sessions. Memory admission becomes a first-order scheduling decision.
The buyer answer
If a team runs one predictable rollout workload at a time, standard cache-aware routing with a measured concurrency limit may be enough. Do not add a complex mixed-workload controller before the traffic justifies it.
If the same inference pool serves RLHF, verifiable-reward training and multi-turn agents, require a trace-driven capacity test before buying more GPUs. Measure prompt length, generated length, session turns, tool-wait time, retained KV state, prefix reuse and completion rate by workload class. Then test whether the router can hold new sessions back when memory is committed but temporarily idle.
This decision often determines deployment shape. A team with uncertain traffic may learn more from GPUMachines GPU Cloud before committing to a fixed cluster. A team that needs dedicated hardware and stable tenancy can compare a Buy & Host deployment with on-premise ownership. Hardware capacity matters, but the useful configuration depends on admission and scheduler behaviour.
Why mixed rollouts are not one queue
The MISA-T paper describes three workload shapes from its trace.
RLVR requests tended to start with short inputs and generate long outputs. KV memory grows during decode, so an admission decision made from the initial prompt can underestimate the eventual footprint.
RLHF requests were more balanced, with medium input and output lengths in the paper's classification. They still vary, but their memory path is less extreme than the longest RLVR samples.
Agentic sessions carried long inputs, often produced shorter outputs, and returned over several turns. Their KV state could remain useful while an external tool or orchestrator performed work. That state occupies memory even when the GPU is not generating tokens for the session.
The paper's reported trace is striking. Under a 128K context setting, the agent workload had an input-token median above 37,000 and a 99th percentile above 123,000, while median output was only 148 tokens. The RLVR workload had a median input of 156 tokens and a median output above 47,000, with the upper percentiles near 131,000 output tokens. RLHF sat between those shapes. These are characteristics of the authors' trace, not sizing defaults for other organisations.
A single "average tokens per request" erases the difference. Two requests with the same total token count can put very different pressure on prefill compute, decode occupancy, KV growth and scheduler residency.
Prefix locality is only part of the answer
Cache-aware routing tries to send a request to a replica that already holds reusable prefix state. That can avoid repeated prefill work. The approach is useful, but it can concentrate sessions on the same replicas.
Under light load, the trade can look excellent: high prefix reuse and good latency. At high admission rates, the chosen replicas approach KV capacity, evictions rise, and the prefix benefit collapses. Other replicas may have free compute or memory but lack the warm prefix.
In one high-load Qwen experiment, the paper reports that permissive concurrency reduced request throughput by 52.5 per cent, prefill token rate by 57.3 per cent and decode token rate by 38.5 per cent compared with a sweep-tuned vLLM Router setup. Prefix hit rate fell from 92.4 per cent to 4.5 per cent. The point is not the exact percentage outside that setup. It is the failure mode: placement can be locally reasonable while admission makes the cluster globally unstable.
This extends the scheduling discussion in the GPUMachines article on agentic AI and KV-cache routing. Session affinity, load balance and memory admission have to be planned together. A routing score cannot recover memory already promised to too many growing sessions.
What MISA-T changes
MISA-T operates at the routing layer. The trainer or workload controller maintains the desired mixture of RLHF, RLVR and agentic samples. The router decides when a new session can enter and which serving replica should receive it.
The policy has three linked ideas.
Adaptive session admission
New sessions are not admitted only because a replica has a free execution slot. The controller can hold them and re-evaluate as memory conditions change. Existing session continuations receive protection because abandoning their resident state may waste prior computation and disrupt the target workload mixture.
This is the part buyers should notice. Concurrency is not a harmless front-end setting. It is a claim on future KV memory whose size may be uncertain at admission time.
Workload-aware capacity allocation
Different workload classes receive capacity treatment that reflects their expected memory path. A long-decode RLVR request and a multi-turn agent should not be budgeted from the same fixed estimate. The system uses labels and observed behaviour to make a more informed allocation.
Labels are also a dependency. If traffic is misclassified or the workload changes, the estimates can be wrong. A production system needs monitoring by class and a fallback policy for unknown traffic.
Residency-time-aware accounting
Agent state can remain in KV cache while the session waits for a tool. That memory is not visible in token-generation throughput at that moment, but it is not free. MISA-T accounts for retained state across the residence period rather than treating a waiting session as absent.
The paper reports tool or orchestration gaps of 6.6 per cent for one Step3.7 workload and 22.7 per cent for a Qwen workload. Again, these are trace-specific. They demonstrate why GPU utilisation alone can understate committed memory pressure.
What the experiments show
The end-to-end experiment used the Step3.7 sparse mixture-of-experts model, described as 196B total parameters with 11B active parameters. Six H200 nodes handled training and four H200 nodes handled rollout, with tensor parallelism of eight on each inference node. The authors matched 50 iterations between the compared approaches.
They report 35.6 per cent higher rollout throughput, 22.8 per cent lower mean iteration time, a prefix hit-rate change from 74.5 to 96.2 per cent and a reduction in workload-mixture deviation from 4.14 to 2.71 percentage points. Task scores were described as comparable.
Rollout-only tests used two H200 inference nodes for Step3.7 and sixteen H100 80 GB GPUs arranged as four tensor-parallel replicas for Qwen3.6-35B-A3B. Across three runs, the paper reports sample-throughput gains of 53.3 per cent and 43.6 per cent against a sweep-tuned router, with prefix-hit rates of 97.8 and 95.3 per cent.
The researchers also tested CPU KV offload with 1.3 TB of host capacity and a backup copy of GPU-resident KV. Under the same MISA-T concurrency policy, GPU KV use remained at or above 90 per cent for much of the interval and mean per-replica requests per minute rose by a reported 35.6 per cent.
These results support further evaluation. They do not establish that CPU offload always improves performance, that the same policy transfers to every model server, or that the reported gains will survive different tool delays and context distributions. Network and host-memory costs become part of the design once KV moves off GPU.
Hardware consequences
GPU memory sets the admission envelope
Model weights, activations, runtime workspaces and KV cache all compete for high-bandwidth memory. A 141 GB H200 provides a larger memory envelope than an 80 GB H100, but a larger envelope does not remove the need for admission. Longer contexts and higher concurrency can consume any fixed capacity.
Size against a percentile distribution, not only an average. Decide what happens beyond that percentile: reject, queue, truncate, spill to host, route to another pool or use a smaller model. Each choice changes service behaviour.
Host RAM can become a cache tier
CPU offload may increase usable capacity, but it creates copies, transfers and eviction policy. Host memory bandwidth, NUMA placement and CPU-to-GPU path matter. If a node has several GPUs and NICs, poor affinity can turn offload into a latency source.
Do not specify host RAM only for the operating system and dataset preprocessing when the serving design includes KV offload. Define how much is reserved, whether it is pinned, how it is protected from other services and what happens when the host tier fills.
Network design depends on cache movement
Local GPU memory has the shortest path. Remote cache, disaggregated prefill and decode, or shared KV tiers add network traffic. The right fabric depends on bytes moved per request, latency tolerance and the number of replicas. The headline GPU count cannot answer that question.
Measure east-west traffic with the actual model server. A 400 GbE or InfiniBand link does not guarantee useful application bandwidth when transfers are fragmented, topology is wrong or host copies dominate.
CPU cores are part of the rollout system
Tokenisation, request admission, routing, reward evaluation, tool orchestration, logging and data movement all use CPU. Sparse GPU utilisation during tool waits can coexist with a busy control plane. Track queue delay and orchestration time separately from model execution.
A trace-driven sizing method
Before selecting hardware, collect a representative trace with these fields:
- workload class and model;
- prompt tokens and generated tokens;
- context reused from a prior turn;
- session identifier and turn number;
- prefill time, decode time and end-to-end completion time;
- tool-wait or reward-evaluation time;
- peak KV bytes and retained KV duration;
- cache hit, eviction and recomputation events;
- serving replica and GPU-memory occupancy;
- queue, rejection and retry outcomes.
Then replay the trace at several arrival rates. Increase load until one of the buyer's service limits fails. That might be rollout completion time, training iteration time, mixture accuracy, queue delay or task success. Do not stop at GPU utilisation. A router can show busy GPUs while wasting work through eviction and recomputation.
Compare at least four policies:
1. load-only routing with a conservative concurrency limit; 2. prefix-aware routing with tuned concurrency; 3. class-aware admission with protected continuations; 4. the intended offload or distributed-cache policy, if any.
Keep model, trace and quality settings fixed. Report useful samples completed per hour, not only tokens per second. For reinforcement learning, one very long failed sample can consume substantial capacity without advancing the training objective.
When to separate the pools
Mixed scheduling is not always better than physical separation. Separate pools may suit teams with strict service levels, different model versions, sensitive tenants or workload classes that peak at different times. Isolation also makes cost attribution easier.
A shared pool is attractive when demand varies and spare capacity can move between classes. It needs stronger telemetry and a controller that protects the target mixture. The saving comes from better utilisation, but the operational state becomes harder to reason about.
Run both designs against the same trace. Include the cost of stranded capacity in separate pools and the cost of controller complexity in a shared pool. The right answer may be a hybrid: a reserved agent pool, a long-decode pool and a shared overflow tier.
The GPUMachines article on LLMVisor and multi-tenant inference latency adds another buyer concern: once requests share batches and memory, teams need fair attribution as well as high aggregate throughput.
What MISA-T does not solve
The paper assumes workload labels. A production router must obtain, verify and update those labels. Unknown or adversarial traffic can defeat class estimates.
The controller also depends on timely serving snapshots. Delayed or incomplete telemetry can make capacity accounting stale. The operations team needs to know the age and confidence of the state used for admission.
MISA-T retains an outer concurrency limit rather than solving global concurrency from first principles. Buyers still need a tuning process and a protection limit for each model and topology.
The evaluation comes from the system's authors and uses selected models, traces and hardware. GPUMachines has not independently reproduced it. Treat the reported gains as a reason to test admission policy, not as a capacity multiplier for a purchase order.
Who should consider it
The approach fits organisations training models with several online reinforcement-learning workloads sharing an inference fleet. It is especially relevant when agent sessions retain long prefixes, RLVR samples decode for a long time, and rollout delays hold up expensive training GPUs.
It is less relevant to straightforward chat inference, a single low-concurrency research job or a team that can schedule workload classes in separate time windows. A smaller system with a simple queue may be cheaper to operate.
Teams early in experimentation should avoid locking the design to one trace. Use hosted capacity to collect data across model and prompt changes, then decide whether the steady workload supports dedicated servers.
What GPUMachines can review
GPUMachines can map rollout traces to GPU memory, host RAM, local NVMe, CPU, network and deployment options. The review should include the training-to-rollout ratio, tensor-parallel arrangement, target context length, peak session count, tool-wait distribution and recovery behaviour.
For dedicated platforms, the design may compare high-memory HGX systems with flexible PCIe GPU servers, or split training and rollout into different node types. For hosted deployment, it can compare shared cloud capacity with dedicated Buy & Host equipment. The best route depends on utilisation evidence and how much scheduler engineering the team can own.
Bring a trace rather than a GPU wish list. A week of representative request data will usually reveal more about the correct platform than a peak-token estimate.
FAQ
Is MISA-T a replacement for vLLM Router?
The paper presents a routing-layer policy and compares it with a tuned vLLM Router configuration. Production adoption would require integration and validation with the serving stack in use.
Why can prefix-aware routing reduce throughput?
It can concentrate warm sessions on a few replicas. If admission allows too many sessions, KV pressure causes eviction and recomputation, erasing the locality benefit.
Does more GPU memory remove the problem?
It raises the capacity ceiling but does not remove it. Workload length and concurrency can grow until the new ceiling is reached.
Should agent KV state stay on GPU while tools run?
Only if the expected reuse justifies the residence time. The decision depends on tool delay, memory pressure, transfer cost and the penalty for recomputing the prefix.
Is CPU KV offload always worthwhile?
No. It trades scarce GPU memory for host RAM, bandwidth, copy overhead and more policy. Test completion time and quality under real load.
What metric should procurement use?
Use useful rollout samples completed within the training target, alongside iteration time, queue delay, cache hit rate, eviction, mixture deviation and task quality.
Sources and Further Reading
- MISA-T research paper
- vLLM Router project repository
- vLLM PagedAttention design documentation
- NVIDIA H200 product information
Verdict
MISA-T shows why mixed reinforcement-learning rollouts need memory admission as well as cache-aware placement. RLVR, RLHF and agentic sessions make different claims on prefill compute, decode time and retained KV state. A scheduler that ignores those differences can lose throughput even when its placement logic looks sensible.
The reported gains are promising but remain tied to the authors' traces and systems. Buyers should not multiply a hardware quote by the paper's percentages. They should run their own trace through tuned placement, admission and offload policies, then size the platform from completed useful samples and training iteration time.
Use GPUMachines GPU Cloud to collect rollout evidence, or discuss a dedicated Buy & Host platform once the workload is stable enough to own.