Two tenants can share the same GPU batch, receive different service, and still leave the operator with one lump of elapsed time. Token counts alone do not explain who caused what.
LLMVisor is a research proposal for breaking that batch latency into additive per-request shares while an inference scheduler is running. Its authors combine a small roofline-informed model with short profiling runs, then estimate the contribution of prompt work, context-dependent KV-cache traffic, attention cost and batch effects.
The paper does not deliver a finished billing platform. It offers a missing measurement primitive for providers that want fractional GPU services without surrendering batching efficiency.
The buyer answer
A shared inference service should not bill, throttle or promise latency using tokens alone. Input length matters, but prefill and decode stress the GPU differently; context length, batch composition, model architecture and tensor-parallel layout can change the cost of a request.
Before buying more GPUs for a multi-tenant service, ask whether the current platform can attribute latency and resource use to a request or tenant. If it cannot, the team may be responding to noisy neighbours, poor admission policy or a badly mixed batch by adding hardware that the scheduler will still use unfairly.
LLMVisor is worth following when all of these conditions apply:
- Several customers or internal teams share an inference engine.
- Continuous batching is needed for acceptable throughput.
- Tenants receive quotas, service targets, chargeback or usage reports.
- Workloads vary in prompt length, context history and generated output.
Dedicated GPU nodes remain simpler when isolation and predictable capacity matter more than packing efficiency. Software attribution should not be sold as equivalent to hardware isolation.
What the research actually covers
LLMVisor was submitted to arXiv on 9 August 2026 by researchers from the University of Michigan and ByteDance. The team integrated a prototype into vLLM 0.7.3 and evaluated it on H100 SXM and A100 SXM servers.
The tested models were Llama 3.1 8B and Qwen 2.5 14B/32B. Tensor parallel configurations ranged from one to four GPUs on the H100 system and one to two on the A100 system. Synthetic workloads used 128 to 2,048 concurrent requests, with total tokens rising to 90 per cent of the engine's KV-cache capacity and high variation between request lengths.
Against a token-based Virtual Token Counter baseline, the authors report lower high-percentile relative error for both prefill and decode. The largest headline improvements were up to 2.5 times at p90 and 3.3 times at p99 for prefill, and up to 3.5 times and 4.4 times for decode. They also report microsecond-scale evaluation in the scheduling path.
Those results belong to the paper's models, runtime version, hardware and generated workloads. GPUMachines has not reproduced them. A provider should treat LLMVisor as a promising attribution method to validate, not a ready-made accuracy guarantee.
Why tokens are a poor accounting unit
A token looks tidy on an invoice. The GPU does not experience every token in the same way.
During prefill, the engine processes the prompt and builds KV-cache state. Self-attention cost grows with prompt structure and length, while matrix work can push the device towards a compute-bound regime. One request with a long document can contribute much more work than several short prompts.
During decode, each active sequence usually produces one token per step, but the engine must read the sequence's existing KV cache. Two requests generating one token at the same moment can have very different context lengths and memory traffic. Batch size also changes how efficiently the GPU runs.
A simple counter may charge one unit to each generated token while ignoring that one request carries a 32,000-token context and another carries 500. It may also miss the fixed work shared by the batch.
LLMVisor models prefill and decode separately. Its fitted features represent prompt tokens processed in the current step, context tokens whose KV state must be read, the quadratic component of attention and the effect of batch size. Because the model remains linear in sums of per-request terms, it can allocate the predicted batch latency back to requests and then tenants.
The attraction is not mathematical neatness. Additive attribution gives a scheduler something it can enforce.
What better attribution could change
Admission control can use cost, not queue length
An operator may have ten requests waiting, but count alone says little. One large prefill can consume more time than a group of short interactive prompts. A cost estimate lets the scheduler decide whether adding a request would push a tenant or batch beyond its budget.
That can support deferral, a separate queue, a lower concurrency limit or a different service tier. It may also prevent a long-context customer from damaging everyone else's time to first token.
Quotas can follow estimated GPU time
Rate limits based on requests per minute reward users who submit enormous prompts and penalise users who send short ones. Token quotas improve that picture but still flatten context and batching effects.
LLMVisor's proposed unit is attributed latency within the batch. Aggregated by tenant, it could support a fractional share of engine time. A platform may then offer a base reservation with controlled bursting, provided the attribution remains stable under the production workload.
That is a scheduling concept, not an instruction to put model predictions directly on an invoice. Commercial billing also needs auditability, dispute handling, version control, rounding rules, availability definitions and tax treatment. A research estimator is one input.
Capacity planning can test workload mixes
Average GPU utilisation can look healthy while one tenant repeatedly drives tail latency. With per-request cost estimates, a platform team can ask what happens if a large-document workload shares a replica with a short conversational service.
If the mix behaves badly, the remedy may be workload segregation rather than another GPU. Place long prefills on a separate pool, split prefill and decode, or assign a model replica to customers with similar service targets.
The GPUMachines article on SMetric and agent scheduling covers a related problem: session-aware cache reuse can improve efficiency while concentrating load on the wrong workers. LLMVisor adds a different question, namely how much of a shared step each request should own.
Attribution is not isolation
This distinction should appear in every buying conversation.
Software co-batching places requests from several tenants inside one inference engine. Better attribution can support fairness and accounting, but it does not automatically create security boundaries, memory isolation or a fixed amount of compute.
NVIDIA Multi-Instance GPU partitions supported devices into instances with dedicated compute and memory paths. MIG can provide stronger performance and fault isolation, though each partition has a fixed profile and large models may not fit the available slice. Dedicated physical GPUs go further by removing co-resident tenant workloads, at the cost of lower packing efficiency when demand is light.
Choose among shared batching, MIG, dedicated GPUs and dedicated servers according to the actual promise:
- Shared batching suits high aggregate throughput where software quotas and measured service are acceptable.
- MIG suits supported models that fit a partition and require firmer hardware separation.
- Dedicated GPUs suit larger models or tenants that need predictable accelerator ownership.
- Dedicated servers suit strict operational separation, custom networking or customer-controlled software.
The GPUMachines Buy & Host service is relevant when a buyer wants owned hardware in a managed facility. Shared GPU Cloud capacity fits shorter commitments and variable demand. Neither route should be chosen without a clear isolation and service definition.
What to measure before adopting an attribution model
vLLM already exposes useful service metrics, including time to first token, inter-token latency, end-to-end latency, queue time, prefill time, decode time, prompt and generated token counts, running and waiting requests, request completion and KV-cache use.
Collect these before adding another estimator. They establish whether the service has a real attribution problem or a simpler capacity fault.
The production trace should retain, with suitable privacy controls:
1. Tenant or service-class identifier. 2. Model, precision and tensor-parallel configuration. 3. Prompt tokens, generated tokens and context length at each decode step. 4. Batch membership and scheduler decision timestamps. 5. Queue time, time to first token, time per output token and end-to-end latency. 6. KV-cache occupancy, preemption and prefix-cache behaviour. 7. GPU compute, memory activity, power and errors. 8. Request outcome, cancellation and retry reason.
Do not retain raw prompts merely because the scheduler team wants easier debugging. Metadata may be enough, and customer content needs its own retention, access and deletion policy.
A production validation plan
An operator should build one shadow measurement path before giving attribution control over live scheduling.
First replay representative traffic across every service class. Include short chat, long documents, agent sessions, bursty batch jobs and the longest supported context. Synthetic traffic can fill gaps, but it should not replace the production distribution.
Fit the model separately for each meaningful combination of GPU type, model, quantisation, runtime version and tensor-parallel layout. LLMVisor uses short profiling runs because coefficients depend on the platform. A fit from H100 and Qwen 32B should not govern B200 and another model family without evidence.
Then compare predicted batch latency with measured step latency. The harder test is whether per-tenant shares remain stable when batch composition changes. A system can predict the batch total well and still divide it unfairly.
Keep the estimator in shadow mode long enough to observe model rotations, software upgrades, traffic peaks and rare long-context requests. Define a drift threshold and a fallback policy. If attribution confidence falls, the scheduler should revert to a conservative quota rather than silently issuing questionable charges.
Only after that should the team test admission decisions. Start with a small tenant group and compare tail latency, throughput, rejected work, utilisation and customer outcomes against the previous policy.
Where LLMVisor may fall short
The published evaluation used three models and two older accelerator generations. It did not show GB200, B200, B300, RTX PRO, AMD Instinct or heterogeneous GPU pools. New attention kernels, speculative decoding, prefix caching and disaggregated serving can change the workload seen by the estimator.
The paper's workloads were generated rather than taken from a named production service. They created useful variation and high KV-cache pressure, but customer traffic can contain tool calls, repeated prefixes, cancellations, structured output and multimodal inputs that need separate testing.
LLMVisor also attributes predicted latency, not every business cost. It does not allocate host CPU, RAM, network traffic, storage, model-loading time, idle reserve, failed requests or support work. A provider building chargeback needs a wider cost model.
Finally, fairness remains a policy choice. Accurate attribution cannot decide whether a medical request, an internal coding agent and an overnight batch job deserve equal priority. The platform owner must define service classes before the scheduler can enforce them.
Who should consider this approach
GPU cloud operators, enterprise private-AI teams and research platforms with several departments on one serving fleet have the clearest use case. The method also suits software vendors offering reserved inference shares without assigning a complete GPU to every customer.
Teams building a private agent fleet may need it when agent sessions vary sharply in context and tool history. One long-running agent can create much more decode memory traffic than a short support exchange, even if both return similar output lengths.
Small teams with a single application should start with ordinary observability and a simpler queue. A dedicated GPU with predictable demand may not need per-tenant attribution at all. Do not install scheduler research merely to make a dashboard look advanced.
Questions for a hosted inference supplier
Ask how the provider defines a billable unit and whether prompt, output, cache hits and retries count differently. Request the service target for queue time, time to first token and inter-token latency, including the percentile and measurement window.
Find out whether tenants share a process, a GPU, a MIG instance, a server or only a cluster. “Dedicated endpoint” does not always mean dedicated hardware. Ask what prevents another tenant's long prefill from affecting your requests.
The provider should explain how model upgrades alter capacity and whether quotas change when the runtime, precision or batching policy changes. Request a path for investigating disputed usage. If the answer is only “we count tokens”, the accounting model is probably too coarse for a serious reserved service.
How GPUMachines can help
GPUMachines can compare a shared inference pool with MIG, dedicated PCIe GPUs, HGX systems and hosted capacity. The useful input is a traffic trace containing request shapes and service outcomes, not customer prompt text.
We can review model memory, context distribution, concurrency, latency targets, tenant count, networking and growth, then separate scheduling problems from genuine GPU shortage. For a new service, the design can include an observability baseline and acceptance criteria before hardware arrives.
If shared batching is commercially sensible, buy enough capacity for failures and peaks rather than assuming attribution will remove every idle period. Fractional allocation improves packing; it does not repeal queueing.
Frequently asked questions
Is LLMVisor a scheduler?
It is a latency attribution model intended to run inside a scheduler. The paper discusses admission, accounting and planning uses, but attribution alone does not define the complete scheduling policy.
Does it measure actual GPU time for each request?
It predicts additive latency shares using fitted features linked to compute and memory traffic. The evaluation compares those predictions with profiled step latency. It is not the same as physically running each request alone and timing it.
Why is decode harder to attribute than prefill?
Decode cost depends on context length, KV-cache traffic, sequence divergence and the active batch. Counting one generated token per request misses those differences.
Can LLMVisor support customer billing?
Potentially as one measurement input after validation. A billable service also needs contracts, auditable records, versioned coefficients, fallback rules and a wider treatment of non-GPU costs.
Does MIG solve the same problem?
No. MIG partitions supported GPU resources for stronger isolation. LLMVisor tries to account for requests sharing one inference engine. A provider may choose either approach or use both at different service tiers.
Will its published accuracy apply to B200 or B300?
That has not been established. The paper evaluated A100 and H100 systems with named Llama and Qwen models. New hardware and runtimes require fresh profiling and validation.
Is token-based accounting always wrong?
It can be adequate for simple public pricing or a homogeneous workload. It becomes less defensible when long contexts, mixed models, co-batching and reserved shares create large differences between tokens.
Sources and Further Reading
- LLMVisor research paper by Jin and colleagues
- vLLM production metrics documentation
- NVIDIA Multi-Instance GPU introduction
- USENIX OSDI paper on fairness in LLM serving and VTC
Verdict
LLMVisor asks the right question for shared inference: if requests run together, how can the operator assign responsibility without destroying the throughput benefit of batching?
Its answer is technically credible and usefully small, but still early. The paper shows that prompt tokens, context traffic and batch effects can beat a simple token counter on its A100/H100 evaluation. It does not yet prove production billing accuracy across current GPU fleets.
For buyers, the next move is measurement. Capture request shape, batch membership, KV-cache pressure and latency by service class. If noisy neighbours or blunt quotas are driving hardware growth, test attribution and workload segregation before adding replicas. If demand still exceeds a well-run pool, GPUMachines can size shared, partitioned or dedicated capacity through the GPU Cloud service.