A benchmark number can be perfectly accurate and still answer the wrong question.
Suppose an LLM serving stack becomes 2.5 times faster after a move to INT4 weights and a different inference engine. It is tempting to credit quantisation, because that is the obvious change in the model artefact. Yet the new stack may also use continuous batching, paged KV-cache management, a different matrix-multiplication kernel, changed sampling and a scheduler that keeps the GPU busier. The final ratio bundles all of it.
A paper submitted to arXiv on 13 July 2026 tries to pull those effects apart. Weijia Han and Lisha Qu study LLM inference on four NVIDIA RTX A5000 GPUs, using a matched FP16 intermediate between a Hugging Face-based reference stack and vLLM with the Marlin INT4 kernel. Their work also compares one sharded model instance with several smaller instances routed independently.
GPUMachines hasn't reproduced these measurements. Every numerical result below belongs to the authors and their stated setup. The paper uses one server, an older professional GPU generation and a small synthetic prompt pool, so it isn't a buying benchmark for current hardware. It is useful for a different reason: it demonstrates how easily a whole-stack speedup can be mislabelled.
Executive Summary
- The study separates an inference-runtime effect from the combined kernel-and-quantisation effect by inserting a vLLM FP16 baseline.
- Under its matched wide-batch measurement, the authors report a 2.58 times end-to-end gain, with roughly two thirds of the logarithmic speedup assigned to the runtime change.
- Four-way tensor parallelism reached about 1.5 times single-GPU throughput in the tested setup, far below linear scaling.
- Independent instances won some workloads, while a sharded instance won others; the larger tested model reversed the smaller model's result.
- INT4 increased sustainable concurrency before the tested 24 GiB devices hit their FP16 memory limit, but quality still needs separate evaluation.
Buyers can use GPU Cloud for a controlled pilot, compare PCIe GPU servers with HGX systems, or ask GPUMachines to review a dedicated Buy & Host deployment.
The Tested Machine and Software Stacks
The physical host contained eight RTX A5000 cards, each with 24 GiB of memory. The authors used four cards for a tensor-parallel configuration. Cards were connected as two NVLink-bridged pairs, while traffic between pairs crossed PCIe. CUDA 12.4 ran on the host.
The comparison used Llama 3.1 8B Instruct in FP16 and GPTQ INT4 forms. The paper describes the FP16 footprint as roughly 16 GiB and the GPTQ INT4 footprint as roughly 5.7 GiB in that setup. A 70B quantised model appears in the larger-model routing comparison.
| Stack | Runtime and scheduler | Weight and kernel path | Why it exists in the study | | --- | --- | --- | --- | | HF-FP16 | Hugging Face ecosystem with SwiftServe batching | FP16 | Reference stack | | vLLM-FP16 | vLLM asynchronous engine with PagedAttention | FP16 | Matched intermediate that changes the runtime without adding INT4 | | vLLM-Marlin | vLLM asynchronous engine | GPTQ INT4 with Marlin | Full optimised stack |
That middle row does the real analytical work. Comparing only the first and third rows can't show whether the gain came from the runtime, weight format or kernel. The FP16 vLLM result creates an intermediate ratio. The paper treats the split as accounting, not proof that the effects act independently.
This qualification matters. Runtime and kernel behaviour can interact. A kernel that looks modest at low load may matter differently once batching fills the device, while a runtime can expose more work to the GPU and change which kernel property dominates. The decomposition helps describe the measured gap; it doesn't turn a coupled system into isolated parts.
Why Runtime Can Outweigh the Quantised Kernel
An inference runtime decides how requests queue, how KV cache is allocated, when batches form and how kernels reach the GPU. Those choices change utilisation before the arithmetic format enters the conversation.
At low concurrency, a GPU can spend time waiting between small pieces of work. As concurrency rises, a capable runtime combines requests and keeps more operations in flight. The paper reports that the runtime factor rose across its concurrency sweep while the kernel-and-quantisation factor narrowed. In other words, the scheduler and memory-management machinery accounted for more of the observed gain at the wider operating point.
This doesn't diminish INT4. Smaller weights reduce memory traffic and leave more VRAM for KV cache or concurrent requests. Specialised kernels can also execute quantised matrix operations efficiently. The point is narrower: "INT4 was faster" doesn't explain a benchmark if the runtime changed at the same time.
For a buyer, this affects hardware sizing. A poorly matched serving stack can make an expensive GPU appear underpowered. Conversely, a runtime change may delay the need for another card. Neither outcome should be assumed from a blog benchmark; test the intended model, prompt lengths, output lengths and concurrency.
The Paper's Headline Result, With Its Limits Intact
Under matched greedy decoding at the wide-batch operating point, the authors report a 2.58 times end-to-end speedup between the reference and the vLLM-Marlin stack. Their logarithmic-share method assigns about two thirds of that gain to the runtime change, with the remainder assigned to the kernel-and-quantisation step.
The study also presents a direct production-stack comparison where the apparent runtime share grows much larger. The authors refuse to treat that as a clean runtime result because batching and sampling differ between environments. That restraint is one of the paper's better features. Production migrations often change half a dozen things, then attach the entire gain to the most marketable one.
Three runs support the matched attribution cells. The production case study uses one run. Those sample sizes don't invalidate the observation, but they set its weight. The paper calls itself a hardware-conditioned case study rather than a universal benchmark.
Quantisation Changes Capacity Even When Speed Is Messy
Throughput is only half the argument for lower-precision weights. The tested FP16 model occupied enough of a 24 GiB card that concurrent users eventually pushed the stack into a reproducible memory cliff. Both INT4 stacks supported roughly four times as many concurrent users before failures in the authors' workload.
That result combines several memory effects. The weight footprint drops, leaving more capacity for KV cache and runtime buffers. The runtime's cache policy also affects how efficiently the remaining memory serves active requests. A smaller checkpoint with a wasteful allocator can still perform poorly.
And there is a quality question. GPTQ INT4 may be entirely acceptable for one model and task, then produce a measurable regression elsewhere. The paper focuses on serving performance and capacity, not a full quality assessment. Production teams should pair throughput tests with their own evaluation set, structured-output checks, refusal tests and difficult prompts.
If accuracy requirements force FP16 or BF16, the buying path changes. More VRAM, more devices or a different model becomes necessary. Quantisation is an option, not free capacity.
Why Four GPUs Didn't Behave Like Four GPUs
Tensor parallelism splits one model's operations across devices. It becomes necessary when the model won't fit on one card, and it can reduce per-request latency when computation and communication balance well. It also creates collective operations that every participating GPU must reach and complete.
The study's four-way sharded instance delivered about 1.5 times the throughput of one card in the reported conclusion, not anything close to four times. A profiler trace attributed most of the per-token shortfall at the studied point to coordination between shards. Some loss also came from lower matrix-multiplication efficiency and host-side scheduling.
The two-pair topology deserves attention. Communication inside each NVLink pair had a faster physical route, while cross-pair operations used PCIe. It would be easy to blame the slower connection. The authors ran a control at their decode message sizes and observed similar realised reduction bandwidth on NVLink and PCIe, which pointed towards launch and synchronisation cost instead of raw bandwidth.
That doesn't prove interconnect never matters. The paper explicitly leaves an all-four-GPU fast-switch arrangement untested. Larger messages, different collectives, newer GPUs or an HGX fabric could move the balance. The safe conclusion is local: adding cards to tensor parallelism carries a coordination bill, and link headline bandwidth alone doesn't calculate it.
Sharding Versus Independent Instances
One sharded model and four separate model replicas solve different problems. Sharding gives one request access to several GPUs. Replication lets several requests run independently and sends each to a separate backend.
For the smaller model in this study, the winner changed with workload. The sharded instance did better on long-output, decode-heavy cells. Independent instances won at wider batches, and they remained available at the largest cell where the four-way sharded engine failed during initialisation.
The 70B comparison flipped the smaller model's result. Two paired instances beat a single four-way sharded instance across the tested workloads. The authors link that reversal to fewer participants in each reduction and a faster shared-memory path inside the smaller groups.
This is the sentence buyers should keep: the best GPU grouping depends on model size and traffic shape. A topology chosen for an 8B chatbot may be wrong for a 70B endpoint, even on the same server.
Replication also has costs. Several instances compete for host CPU, memory bandwidth and storage. They duplicate weights and may strand capacity when traffic is uneven. A router must know which backend can accept work, and observability must remain coherent across processes. The paper reports sub-linear multi-instance scaling under shared-host contention, so "four replicas equal four times throughput" is no safer than the tensor-parallel assumption.
Our Technical View
The study's hardware is no longer the obvious choice for a new deployment, but that makes the lesson more useful, not less. Many universities, studios and engineering teams still operate mixed PCIe GPU estates. They often have enough aggregate VRAM to load a model, yet no clean answer about how the cards should cooperate.
Before buying another GPU, separate four questions. Can the model fit on one device at the chosen precision? Does one request need several devices to meet latency? Is aggregate throughput more important than single-request speed? Can the host feed several independent instances without CPU, RAM or PCIe contention?
If the model fits and traffic contains many independent requests, replication is usually the first design worth testing. It avoids frequent cross-GPU collectives and gives the scheduler more freedom. Tensor parallelism earns its place when the model or KV cache won't fit, or when single-request latency benefits enough to cover coordination.
HGX changes the physical fabric, but it doesn't remove software overhead. NVSwitch can give GPUs a more uniform high-bandwidth path, which matters for large scale-up workloads. The runtime, kernels and batch policy still decide whether that fabric stays busy.
A Better Way to Benchmark a Planned Purchase
Build a matched ladder instead of comparing two complete stacks. Keep the model and precision fixed while changing the runtime. Then hold the runtime fixed while changing the weight format and kernel. Test device grouping separately.
Use production-shaped prompts. Record input tokens, requested output tokens, concurrency and arrival pattern. A ten-prompt synthetic pool is useful for controlled research, but a buyer should add retrieval context, system prompts, tool schemas and the ugly long-tail requests that appear after deployment.
Measure more than aggregate tokens per second:
- time to first token and inter-token latency;
- request completion rate at the target concurrency;
- P95 and P99 latency, not only the average;
- GPU memory, KV-cache occupancy, power and CPU saturation;
- model-quality checks after any quantisation change;
- queue time and failed requests during bursts.
Keep sampling the same. Greedy decoding, temperature settings and stopping rules can alter generation length and kernel behaviour. The paper's own caveat shows why that detail can't sit in a footnote.
Practical Configuration Paths
One model that fits on one GPU: test several independent instances first, especially when aggregate throughput matters more than one user's latency. Give the host enough CPU cores and RAM bandwidth, and pin processes sensibly across NUMA domains.
A model that barely misses one GPU: compare quantisation with two-way tensor parallelism. Quantisation may simplify the whole service, but only if quality remains acceptable. Two GPUs may preserve precision at the price of communication and extra power.
Large model with steady demand: an HGX server can make sense when scale-up communication and a common memory pool across accelerators dominate the workload. Test the chosen runtime on the exact GPU generation; don't transfer A5000 ratios onto H200, B200 or B300.
Mixed models or research users: a PCIe GPU server with independent cards can offer better scheduling flexibility. Researchers can reserve different devices, run separate runtimes and avoid forcing every job through one large shard.
Demand still uncertain: use GPU Cloud to collect traffic traces and quality results. Once utilisation and grouping have settled, Buy & Host can provide dedicated capacity without asking the buyer to operate the rack.
Who Shouldn't Generalise This Result
Operators using current data-centre accelerators shouldn't assume the RTX A5000 scaling ratio applies to them. HBM bandwidth, NVLink generation, switch topology, kernel support and host design differ. The paper also doesn't test a four-GPU all-NVLink switch fabric.
Low-latency single-user applications may care little about wide-batch throughput. Batch services can accept queueing that interactive chat cannot. Agent workloads with long tool pauses create another traffic pattern entirely.
The model set is narrow, and the prompt pool contains ten short synthetic prompts. Real RAG and coding services can carry far longer prefills. Different context lengths alter the relationship between compute, KV cache and communication.
FAQ
Does INT4 always make LLM inference faster?
No. INT4 reduces weight memory and may speed supported matrix operations, but the gain depends on the kernel, GPU, runtime, batch and workload. It can also trade away model quality.
Should I shard an 8B model across four GPUs?
Usually not if it already fits comfortably on one device and aggregate throughput is the goal. Independent instances deserve the first test. Sharding may still help a long-output or latency-sensitive workload, so measure both routes.
Is NVLink unimportant for inference?
No. The study found that raw link bandwidth didn't explain its result at the tested decode message sizes. Other models, collectives and platforms can depend heavily on NVLink or NVSwitch.
Why can a runtime produce more gain than quantisation?
A runtime controls batching, scheduling and KV-cache use. At higher concurrency, those choices can keep the GPU occupied and avoid memory waste. Quantisation changes another part of the same system.
How much VRAM should I configure?
Start with weights at the chosen precision, expected KV cache, runtime reservation and the highest planned concurrency. Add operating headroom for model updates and real prompt variance. GPUMachines can review that budget against available workstation, PCIe and HGX platforms.
Can GPUMachines benchmark my serving stack?
GPUMachines can help define a hardware and deployment test plan, then map measured behaviour to GPU, CPU, RAM, NVMe and networking choices. Any performance claim should come from the buyer's actual model and traffic rather than this paper's A5000 figures.
Sources and Further Reading
- Decomposing Runtime, Kernel, and Quantization Speedups on Four NVIDIA RTX A5000 GPUs
- Experimental HTML version of the paper
- NVIDIA GPU Performance Background Guide
- NVIDIA CUDA C++ Best Practices Guide
Verdict
The paper doesn't prove that runtime work matters more than quantisation on every GPU. It proves something more practical: a whole-stack ratio can't tell you which component earned the gain.
For GPUMachines buyers, that changes the order of operations. Measure the runtime, precision and GPU grouping independently; preserve quality checks; then buy the platform that solves the observed limit. A larger server can hide a poor serving design for a while, but it can't explain it.