A fast GPU can still make an AI agent feel slow. The reason is easy to miss in a system quote: reading a long prompt and producing the next token are different jobs, and one accelerator design does not always perform both at the best operating point.
NVIDIA used Hot Chips 2026 to put that split at the centre of its Vera Rubin story. Rubin GPUs remain the broad compute engine for training and inference, while NVIDIA Groq 3 LPX adds a specialised rack for fast, predictable token generation. NVIDIA Dynamo coordinates the work between them.
For buyers, the announcement does not mean every Vera Rubin deployment needs LPX. It means the old question, "Which GPU should we buy?", is becoming too blunt for the largest inference services. A better question is: where does our latency come from, and is that delay large enough to justify separate hardware for context processing and token generation?
The short answer is that LPX makes most sense for large, heavily used agentic or interactive services where long context, extended reasoning and strict per-user latency occur together. A research cluster used mainly for training, a batch inference service, or a smaller private deployment will usually get more value from a simpler GPU-only platform.
GPUMachines has not benchmarked Vera Rubin with Groq 3 LPX. Performance figures in this article are NVIDIA or third-party benchmark claims cited by NVIDIA, not independent GPUMachines results. The useful work here is translating the architecture into a procurement test.
What NVIDIA announced at Hot Chips 2026
NVIDIA's Hot Chips programme covered the Vera CPU, Rubin GPU, BlueField-4 and Spectrum-X networking. Its accompanying material presented the wider Vera Rubin platform as a set of specialised systems rather than one universal rack.
The part that changes inference design is the pairing of Vera Rubin NVL72 with Groq 3 LPX. NVIDIA describes Rubin as the flexible processor for long-context prefill, attention, training and high-throughput inference. LPX uses Groq 3 language processing units with compiler-scheduled execution and on-chip SRAM, aiming at stable token generation latency.
NVIDIA says an LPX rack contains 256 LPUs. Its technical material reports 40 PB/s of on-chip SRAM bandwidth and 640 TB/s of rack-scale chip-to-chip communication. Those figures describe the vendor's architecture; they do not tell a buyer how a particular model will perform once request routing, model compatibility, network transfers and service-level targets enter the picture.
An NVIDIA post published alongside Hot Chips reports 3,400 output tokens per second for Gemma 4 31B with a 100,000-token context in an Artificial Analysis benchmark, described as four times the nearest alternative platform. That is a striking result, but it remains one named model, context and benchmark route. It should prompt a proof of concept, not become a sizing ratio for an unrelated workload.
The more important announcement is architectural: NVIDIA is treating inference as several workloads that can run on different processors.
Prefill and decode do not stress hardware in the same way
An LLM request starts with prefill. The server reads the input tokens, runs them through the model and builds the key-value cache needed for generation. Long prompts, retrieved documents, tool histories and agent memory can make this stage compute-heavy.
Decode follows. The model generates output tokens one after another, repeatedly reading weights and the growing KV cache. Decode often feels memory-bound, and users notice its variability directly as inter-token latency. Batching can improve aggregate throughput, but aggressive batching may make an individual session less responsive.
That difference creates an awkward compromise on a shared pool. Hardware and scheduling tuned for large prefill batches may not deliver the most predictable token stream. A pool tuned for immediate decode may leave expensive compute capacity underused during other work.
Disaggregated inference separates the stages so operators can scale them independently. It is already possible to place prefill and decode on different GPU workers. The Vera Rubin and LPX design goes further by introducing a different processor type for selected decode work.
This is not a universal efficiency trick. Disaggregation adds queues, routing decisions, state transfer, more software versions and another failure boundary. It wins only when the saved processing time exceeds that coordination cost across the workload that matters.
The three serving patterns buyers need to distinguish
NVIDIA describes several ways to combine Vera Rubin and LPX. They should not be collapsed into a single claim about "faster inference" because each one moves different data and fails differently.
Prefill-decode disaggregation
Vera Rubin processes the prompt and builds the KV cache. The serving layer then transfers that cache to LPX, which handles decode for the turn.
This is the cleanest conceptual split. It can suit sessions where a substantial prefill phase precedes a long generation phase, because the transfer occurs once per turn. The difficulty is KV-cache movement. Cache size grows with model structure, context length, sequence count and precision. A design that ignores the transfer path may trade compute delay for network delay.
Ask for measured transfer time at the context percentiles seen in production, not at one convenient prompt length. The test should include queueing and admission behaviour when several long prompts finish prefill together.
Attention-FFN disaggregation
During decode, Rubin GPUs retain the KV cache and execute attention. LPX runs feed-forward or mixture-of-experts layers, with intermediate activations moving between the two systems during the token loop.
This arrangement can exploit LPX for latency-sensitive pointwise or expert work while leaving attention and the large KV state on GPU memory. It also makes the connection between racks part of every generated token. Tail latency now depends on compute, transfer latency, congestion and orchestration staying within budget together.
A buyer considering this route should request p50 and p99 inter-token latency under realistic concurrency, then repeat the run while other traffic uses the fabric. Average tokens per second can conceal occasional stalls that make a coding assistant, voice service or multi-step agent feel unreliable.
External-drafter speculative decoding
LPX can run a smaller draft model that proposes several tokens. The larger model on Rubin verifies the candidates and accepts valid runs in parallel. Each system keeps its own KV cache, while draft tokens and rejection information cross the connection.
The gain depends on acceptance rate. A draft model that predicts well can reduce the number of costly target-model steps; a poor match adds work without committing many tokens. Model choice, domain, sampling settings and prompt distribution all affect the result.
Do not approve this configuration from a vendor demonstration alone. Measure acceptance rate across the buyer's own prompts, then account for the extra rack, power, software and operational work.
Why agents make the latency problem harder
A single chatbot turn hides some infrastructure delay because a person reads the answer before asking another question. An agent may call a model, query a database, run code, inspect the result and call the model again. Ten modest delays can become one slow task.
Long context adds pressure at the start of each inference step. Extended reasoning increases the number of generated tokens. Tool use introduces CPU, storage and network pauses between model calls. Multi-agent designs multiply concurrent sessions and create uneven request shapes.
This is why token generation speed alone is an incomplete metric. An infrastructure buyer needs end-to-end task completion time, including prefill, decode, routing, tool execution and any wait for a free worker. An LPX rack cannot fix a slow retrieval service or saturated database. It may make the idle time elsewhere more visible.
The GPUMachines agent-native telemetry guide explains a related operational problem: once autonomous systems act on infrastructure, the platform must record intent, authority and observed state change. A heterogeneous inference service needs similar discipline around routing decisions and fallback, even if no agent controls the cluster itself.
The buying answer: start with the workload trace
Before asking for an LPX quotation, collect request traces from the intended application. Synthetic prompts are acceptable for an early lab test, but they should reproduce the distribution of the real service rather than one average request.
At minimum, capture:
- input-token distribution, including p50, p95 and the longest accepted context;
- output-token distribution and how often users stop generation early;
- concurrent sessions, burst shape and queueing limits;
- time to first token, inter-token latency and end-to-end task time;
- model family, parameter count, quantisation, routing pattern and software versions;
- prefix-cache hit rate, speculative acceptance rate where used, and KV-cache pressure;
- power draw and completed useful requests during the same measurement window.
These measurements reveal whether the service is prefill-heavy, decode-heavy, queue-bound or blocked outside inference. They also show whether one configuration must serve conflicting jobs. Without them, a specialist rack is an expensive guess.
Our NVIDIA B300 INT8 software-support analysis reaches the same conclusion from another direction: a format named in the accelerator specification does not prove that the selected kernel, library and serving engine can execute the intended path. LPX procurement needs an even stricter software check because the serving graph crosses different processor types.
When LPX could earn its place
The strongest case combines scale, latency sensitivity and sustained utilisation.
An AI coding service may need long repository context and a fast visible token stream. A voice agent has little tolerance for pauses between tokens. A multi-agent research system can produce many sequential model calls, so reduced decode latency may shorten the whole task. Large service providers may also have enough varied traffic to send latency-sensitive sessions to LPX while keeping throughput-first jobs on Rubin.
Mixture-of-experts models are relevant because attention and expert feed-forward work can stress different parts of the system. NVIDIA specifically positions attention-FFN disaggregation for this pattern. Yet the model must fit the supported compiler and serving path, and the rack-to-rack activation exchange has to remain cheap enough at production load.
Another credible use is premium service tiers. If a provider can attach real commercial value to lower and more predictable latency, it can reserve specialised capacity for those requests. That case needs queue isolation and admission control; otherwise ordinary batch traffic can consume the resource that customers paid to access.
Who should not buy it
Training-led research teams do not need LPX merely because they plan to buy Rubin. LPX targets inference. Money allocated to it may produce more value as additional GPU capacity, storage throughput, networking or engineering time.
Small private deployments should be wary as well. If one or two GPU servers meet the latency target, disaggregating the pipeline creates more systems to patch and monitor without solving a real shortage. The same applies to internal services with low concurrency or relaxed response times.
Batch inference rarely needs the lowest per-user token latency. A well-batched GPU service may deliver better economics and a simpler operating model. Teams that have not yet tuned batching, prefix caching, quantisation, tensor parallelism and admission control should do that work before buying another accelerator class.
LPX is also a poor remedy for a model that the software path does not support. Compiler coverage, operation support, model revisions and orchestration compatibility must be verified against the planned release. Hardware cannot compensate for a missing execution path.
The rack and network consequences
Once inference spans Rubin and LPX racks, the interconnect carries application state rather than background traffic. The fabric has to support the chosen serving pattern under failure and congestion, not merely advertise a high port rate.
Prefill-decode disaggregation needs efficient KV movement. Attention-FFN disaggregation sends smaller intermediate data more often, placing tighter demands on latency and jitter. Speculative decoding moves less state but depends on timely coordination between draft and verification work.
Capacity planning should therefore include the ratio of Rubin to LPX resources. That ratio will vary with prompt length, output length, model structure, concurrency and the fraction of traffic assigned to the low-latency path. A static one-to-one assumption is unlikely to survive a workload change.
Failure handling deserves equal attention. If LPX becomes unavailable, can Dynamo send the session back to a GPU-only path? What latency penalty follows? Does the application preserve the KV state, restart the turn or return an error? The procurement test should disconnect a worker or link deliberately and observe what users experience.
At larger scale, use the GPUMachines GPU Cluster Configurator to map compute, switch ports, rack positions and management networks before fixing the final bill of materials. The logical inference graph and the physical cable plan need to agree.
Software becomes part of the machine
NVIDIA Dynamo is responsible for classifying and routing requests across the heterogeneous service. It also has to coordinate KV-aware placement, disaggregated execution and latency targets. That makes the orchestrator part of the performance envelope.
Version control matters. Record the exact Dynamo, CUDA, model runtime, compiler, firmware and model revisions used during acceptance testing. A later upgrade may improve support, but it can also change graph partitioning, cache movement or fallback behaviour. Keep a known-good path and test rollback.
Observability must separate each stage. A single request latency metric cannot show whether the delay came from queue admission, prefill, KV transfer, attention, LPX execution, verification or a tool call. Track stage timings and correlate them with fabric congestion, power and worker availability.
The scheduler also needs an honest objective. Maximising aggregate tokens per second may hurt interactive p99 latency. Protecting every request from queueing can leave hardware idle. Decide which service tiers exist and what each one promises before tuning the system.
A procurement acceptance test
Run the proposed architecture against the exact model and software stack intended for production. Then make it uncomfortable.
1. Baseline a GPU-only path. Measure the same requests on the proposed Rubin or current-generation GPU system before adding LPX. The extra system needs to beat a credible baseline, not an untuned one. 2. Replay the request distribution. Include long-context outliers, short interactive prompts, bursts and sustained concurrency. Report results by percentile. 3. Measure both user and estate outcomes. Track time to first token, inter-token latency, completed tasks, queue depth, errors, power and occupied capacity. 4. Expose transfer cost. Record KV or activation movement separately. Repeat the test while the network carries representative background traffic. 5. Test model coverage. Exercise every required operation, quantisation and context length. A successful demo on a neighbouring model does not count. 6. Break a component. Remove an LPX worker, interrupt one network path and restart an orchestration service. Confirm the fallback and recovery behaviour. 7. Change the workload mix. Increase short prompts, then long generations. The Rubin-to-LPX ratio should still make operational and financial sense. 8. Repeat after upgrade. Re-run the acceptance suite on the planned production firmware and software release, not only the vendor's demonstration image.
Ask for raw traces or exportable measurements where the supplier can provide them. A slide with one maximum throughput result cannot prove service quality.
Current alternatives
Vera Rubin without LPX remains the simpler choice for mixed training and inference, especially where flexibility matters more than the lowest token latency. Rubin GPUs can run prefill and decode; LPX adds another operating point rather than making the GPU path obsolete.
Current HGX B300 and B200 systems offer an earlier procurement route for teams that need dense GPU compute but are not ready for the full Rubin platform. The GPUMachines HGX server range includes platforms that can support workload profiling, software qualification and staged cluster deployment before a future rack-scale decision.
Hosted capacity is sensible when the application is still changing. It lets a team measure request shape and utilisation without committing its facility to a specialised rack design. GPUMachines Buy & Host can also suit buyers who want to own the hardware while placing it in a facility with defined power, cooling and remote-hands support.
For a general explanation of the rack, NVLink domain and facility requirements, read NVIDIA Vera Rubin NVL72 explained. That article answers what the platform is. The decision here is whether heterogeneous inference belongs in the first deployment or only on the roadmap.
Questions buyers are likely to ask
Does Groq 3 LPX replace Rubin GPUs for inference?
No. NVIDIA positions LPX beside Vera Rubin. Rubin handles broad training and inference work, including long-context prefill and attention, while LPX targets fast token generation and selected decode operations. The combined system depends on the chosen serving pattern.
Is prefill-decode disaggregation the same as attention-FFN disaggregation?
No. Prefill-decode disaggregation separates the two main request phases. Attention-FFN disaggregation splits work inside the decode loop, leaving attention and KV state on GPUs while LPX runs feed-forward or MoE operations.
Will LPX reduce the number of GPUs a buyer needs?
Possibly for a suitable inference service, but no general reduction can be assumed. Model support, traffic shape, latency target, rack ratio and transfer cost determine whether the combined platform needs less power or capacity for the same useful workload.
Is this relevant to ordinary enterprise RAG?
Usually not at first. Many RAG services can meet their targets on one GPU server or a modest cluster. LPX becomes more relevant when long context, sustained concurrency and strict token latency have already pushed a tuned GPU-only service beyond its operating target.
What should be measured before requesting a quotation?
Collect prompt and output lengths, concurrency, queueing, time to first token, inter-token latency, model and quantisation details, KV-cache behaviour and completed tasks per unit of power. Keep the raw workload distribution rather than relying on an average.
Can GPUMachines help plan a Rubin and LPX deployment?
GPUMachines can help turn the workload profile into a hardware, fabric, storage, rack-power and hosting plan. Availability, final configuration and software support must be confirmed against the specific production platform before an order is approved.
Sources and Further Reading
- NVIDIA at Hot Chips 2026
- NVIDIA: With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents
- NVIDIA technical blog: How Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on Vera Rubin
- NVIDIA technical blog: Inside NVIDIA Groq 3 LPX
- NVIDIA technical blog: Deploying Disaggregated LLM Inference Workloads on Kubernetes
- Hot Chips 2026 conference programme
Verdict
Vera Rubin plus Groq 3 LPX is a serious architecture for a narrow class of demanding inference services. It makes sense when long context and rapid token generation collide at high utilisation, the software supports the exact model, and the commercial value of lower latency pays for another rack type and its operational burden.
It is not the automatic premium option for every Rubin buyer. Training clusters, batch services and smaller private deployments should favour the simpler GPU path until workload traces prove that decode latency limits useful output.
GPUMachines can review those traces, compare GPU-only and heterogeneous deployment paths, and map the selected design into compute, networking, storage, power and hosting requirements. Start with the GPU Cluster Configurator, then validate the proposed serving graph before fixing the hardware ratio.