> Source: This article analyses and summarises "Next-generation LLM Inference Network: How ZCube Alleviates Network Bottlenecks?" published by Z.ai on 20 May 2026. Original post: z.ai/blog/zcube. All performance numbers, figures and architectural claims belong to Z.ai, Harnets.AI and Tsinghua University.
For most of the last decade, the inter-GPU network has been the boring part of an AI cluster. You sized your spine-leaf fabric, ordered the optics, and moved on. The interesting work happened inside the GPU. That assumption is now breaking — and a new joint deployment by Z.ai, Harnets.AI and Tsinghua University shows just how much performance and cost is still trapped in the fabric.
Their architecture, called ZCube, was published at ACM SIGCOMM 2025 and has now been deployed in a production thousand-GPU cluster serving the GLM-5.1 coding model. The headline numbers from Z.ai's report:
- +15% average GPU inference throughput
- -40.6% Time-to-First-Token (TTFT) P99 tail latency
- -33% switch and optical-module CapEx
- Same GPUs, same software stack, same application code
For a sector where every percent of GPU utilisation is fought over, those are very large numbers — and they come purely from changing how the switches are wired.
Why the network suddenly matters for inference
LLM inference used to look like a friendly traffic pattern: model weights live on the GPU, requests come in, tokens go out. That has changed for two reasons:
1. Long-context inference. Prompts of tens or hundreds of thousands of tokens generate huge KV caches. 2. Prefill / Decode (PD) disaggregation. Modern serving stacks split the compute-heavy prefill phase from the memory-bound decode phase onto different GPUs. The KV cache for each request must then be shipped across the network between them.
Z.ai ran a controlled study on a 32-GPU testbed where only the NIC bandwidth cap was changed. Going from 100 Gbps to 200 Gbps, with everything else identical, lifted cluster throughput by ~19% and reduced TTFT by ~22%. The network is no longer a passive substrate — it is on the critical path of throughput, latency and per-token cost.
The hidden problem inside Rail-Optimised Fat-Trees
Most modern AI clusters use a Clos / Fat-Tree topology, often in its Rail-Optimised Fat-Tree (ROFT) variant. ROFT groups GPUs by NIC index and connects same-index NICs to the same leaf switch, which is efficient for the symmetric collectives (AllReduce, All-to-All) of training.
PD-disaggregated inference is not symmetric. Z.ai observed:
- KV-cache transfers are strongly source/destination asymmetric — different NICs on the same machine carry wildly different loads.
- A handful of leaf switches become persistent hotspots, and multiple KV flows end up fighting for the same links.
- Egress queues on those switches sit deep for long periods and repeatedly trigger PFC backpressure.
- Aggregate bandwidth looks fine on paper, while localised congestion kills tail latency.
They draw a useful distinction between two kinds of congestion:
- Unavoidable — multiple senders genuinely want the same final-hop link. Congestion control and traffic shaping help here.
- Avoidable — caused by topology and routing choices that funnel unrelated flows onto the same path. This is an architecture bug, and no amount of clever transport (adaptive routing, packet spraying, MRC) fully fixes it.
ZCube is an attempt to eliminate the second category at the source.
What ZCube actually is
ZCube throws away the hierarchical Clos idea of stacking spine and leaf layers. Its core moves are:
1. Delete the spine layer entirely. 2. Split leaf switches into two equal groups (odd-numbered and even-numbered). 3. Wire those two groups as a complete bipartite graph — every odd switch connects to every even switch. 4. Use dual-ported NICs with a hybrid access pattern: one port joins a single-rail group (contiguous GPU IDs per switch), the other port joins a multi-rail group (same relative index across groups).
The result is a flat, two-hop network: any pair of GPUs reaches each other through exactly two switches. That is one hop less than a typical two-tier ROFT, which already removes a slice of latency. More importantly, ZCube's routing strategy gives each GPU pair a unique optimal path, so there is no ECMP coin-flip that can accidentally collide flows on the same link.
The hybrid single-rail / multi-rail pattern is the clever bit. It means the same physical fabric balances well under both:
- Training-style traffic (AllReduce, All-to-All) — which ROFT was designed for, and
- Inference-style traffic (asymmetric, dynamic, source/destination unknown in advance) — which ROFT mishandles.
Scale and cost
According to Z.ai, a single layer of 51.2T switches (128 × 400 Gbps ports) is enough to wire 16,384 × 400 Gbps NICs in a flat ZCube. Higher-radix switches or multi-plane variants push it into the hundreds of thousands of GPUs — comfortably into hyperscale territory, without ever introducing a spine layer.
Because the spine is gone, so are the optics and switches that would have built it. Z.ai estimates a ~33% reduction in switch + optical-module CapEx versus an equivalent Clos / ROFT cluster, and quotes a saving of roughly RMB 210M–640M (approx. USD 29M–88M) on a 10,000-GPU build.
Production results on GLM-5.1
The production validation is the part operators will care about most. Z.ai migrated a thousand-GPU GLM-5.1 coding inference cluster from ROFT to ZCube without changing GPUs, the software stack, or applications. They could not reuse the old cabling, IP plan, routing policy or switch configs — the Harnets.AI network team had to build a full migration toolchain (a ZCube Controller, a data-centre layout designer, and a cabling-correctness verifier) to do the cutover in a tight window.
After migration, on the same workload:
| Metric | Change vs ROFT | |---|---| | Average GPU inference throughput | +15% | | TTFT P99 tail latency | -40.6% | | Switch + optical CapEx | -33% | | GPU / software / application changes | None |
Z.ai reports the cluster has been running stably in production for over two weeks at time of publication, serving live GLM-5.1 coding traffic.
Why this matters for anyone buying GPU servers in 2026
Three takeaways stand out for teams designing AI infrastructure right now:
1. Inference is a network problem, not just a GPU problem. If you are scoping an H200, B200 or HGX cluster purely on TFLOPs and HBM, you are likely leaving 15–20% of effective throughput on the table at the fabric layer. 2. PD disaggregation breaks old topology assumptions. Rail-optimised Fat-Trees were tuned for training-style collectives. Once KV caches start flowing between prefill and decode pools, your traffic matrix no longer matches the topology you bought. 3. Architecture beats transport tweaks. Adaptive routing, packet spraying and modern congestion control all help, but the largest single win in Z.ai's deployment came from changing the shape of the network so that conflicts never form in the first place.
For our customers planning H100/H200/B200 inference clusters, this is a strong argument to treat the InfiniBand or Ethernet fabric as a first-class design decision — sized to the real PD traffic pattern of the model you intend to serve, not just to the GPU count.
Read the original
The full Z.ai write-up — with the figures showing PFC pause events, NIC load imbalance, and the ZCube topology diagram — is well worth reading in full:
➡️ Next-generation LLM Inference Network: How ZCube Alleviates Network Bottlenecks? — z.ai/blog/zcube
The underlying paper is Yan, Z. et al., "From ATOP to ZCube: Automated topology optimization pipeline and a highly cost-effective network topology for large model training", ACM SIGCOMM 2025, pp. 861–881.
---
Editorial note from GPUmachines: this article is a third-party summary and commentary. All architectural claims, figures and performance numbers are attributable to Z.ai, Harnets.AI and Tsinghua University as published in the source post above.
What Buyers Should Take From ZCube
Most buyers will not copy ZCube directly. A complete bipartite switch design, custom controller tooling and a thousand-GPU migration are not normal procurement tasks. The lesson is more useful than the exact topology: inference traffic has changed, and the network has to be designed around that change.
Prefill/decode disaggregation moves key-value cache traffic between GPU groups. Long-context serving makes that traffic larger. If routing funnels too many flows through the same leaf switch, the cluster can look well provisioned on paper while still delivering poor tail latency.
For GPUMachines buyers, that means the network discussion should happen before the GPU order is treated as final. Ask what the model-serving stack will do, where prefill and decode will run, how KV cache moves, and whether the planned fabric has enough useful path diversity.
Why This Matters Below Hyperscale
ZCube's reported deployment is large, but the pattern shows up in smaller clusters too. A four-node or eight-node inference cluster can still suffer from poor NIC placement, uneven routing, weak top-of-rack design or storage traffic sharing the wrong links.
Smaller buyers also have less room to hide mistakes. If a hyperscale team loses a few percent to network design, it has engineers and spare capacity. A smaller operator may feel the same mistake as missed latency targets, poor user experience or a need to buy more GPUs earlier than planned.
This is where GPUMachines can help with a sober design review. The right answer might be InfiniBand. It might be Ethernet with RoCE. It might be a simpler PCIe GPU layout, fewer nodes, better storage, or hosted capacity in a facility with the right network already in place.
Questions to Ask Before Building an Inference Cluster
Before copying any topology idea, write down the traffic pattern. Does the model run on one GPU, several GPUs inside one server, or multiple servers? Does the serving stack separate prefill and decode? How large is the context window? How much KV cache moves across the network? Which traffic shares the same links?
Then check operational reality. Can the team cable the design correctly? Can it verify cabling before production? Can it monitor queue depth, dropped packets, congestion and tail latency? Can it roll back a fabric change without taking the whole service offline?
ZCube is exciting because it attacks avoidable congestion at the architecture level. The buyer lesson is quieter: if you cannot see the network, you cannot tune the cluster.
GPUMachines Takeaway
For buyers, ZCube should make network design feel less optional. The fabric is not only a transport layer; it is part of inference cost. A design that reduces hotspots can improve GPU use without changing the GPU model. A poor design can create the opposite result, where more GPUs are bought to cover a bottleneck that lives in switching and cabling.
That matters for both HGX and PCIe clusters. HGX can solve scale-up communication inside the node, but scale-out traffic still leaves the server. PCIe clusters may rely even more heavily on external fabric design because the GPUs do not have the same internal NVLink domain.
Before ordering a model-serving cluster, GPUMachines should review node count, NIC type, switch plan, storage path, expected serving stack and whether the deployment will use disaggregated prefill/decode. The answer may be a conventional fabric. It may be a flatter design. It may be hosted infrastructure. The point is to decide before the GPUs are waiting for packets.
Final Buyer Checks
ZCube should not send every buyer searching for a new topology. It should make buyers ask better questions about inference traffic. If the serving stack keeps GPU workers mostly independent, the network design can be simpler. If the stack moves large KV cache traffic between prefill and decode workers, the fabric needs more careful review.
Look for symptoms before redesigning anything. High P99 latency, uneven NIC utilisation, persistent switch queues and GPU idle time during heavy request load are signs that the network may be part of the cost problem. If those signals are not measured, changing topology becomes guesswork.
GPUMachines can help buyers test the boring version first: correct NIC placement, clean cabling, sensible switch choice, storage separation and monitoring. Only then does it make sense to discuss flatter fabrics, alternative routing or larger topology changes.
The useful lesson from ZCube is not that every cluster needs ZCube. It is that inference networking has become part of model-serving economics, and it deserves design time before the purchase order is locked.