Spectrum-X Multiplane Network Guide

Multiplane can route GPU traffic around a damaged path. It does not make compute, storage, access and management the same network design problem.

One degraded 200 Gb/s path should not throttle every healthy plane beside it. That is the problem behind NVIDIA's current Spectrum-X Multiplane work, and it is more useful to a buyer than the largest switch-bandwidth number on the slide.

Spectrum-6 increases the switching capacity available to future AI fabrics. Multiplane changes how endpoint bandwidth is divided across several independent network planes and how traffic avoids a damaged plane. These are related ideas, but they do not answer a separate architecture question: should compute, storage, tenant access and management all share one physical Ethernet fabric?

Usually, no. Multiplane is primarily a scale-out compute design. It can present several paths as one logical RoCE device to the application, but that does not make every traffic class equivalent. Storage and access traffic may share Spectrum-X equipment where isolation, capacity and failure tests support it. Management should retain an independent recovery path. The correct answer is a small number of deliberately defined fabrics, not “one network for everything” or automatic physical separation of every packet.

This article updates our Spectrum-6 analysis with NVIDIA's Hot Chips 2026 multiplane material. GPUMachines has not independently benchmarked production Spectrum-6 or ConnectX-9 systems. Vendor scale examples and fault tests are identified as such and should not be used as sizing guarantees.

The short buying answer

Use Spectrum-X Multiplane for the GPU scale-out fabric when endpoint bandwidth is split across multiple NIC ports or rails and the cluster needs path diversity without adding another switch tier.

Keep management physically independent so operators can reach servers and switches during a compute-fabric fault or configuration error.

Decide storage separately. It may share the Ethernet switching platform, a reserved set of planes, or a distinct fabric. Base the choice on storage traffic, failure blast radius, tenant isolation and operational ownership.

Do not collapse BlueField Scale-In access and ConnectX scale-out collectives into one undifferentiated network. NVIDIA assigns them different roles even when both use Spectrum-X Ethernet.

For small clusters, a conventional single- or dual-plane leaf-spine may be simpler. Multiplane is useful only when the endpoint, switch and software design can operate it correctly.

What Spectrum-6 changes

NVIDIA describes the Spectrum-6 switch ASIC with 102.4 Tb/s of bidirectional switching capacity. The platform supports dense high-speed ports that can be broken into lower-rate endpoints. This gives designers more options for connecting many GPU-facing links while keeping a shallow topology.

Switch capacity is not application throughput. Useful distributed-training performance depends on:

the number and rate of endpoint ports;
rail mapping from GPUs and NICs to switches;
oversubscription between leaf and spine layers;
packet loss, retransmission and congestion control;
collective communication pattern;
optics, fibre quality and cable length;
switch software and endpoint firmware;
topology-aware scheduling.

A 102.4 Tb/s switch can still be deployed badly. Conversely, a smaller switch can serve a modest cluster well when the topology matches the workload.

The purchasing value of Spectrum-6 is density and design headroom. Fewer high-radix switches can reduce tiers, optics and failure points for a given endpoint count. That needs to be demonstrated in a bill of materials rather than inferred from the ASIC number.

What “multiplane” means

An AI server may have several scale-out network interfaces. Instead of connecting every interface into one large shared spine, a multiplane design divides the fabric into independent planes. Each plane contains its own switches and paths. A server reaches several planes through separate NIC ports or rails.

The application should not have to choose a plane for every flow. NVIDIA's current design uses Plane Load Balancing in the SuperNIC path. The endpoint observes plane condition and sends traffic through healthy capacity while presenting a unified RoCE device to the operating system and collective libraries.

This is different from blindly spraying packets across all planes. If traffic is distributed evenly without regard to path health, one degraded plane can become the pace setter for the whole job. NVIDIA's Hot Chips material describes PLB filtering or avoiding the damaged plane so remaining capacity can continue to work.

The commercial benefit is failure containment and shallow scaling. The operational cost is that NIC, firmware, Kubernetes or bare-metal configuration, switch topology and telemetry all have to agree about plane membership.

Rails, planes and fabrics are not synonyms

These words are often mixed in quotations, which makes port counts hard to audit.

A rail is commonly the repeated network path associated with corresponding GPU or NIC positions across nodes. Rail-aware designs keep communication on predictable paths and can reduce cross-traffic.

A plane is an independent switching domain within the scale-out network. A rail may connect into one or more planes depending on the reference architecture.

A fabric is the broader network system, including endpoints, switches, links, routing, congestion control and operations. One logical compute fabric may contain several physical planes.

A traffic class is the work carried across a fabric: collectives, storage, tenant access, management or telemetry.

The design review should show all four. A diagram containing coloured lines without port identifiers, speeds and failure boundaries is not enough.

One fabric for compute and storage?

Ethernet makes convergence possible. It does not make convergence free.

Shared switches can reduce hardware count and simplify procurement. A Spectrum-X platform can carry RoCE compute traffic and storage access, particularly when BlueField offload, congestion controls and capacity reservations are used. This can be sensible for smaller estates or operators with strong Ethernet automation.

The risk is correlated congestion and failure. Checkpoint writes, dataset reads and rebuild traffic can burst while collectives are sensitive to delay. A switch software fault or mistaken policy can then affect both compute and data access. Troubleshooting becomes harder when one queueing problem appears as GPU, storage and application delay at once.

Three patterns are common:

Shared physical fabric with logical isolation

Compute and storage use the same switches but separate VLANs, VRFs, queues, policies or ports. This saves equipment but still shares switch silicon, software and some failure modes. It suits environments that can prove performance isolation under concurrent load.

Shared platform, reserved planes or switches

The operator standardises on Spectrum-X but assigns particular planes or switch groups to storage and access. Operations stay familiar while capacity and failure domains are clearer. The endpoint and routing design must prevent accidental crossover.

Separate compute and storage fabrics

Each traffic class receives independent switches and links. This costs more ports, optics and rack space but provides clearer ownership and fault isolation. Large training clusters with heavy checkpoint and dataset traffic may prefer this approach.

There is no universal winner. Run the storage workload beside the distributed job and observe p99 iteration time, checkpoint duration and recovery after a failed path. The AI storage infrastructure guide explains the storage-side measurements.

Management should stay reachable

Out-of-band management is the route used when the data network is misconfigured, overloaded or unavailable. If it shares the same switches, credentials and control plane as the compute fabric, an incident can remove the tools needed to repair it.

Use a physically independent management network for baseboard controllers, console access and core switch management wherever the operational risk justifies it. It does not need the bandwidth of the AI fabric. It needs predictable reachability, restricted access, monitoring and a tested break-glass procedure.

In-band telemetry can still travel through the high-speed fabric. It should not be the only evidence source during a fabric failure.

Scale-Out and Scale-In need separate design decisions

NVIDIA uses Scale-Out for GPU workload communication across servers. ConnectX-9 SuperNICs carry tenant job traffic in the Vera Rubin architecture.

NVIDIA uses Scale-In for the services surrounding compute: tenant access, security, provisioning, enterprise data and external storage. BlueField-4 DPUs provide host-independent processing for those services, with Spectrum-X as an Ethernet path.

Both can use Ethernet switches from the same product family. They are still different operational domains. Scale-out collectives care about synchronised communication and job goodput. Scale-In services care about isolation, policy, storage behaviour and north-south access.

The BlueField-4 versus ConnectX-9 guide explains when a DPU belongs in that second path. The important point here is to keep port and traffic roles explicit.

How Plane Load Balancing changes failure behaviour

NVIDIA's published example describes an eight-plane network in which a 20% loss of connectivity on one plane would limit a conventional oblivious scheme to the degraded plane's capacity. The vendor says its hardware-accelerated Plane Load Balancer can avoid the affected plane and preserve more aggregate bandwidth.

That is a useful architecture claim, not a universal result. Buyers should ask:

which NIC and firmware versions support the required PLB mode;
whether load balancing is implemented in software or hardware;
which modes are generally available and which remain technical preview;
how quickly plane health is detected;
how existing flows are treated;
what telemetry identifies the failed or congested plane;
how the scheduler reacts when capacity falls;
whether behaviour changes in virtualised or Kubernetes environments.

NVIDIA's current Kubernetes documentation lists single-, dual- and quad-plane modes for particular GPU and SuperNIC combinations. It also distinguishes generally available software PLB from hardware PLB modes that are technical preview in some configurations. Version status can change. Record the exact compatibility matrix in the procurement baseline.

Multiplane does not remove topology work

The reference architecture still needs correct endpoint placement. If every node connects its first port to Plane 1 and second port to Plane 2, cabling and switch membership must remain consistent across racks. One reversed connection can create asymmetric or hidden paths.

Rail mapping should follow the server platform and collective design. Cable labels should include rack, node, NIC, port, rail, plane, switch and switch port. Store that data in a source-controlled inventory, not only a spreadsheet on an installer's laptop.

Switch tiers must be calculated from endpoint count and oversubscription. Breakout configurations change port arithmetic. Spare ports should be reserved for failure recovery and growth rather than counted twice.

Optical power, fibre type and connector loss need engineering at the intended distance. A link that negotiates successfully can still flap under marginal conditions. Multiplane can contain the effect, but it should not become an excuse for weak cabling standards.

The AI cluster network architecture guide covers rail and switch arithmetic in more detail.

When Spectrum-X Multiplane is worth buying

Endpoint bandwidth spans several links

High-bandwidth GPU systems expose multiple scale-out interfaces. Multiplane lets those interfaces use independent switching paths while appearing as one logical device to the application.

The cluster must tolerate routine link faults

At large scale, a fibre flap or degraded optic is normal operational work. A design that contains the fault and keeps jobs moving can improve useful cluster time.

A shallow topology reduces the bill

High-radix Spectrum-6 switches may support more endpoints with fewer tiers. The proposal should show the resulting switch, optic, cable, rack and power counts.

The operator can manage the software matrix

Multiplane behaviour relies on supported NICs, firmware, drivers, Kubernetes or bare-metal configuration and telemetry. Mature automation is part of the requirement.

When a simpler fabric is better

A small cluster may not have enough endpoints to justify several planes. A conventional non-blocking leaf-spine can be easier to cable, monitor and recover.

A team without rail-aware inventory and configuration automation should establish those practices before adding planes. Complexity introduced ahead of operational maturity can reduce reliability.

If the workload rarely communicates across nodes, local GPU memory or scale-up bandwidth may matter more than scale-out fabric. Do not buy a large network to serve an application that remains inside one server.

If the storage and compute estate is modest, one well-engineered Ethernet fabric with strict management separation may be the economical answer. Physical separation should follow measured contention and risk.

An RFQ-ready acceptance test

Require the proposed network to prove normal operation, contention and failure.

1. Show the physical map. List every server port, rail, plane, switch port, breakout, optic and cable. 2. State the software matrix. Record NIC firmware, switch OS, driver, CUDA or ROCm components, collective library and PLB mode. 3. Run the target collective. Measure all-reduce or the actual distributed workload across the planned node count. Report p50 and p99 iteration time. 4. Add storage traffic. Run checkpoint writes, dataset reads or inference data access beside the compute job. 5. Degrade one link. Introduce errors or reduced capacity on a plane. Record detection, traffic movement, application effect and recovery. 6. Remove a switch path. Confirm the blast radius matches the topology drawing. 7. Verify management access. Reach servers and switches while the compute fabric is impaired. 8. Inspect telemetry. Confirm operators can identify the plane, rail and physical component responsible for loss or congestion. 9. Test scheduler behaviour. Drain or avoid nodes whose network capacity has fallen. Do not let one sick endpoint repeatedly harm new jobs. 10. Repeat after upgrade. Re-run the suite on the production firmware and switch release.

Ask for raw counters and configuration exports. A successful ping test is not an AI fabric acceptance test.

Building the bill of materials

The fabric quotation should include more than switches and NICs. Count:

endpoint ports by speed and traffic role;
switch ports after breakout;
uplinks between tiers or planes;
optics and fibre assemblies;
spare optics, cables and switch capacity;
management switches and cables;
rack units and switch power;
cable managers and service loops;
monitoring, support and software subscriptions;
commissioning and acceptance testing.

Avoid one generic “networking” line. The GPU Cluster Configurator can expose node, switch, port, rack and power assumptions before a supplier fixes the BOM.

For buyers deciding whether Ethernet is the right base at all, compare it with the InfiniBand and RoCE guide.

Questions buyers ask

Is Spectrum-6 the same as Spectrum-X Multiplane?

No. Spectrum-6 is a switch generation and capacity platform. Multiplane is a topology and endpoint load-balancing approach that uses several independent network planes. They can be part of the same design.

Does multiplane make one Ethernet network safe for every traffic type?

No. It improves path use and failure handling for the scale-out fabric. Storage, tenant access and management still need separate capacity, policy and failure decisions.

Should storage use the GPU compute planes?

It can in some designs, but the operator must prove performance isolation and recovery under concurrent load. Heavy storage or stricter fault isolation may justify reserved planes or a separate fabric.

Does management need high-speed Ethernet?

Usually not. It needs independent and secure reachability. A modest out-of-band network is more valuable than sharing a fast fabric that may be unavailable during the incident.

Will PLB prevent every network slowdown?

No. It can help avoid degraded planes in supported configurations. Endpoint faults, congestion outside the observed path, bad topology, software bugs and application imbalance can still reduce performance.

Can GPUMachines design and source a multiplane fabric?

GPUMachines can map GPU systems, NIC rails, switch planes, optics, storage and management networks into a complete cluster proposal, subject to product availability and current compatibility requirements.

Sources and Further Reading

Verdict

Spectrum-X Multiplane is a scale-out network design, not permission to collapse the whole data centre into one fabric. It can divide high endpoint bandwidth across independent planes and use endpoint intelligence to avoid a damaged path. Spectrum-6 supplies the switch density for much larger deployments.

Keep management independent. Decide storage and Scale-In access from their own traffic and failure requirements. Share hardware only where the acceptance test proves isolation; separate it where correlated congestion or recovery risk costs more than the extra switches.

GPUMachines can turn the workload and rack count into a plane, rail, switch, optic and cable design, then test the proposed failure behaviour before deployment. The goal is not the greatest number of fabrics. It is the smallest network architecture that keeps useful GPU work moving when one ordinary component stops behaving perfectly.

NVIDIA Spectrum-6 Multiplane: One Fabric or Separate Traffic?