Rack-scale AI systems concentrate hundreds of accelerators, switches and links into one computing domain. That architecture can deliver very high model throughput, but it also changes the meaning of reliability. A single failed link or process should not force an operator to discard a large inference request, restart an entire rack or wait several minutes for services to rebuild.
NVIDIA says NVLink 6 and the surrounding Vera Rubin software stack address faults at several layers: physical link repair, fabric rerouting, management-plane containment, inference-process recovery and service procedures for failed hardware. The important buyer question is not whether those features exist on a diagram. It is whether the proposed system can demonstrate recovery within the latency, data-integrity and availability limits of the intended workload.
This article focuses on fault behaviour. For the wider platform architecture, including Vera CPUs, Rubin GPUs, NVLink 6 and rack composition, read our NVIDIA Vera Rubin NVL72 explainer.
Resilience starts below the application
NVIDIA describes a layered model because no single mechanism can cover every fault. A transient bit error needs a different response from a dead cable, a switch failure or a crashed inference process.
At the physical layer, NVLink 6 uses forward error correction to repair some errors without asking the sender to retransmit data. If that is not enough, Physical Layer Retry can resend affected traffic. NVIDIA also describes UPHY recovery for more persistent link problems. These mechanisms aim to repair small faults close to where they occur, before software sees a broken communication path.
At the link and fabric layer, credit-based flow control manages buffer availability, while link rebalancing moves traffic away from a degraded path. Fault containment is meant to stop one problem from spreading through the scale-up domain. Buyers should not interpret this as proof that every workload is unaffected. Rerouting may reduce available bandwidth or change congestion elsewhere in the fabric.
What "contain and drain" means
The NMX Controller is NVIDIA's management component for the NVLink fabric. NVIDIA describes a contain-and-drain process in which the system isolates the affected component, allows in-flight work to reach a controlled state and then moves or restarts the affected workload.
NMX high availability is intended to keep fabric management available if a controller fails. NVIDIA also says the data plane can continue forwarding traffic across a management CPU reset. That distinction matters. A management service may be temporarily unavailable while the already-programmed fabric continues to carry data.
An acceptance test should verify both cases. Restart a management controller and observe whether application traffic continues. Then create a condition that requires a new routing decision and confirm that the surviving controller can make it. Uptime during a passive reset does not prove that failover works during an active fabric event.
Fast inference recovery needs prepared capacity
At the software layer, NVIDIA describes Shadow Engine Recovery in NVIDIA Dynamo. A standby process is kept warm with independent NCCL and NIXL communicators so it can take over without rebuilding the full serving environment from cold state.
NVIDIA reports a test on B200 in which cold restart took 283 seconds while shadow-engine recovery took 7.3 seconds. This is a vendor-reported comparison, not a GPUMachines benchmark. The result shows why pre-warmed capacity can matter, but it does not establish a universal recovery time for other models, context lengths, cache sizes or storage systems.
Standby capacity is not free. It consumes memory, compute reservation and operational attention. A buyer should decide which services justify it. A customer-facing inference endpoint may need a warm standby, while a batch job may be allowed to restart from a checkpoint.
Checkpoint recovery is still a roadmap item
NVIDIA also discusses prototype NCCL support for cuda-checkpoint, which is intended to help applications save and restore GPU process state. The company says general availability is expected by the end of 2026.
That date and support status belong in procurement records. A prototype or announced roadmap should not be treated as an available production feature. If checkpoint recovery is required at deployment, ask for the supported software versions, framework coverage, storage requirements and a live demonstration on the quoted system.
Checkpoint speed can also be limited by storage. Restoring a large distributed job may move terabytes of state. The scale-up fabric can be healthy while the recovery path stalls on the file system, object store or host network. Resilience testing therefore has to include storage and the GPU cluster configuration, not only the GPU links.
NVLink is scale-up, not the whole cluster network
NVLink connects accelerators within a tightly coupled scale-up domain. Ethernet or InfiniBand normally connects racks and services at the scale-out layer. A resilient NVLink fabric does not remove the need to design redundant network paths, management access, storage connectivity and control services.
This is particularly important for large-model inference. A request may depend on the scale-up fabric inside a rack, the scale-out network between services, a key-value cache tier and external storage. An availability claim should identify which of those dependencies were included.
For smaller workloads, a full rack-scale domain may be unnecessary. A buyer may achieve a simpler failure boundary with one or more HGX servers, accepting a lower scale-up ceiling in return for easier service isolation. The correct architecture follows the model size, parallelism method, service target and growth plan rather than the newest interconnect specification.
A practical acceptance-test matrix
| Fault to inject | Expected system behaviour | Evidence to collect | | --- | --- | --- | | Transient link errors | Error correction or retry without application failure | Link counters, latency trace, application errors | | One degraded NVLink path | Traffic rebalanced with a documented performance impact | Route state, bandwidth, tail latency | | Fabric management CPU restart | Data plane continues; management service returns cleanly | Packet/collective continuity, controller logs | | Active NMX controller failure | Standby assumes control within the stated target | Failover time, configuration consistency | | Inference worker crash | Warm process takes over or request is retried safely | Recovery time, lost requests, output integrity | | GPU or switch taken out of service | Workload drains and failed component becomes serviceable | Scheduler state, job completion, maintenance procedure | | Checkpoint restore | Job resumes from a known point without corrupt state | Restore time, model state validation, storage load |
Run these tests under representative load. A failover that works while the cluster is idle may behave differently when links and memory are close to saturation. Measure both time to detect the fault and time to restore useful service. Vendors sometimes report only the latter.
Questions to put in the purchase specification
1. Which faults recover automatically and which require an operator? 2. What performance reduction should be expected after one link or switch is isolated? 3. Which software versions provide NMX high availability and Shadow Engine Recovery? 4. How much standby GPU memory and compute must be reserved? 5. Are in-flight requests replayed, resumed or failed back to the client? 6. How is output integrity checked after recovery? 7. Can a failed component be replaced without shutting down the full rack? 8. Which recovery features are shipping, optional, preview or roadmap?
Answers should be attached to the system acceptance plan. Terms such as self-healing or lossless recovery are too broad unless the supplier defines the failure, workload and permitted service impact.
When rack-scale resilience is worth paying for
The strongest case is a large service that needs a single scale-up domain and cannot tolerate the restart cost of a rack-sized model deployment. The value comes from protecting expensive work and maintaining service, not from adding another feature to a specification sheet.
The case is weaker when the workload fits comfortably inside a smaller server, batch jobs can restart cheaply, or the organisation cannot operate the management and observability stack. In those cases, simpler failure domains or hosted operations may produce better availability in practice. Teams comparing ownership with an operated facility can examine Buy & Host and specify recovery tests as part of the service agreement.
NVLink 6 resilience should therefore be purchased as a verified operational outcome. Physical repair, rerouting, controller failover and process recovery are useful building blocks. The buying decision rests on whether those layers work together on the proposed hardware, software and workload.