Storage for a 500-GPU AI Cluster

A 500-GPU cluster can lose more time to slow checkpoints and dataset staging than to compute faults. Size the storage path from job behaviour, not a petabyte target.

A 500-GPU cluster is large enough for storage mistakes to become expensive in minutes. A checkpoint that runs long, a metadata service that folds under thousands of file opens, or a staging job that misses its window can leave an entire training allocation waiting. The invoice still arrives for every idle accelerator.

That makes storage a scheduling problem as much as a hardware purchase. Capacity matters, but the useful design starts with four measurements: how quickly training workers consume data, how large the coordinated checkpoints are, how many files the pipeline touches, and what must happen while a drive, node or switch is unavailable.

This guide treats "500 GPUs" as a class of deployment. Sixty-three eight-GPU servers give 504 GPUs, while a four-GPU design needs 125 compute nodes. Those two clusters have similar accelerator counts and very different client counts, NIC populations, rack layouts and metadata pressure. GPUMachines sizes the storage and fabric from the actual node contract rather than applying one bandwidth figure to both.

Executive decision

A 500-GPU AI cluster normally needs more than one storage tier:

Local NVMe absorbs temporary files, caches and work that belongs to one node.
A high-performance shared tier serves active training data and receives coordinated checkpoints.
Object or capacity storage holds the full dataset estate, model artefacts and material that does not need flash latency.
An independent protection path covers deletion, corruption, compromise and site loss.

The shared tier may use community Lustre, DAOS or Ceph where the operating team wants control and has the right skills. Commercial routes such as WEKA, DDN EXAScaler and PEAK:AIO package different mixes of performance engineering, management, support and qualified hardware. They are not interchangeable, and none should win a tender before a representative workload test.

For a project at this scale, GPUMachines can design the storage servers, GPU client paths and storage fabric together. That work should start before the final GPU node and switch quantities are frozen.

What "500 GPUs" changes

An eight-GPU server can read training shards from local NVMe very quickly. Sixty-three of those servers asking for data at once turn a comfortable single-client test into a distributed systems test. The storage layer has to serve concurrent readers, preserve fair access, deal with hot directories and accept checkpoint bursts without allowing one workload to flatten every other queue.

Client count also changes the failure model. A bad route, mismatched MTU or overloaded leaf switch can affect a whole rack of compute. One slow worker can hold a synchronous training step behind the other ranks. Average throughput looks healthy while job time deteriorates.

Then there is restart behaviour. A failed training job may cause dozens of nodes to reload the same model state at nearly the same time. This "restart storm" has a different read shape from steady training. Procurement tests that exercise only warm-cache sequential reads miss it.

Measure the four storage workloads separately

Training reads

Training input can look friendly when data is packed into large shards and workers read ahead. It becomes harder when the dataset contains millions of small objects, random sample selection defeats caching, or preprocessing opens and closes files repeatedly. Record the file-size distribution and the number of opens, stats and directory operations per second. Aggregate read bandwidth on its own cannot describe this workload.

The data loader matters too. CPU decoding, Python worker limits, container settings and remote object clients can starve a GPU even when the array has spare capacity. Test the whole path from storage media to framework queue.

Checkpoint writes

Checkpointing produces a deadline, not a steady bandwidth target. Suppose a job must write C bytes and the acceptable pause is T seconds. The minimum application-level write rate is:

checkpoint write rate = C / T

That result needs headroom for protection work, protocol overhead, metadata, competing jobs and a degraded component. Do not multiply vendor drive bandwidth and call the job finished. Measure the committed checkpoint at the storage system, then confirm that it can be read back within the required recovery window.

Parallel jobs may write one file per rank, a smaller set of aggregated files, or a distributed checkpoint format. Each form stresses the storage platform differently. The framework and checkpoint library belong in the acceptance test.

Metadata and small files

Directory walks, file creation, stat calls and deletes are often the first limit in research estates. Thousands of experiments create temporary files and model versions even when the bulk dataset changes slowly. Lustre Metadata Targets, CephFS Metadata Servers, DAOS services and commercial metadata implementations all need sizing against this behaviour.

Small-file tests should include cold cache, realistic directory depth and the intended client count. A benchmark from one client against one directory says little about a 500-GPU environment shared by several teams.

Ingest, staging and export

The hot filesystem cannot help if data arrives through a narrow object gateway or WAN connection. Calculate how much material must move into the training tier before a reservation starts, how often that happens, and whether staging competes with checkpoints. Export matters as well: completed models, logs and derived datasets need somewhere durable to go.

A fast shared tier with a slow capacity tier creates an operational queue. Somebody then starts retaining everything on expensive NVMe because demotion takes too long.

A sizing worksheet that survives scrutiny

The right worksheet keeps workload evidence separate from protection and hardware assumptions.

| Input | What to record | Why it changes the design | | --- | --- | --- | | Compute nodes and GPUs per node | Physical clients, GPU density and rack location | Determines client fan-out and storage-network endpoints | | Active dataset | Working set used during one training window | Sizes the hot tier without putting every cold byte on NVMe | | Read demand | Sustained bytes per second per active node and concurrent node count | Sets the application-level read target | | Checkpoint pattern | Bytes per job, writer count, interval and completion deadline | Sets burst write and metadata targets | | File population | Count, size distribution and directory shape | Exposes metadata and small-I/O risk | | Protection | Replication or erasure coding, snapshots and backup path | Changes usable capacity and write cost | | Failure objective | Allowed effect of a lost drive, server, NIC, switch or rack | Sets redundancy and degraded-mode headroom | | Growth window | Expected data and client growth before expansion | Prevents a forklift replacement after the first research cycle |

Use measured application demand where possible. If a new workload has no evidence, run a smaller representative pilot and preserve its traces. Numbers copied from another organisation's model rarely survive changes in data format, augmentation, cache policy or checkpoint code.

The storage architecture should have layers

Local NVMe is a tool, not the shared filesystem

GPU servers often include local NVMe. Use it for container layers, scratch files, dataset cache and temporary spill where losing a node does not lose the authoritative copy. Local media removes network traffic and can smooth repeated reads.

It also fragments capacity. A scheduler must know which node holds which cache, and jobs need a predictable fill and eviction process. Without that ownership, local drives become abandoned islands full of old datasets.

The active shared tier

This tier needs predictable concurrency. It may be all-NVMe, hybrid, or built from storage servers with separate metadata and data media. Its usable size should follow the active working set, checkpoint retention and operational spare requirement.

Reserve capacity for rebuild and maintenance. A filesystem that reaches its target only when every drive is healthy and the cluster is nearly empty has not met the service target.

Object and capacity storage

Object storage works well for durable dataset repositories, model artefacts and workflow exchange when applications can use object APIs or stage data. It can also back a tiered commercial filesystem. Object access is not a drop-in substitute for a low-latency POSIX namespace, especially when a training framework performs many small reads.

Ceph RGW, a dedicated object platform or cloud object storage can fill this role. The choice depends on data gravity, egress cost, support and the time allowed for promotion into the hot tier.

Backup is separate

Replication keeps a service available after component failure. It does not protect against every operator mistake, software defect or compromised credential. Keep an independent copy, set a recovery objective and test restoration. Snapshots help, but snapshots inside the same administrative and failure boundary do not complete the plan.

Storage network design at 500-GPU scale

NVIDIA's DGX BasePOD material separates compute, management/storage and out-of-band networks. A 500-GPU design may use a different reference architecture, but the lesson holds: assign each network a role and count its endpoints before selecting switches.

Storage traffic can run over Ethernet, InfiniBand or both, depending on clients and platform support. What matters is an end-to-end path with enough bandwidth, redundant links and a tested failure response. The storage fabric must account for client reads, checkpoint writes, rebuild traffic, tiering and administrative movement.

Do not infer an installed storage NIC from a server's available PCIe slots. Verify the adapter model, port speed, NUMA location and supported software path. For GPUDirect Storage, NVIDIA's guidance makes PCIe affinity, NIC placement, ACS and IOMMU behaviour part of the benchmark. A filesystem logo cannot repair a poor path between GPU and NIC.

Open-source choices

Lustre

Lustre is the most direct open-source candidate when the dominant requirement is a shared POSIX namespace with high parallel throughput. Metadata Servers expose Metadata Targets; Object Storage Servers expose Object Storage Targets. Clients stripe file data across targets.

That separation gives architects useful control. Metadata and bulk throughput can scale on different hardware. It also creates work: target layout, failover, client versions, tuning, monitoring and upgrades need owners. Community Lustre can be economical for a team that already operates HPC storage. It is a risky way for a thin platform team to learn distributed filesystems while a 500-GPU cluster waits.

DAOS

DAOS is not a conventional parallel filesystem. It is an open-source distributed object store designed around NVMe, user-space I/O and low-latency fabrics, with native APIs plus interfaces for MPI-IO, HDF5 and POSIX access through dfuse. DAOS 2.6 supports persistent memory or a Metadata-on-SSD path using DRAM with NVMe logging for its storage-class-memory role.

DAOS deserves consideration where applications can use its data model or where its POSIX path has been validated with the actual framework. Check client and server platform support carefully. Its architecture can suit data-intensive research, but it asks more of application integration and operating knowledge than a familiar NFS mount.

Ceph

Ceph provides object, block and file services over RADOS. That breadth makes it a strong durable data platform for private clouds, Kubernetes volumes, S3-compatible repositories and shared project storage. CephFS can serve POSIX clients directly against RADOS while Metadata Servers coordinate namespace operations.

For the hottest synchronous training tier, Ceph must earn its place in testing. Recovery and rebalancing consume the same CPU, media and network resources that serve clients. A good design leaves room for those events. Many estates use Ceph as the durable capacity layer and a more focused parallel filesystem for active training.

Commercial choices

DDN EXAScaler

DDN EXAScaler packages Lustre with DDN's hardware designs, management, support and performance engineering. It is a natural shortlist candidate where the buyer wants Lustre semantics and scale without owning every integration decision. DDN publishes large performance and utilisation claims; treat them as vendor statements until a configuration reproduces the required workload.

The value is not a different filesystem name. It is the qualified system, escalation route and operating tooling around Lustre.

WEKA

WEKA uses an NVMe-led software architecture and supports POSIX, NFS, SMB, S3 and GPUDirect Storage paths. Its object tier can move colder file data away from SSD while metadata stays on flash. That combination suits organisations that want one managed namespace across active flash and a capacity tier.

WEKA dedicates host resources and has specific networking, SR-IOV, RDMA and compatibility requirements. Include those CPU cores, NICs and operational constraints in the bill of materials. The licence price should be compared with the people and validation work required by a self-managed platform.

PEAK:AIO

PEAK:AIO positions its AI Data Server software around standards-based NFS, RDMA, NVMe-oF, GPUDirect Storage and newer pNFS work. Its published single-node designs target a simpler footprint than a large traditional parallel filesystem. That can be attractive for smaller and mid-scale GPU estates or a modular storage service.

A 500-GPU procurement needs a validated scale-out design, not an extrapolation from one appliance. Ask PEAK:AIO to document metadata scaling, client count, multi-node failure behaviour, aggregate bandwidth and the supported network topology for the proposed configuration. The answer may be a fit; the test still has to happen.

Shortlist by operating problem

| Main requirement | Strong candidates to investigate | Main caveat | | --- | --- | --- | | Open POSIX parallel scratch and training | Community Lustre | Requires Lustre engineering and disciplined operations | | Supported Lustre platform | DDN EXAScaler | Commercial cost and qualified configuration boundaries | | Native object/data APIs with NVMe and POSIX access | DAOS | Application path, platform support and skills need validation | | Multi-service object, block and file capacity | Ceph | Hot training performance and recovery interference need proof | | Commercial NVMe namespace with object tiering and GDS | WEKA | Dedicated resources, compatibility and licence model | | Compact standards-based AI data server or pNFS route | PEAK:AIO | Confirm reference scale before applying it to 500 GPUs |

This table creates a shortlist. It does not rank products because the workload has not supplied evidence for a ranking.

Acceptance testing before purchase

Run the candidate platform with the intended clients, network and protection settings. At minimum, test:

Cold-cache training reads using the real dataset layout and loader.
Checkpoint writes from the expected number of ranks, followed by a timed restart.
Metadata-heavy experiment creation, enumeration and cleanup.
A drive or storage-node failure while jobs continue.
A path or switch failure, including reconvergence and client behaviour.
Rebuild, tiering or backup traffic alongside foreground work.
Growth beyond the day-one capacity and client count.

MLPerf Storage provides a public framework for examining how storage supplies training data, while NVIDIA supplies gdsio for GPUDirect Storage paths. Use these tools where they fit, then add application tests. Synthetic throughput is evidence about one layer, not the whole service.

Configuration paths

Research team with Lustre staff

Build a community Lustre active tier on qualified storage servers, separate metadata and data roles, and use object or Ceph storage for durable capacity. Keep spare parts, automate client deployment and rehearse upgrades on a staging cluster.

Enterprise team seeking one support path

Shortlist DDN EXAScaler or WEKA, depending on whether Lustre semantics or a multiprotocol NVMe data platform better fits the applications. Put support response, upgrade ownership and degraded-mode performance into the contract and acceptance plan.

Private-cloud estate already using Ceph

Use Ceph for durable object, block and shared project data. Test CephFS for active training, but allow a separate fast tier if checkpoint and metadata results miss the target. Existing skills make Ceph valuable; they do not make every workload a Ceph workload.

NVMe-focused engineering environment

Evaluate DAOS where applications can benefit from native or middleware integrations. Compare its dfuse path with the frameworks that cannot use DAOS APIs. A PEAK:AIO design may also fit teams that want standard NFS or pNFS access with a smaller operational footprint, subject to scale validation.

Who should not build at this scale yet?

A team with unmeasured workloads, no storage owner and no data-protection plan should not buy a 500-GPU storage system. Start with a pilot or hosted capacity, gather traces and settle the operating model. The full cluster will magnify every unresolved dependency.

It may also be wrong to put the entire dataset estate on the fastest flash tier. If only a small working set trains each week, spend on staging automation and a balanced capacity layer instead.

Buying through GPUMachines

GPUMachines can turn the workload evidence into storage-node, NVMe, NIC, switch, rack-power and cooling requirements. For open-source deployments, that includes server and fabric sourcing plus a review of failure domains. For commercial routes, GPUMachines can scope WEKA, DDN and PEAK:AIO options and coordinate the required hardware and deployment path, subject to project availability and regional terms.

The same exercise can compare on-premise deployment with GPU Cloud or Buy & Host. At 500 GPUs, data location and facility readiness can decide the operating model before a filesystem does.

FAQ

How much throughput does a 500-GPU cluster need?

There is no safe per-GPU constant. Measure bytes consumed per active client, concurrent clients and checkpoint deadlines, then add headroom for failures and background work. Dataset format and preprocessing can change the result by a large margin.

Does a 500-GPU cluster require GPUDirect Storage?

No. GDS can reduce CPU copies and improve supported coarse-grained transfers, but the application, filesystem, NIC, GPU and PCIe topology must use the direct path. A well-designed standard path can still be correct; a misconfigured GDS path can fall back without delivering the expected result.

Is Lustre always the best option for distributed training?

No. Lustre is strong for parallel POSIX throughput, especially when the team understands it. WEKA or DDN may reduce integration and support risk, DAOS may fit applications using its data model, and Ceph may be better for a broad durable service.

Can Ceph support the whole environment?

It can provide file, object and block services, but the hot training tier still needs a workload test during recovery. A two-tier Ceph plus parallel-filesystem design is often easier to justify than forcing one platform to handle every I/O pattern.

Where does PEAK:AIO fit?

PEAK:AIO is worth examining for standards-based AI storage with NFS, RDMA, NVMe-oF, GDS and pNFS options. For a 500-GPU design, require a vendor-backed scale-out architecture and acceptance results for the proposed client count.

Should storage use the compute InfiniBand fabric?

It can, but model compute and storage as separate network roles even when they share physical switches. Prove port capacity, quality of service, failure behaviour and the consequence of concurrent collective and checkpoint traffic.

Verdict

Storage for 500 GPUs cannot be selected from a capacity figure or a vendor leaderboard. Build the target from training reads, checkpoint deadlines, metadata behaviour and failure objectives. Then decide how much engineering the organisation wants to own.

Community Lustre, DAOS and Ceph can produce capable systems when the team has the right skills. WEKA, DDN EXAScaler and PEAK:AIO trade software and support cost for different amounts of packaged engineering. The winning design is the one that survives the workload and failure test with an operating model the buyer can sustain.

Plan storage and networking for a 500-GPU AI cluster with GPUMachines.

Sources and further reading

Named performance and scale statements on vendor pages are vendor claims. Reproduce the required service level with the proposed clients, network, protection policy and failure conditions before purchase.

Storage for a 500-GPU AI Cluster: Architecture, Sizing and Vendor Shortlist