Storage Solutions for GPU Clusters

GPU cluster storage is a pipeline: local NVMe, shared files, object storage, archive and the fabric between them. One fast appliance does not design that pipeline.

A GPU cluster does not have one storage problem. Raw data arrives through one path, prepared datasets train through another, checkpoints create bursts, users expect an ordinary project namespace, and completed models need durable retention. Buying one fast appliance may improve a phase; it does not design the pipeline.

Storage architecture starts by assigning each dataset state to a service. Local NVMe handles disposable work close to compute. A shared high-performance tier serves active training and checkpoints. Object or capacity storage retains the wider data estate. Backup and archive sit outside the availability boundary of the live platform.

Open-source Lustre, DAOS and Ceph can fill different roles in that system. Commercial WEKA, DDN EXAScaler and PEAK:AIO reduce or reshape the engineering the buyer owns. The correct combination depends on access methods, staff, failure objectives and the cost of moving data between tiers.

Executive architecture

Most serious GPU environments should plan five storage functions:

1. A source of truth for datasets and model artefacts. 2. A fast shared namespace for active jobs. 3. Local cache or scratch on compute nodes. 4. A controlled path for ingest, promotion and demotion. 5. Independent recovery copies and tested restoration.

Some platforms combine several functions, but the responsibilities remain. A WEKA tiered namespace still depends on its object tier. A Ceph cluster serving object, block and file still needs backup. A Lustre scratch system still needs a durable place for material that outlives a training campaign.

GPUMachines designs scale-out storage as part of the GPU infrastructure, including client NICs, switch ports, storage-node balance, rack power and deployment location.

Start with data states, not product names

Map the lifecycle of one dataset before choosing a filesystem.

It may arrive as objects from a laboratory, customer or cloud bucket. A preparation job validates, filters and reshards it. Training jobs read the prepared form many times, then write checkpoints and logs. Some outputs become released models; others can be deleted. Governance may require retention of the original data and the exact material used for a model.

Each transition has a deadline and an owner. If promotion from object storage takes six hours, the scheduler should know when the data is ready. If checkpoint retention fills flash in two days, automation must demote or expire it. Storage hardware cannot compensate for missing lifecycle policy.

The five-tier model

1. Ingest and source repository

Object storage is often the natural entry point because applications can upload independent objects, metadata can describe provenance, and capacity scales without a POSIX directory tree. Ceph RGW, cloud object services and commercial object platforms can serve this function.

The source repository optimises durability, governance and capacity. It may not meet the latency or small-read behaviour of training. Measure staging time and object-request patterns before deciding whether jobs should read it directly.

2. Active shared storage

The hot tier supplies datasets to many nodes and accepts checkpoints inside a defined window. It usually needs a POSIX or application-specific shared interface, predictable client concurrency and enough metadata capacity for the experiment structure.

Lustre, DDN EXAScaler, WEKA, DAOS and other platforms can fill this role. CephFS and PEAK:AIO may fit when their tested access paths meet the service target. The platform choice follows the workload; "all flash" does not settle the protocol, metadata or recovery question.

3. Compute-node NVMe

Local NVMe holds cache, scratch, container layers and temporary preprocessing output. It reduces repeated shared reads and gives independent jobs high bandwidth without a network round trip.

Local storage needs orchestration. Decide how the scheduler fills it, how jobs discover cached versions, what can be evicted and what happens after a node failure. Never let local scratch become the only copy of a valuable checkpoint.

4. Capacity and warm data

Prepared datasets may remain useful without requiring flash performance every day. A capacity tier based on object storage, Ceph, dense HDD or high-density flash can hold them. Commercial filesystems may extend one namespace into object storage; separate systems can use an explicit data mover.

One namespace reduces visible movement but does not make every access equally fast. Explicit staging adds workflow work but gives the scheduler a clear readiness state. Pick the operational model the team can observe.

5. Protection and archive

Snapshots, replicas and erasure coding improve the live service. Backup protects against a different class of event. Keep recovery copies outside the same administrative and failure boundary, set retention from governance needs and practise restoring at useful scale.

Archive may use object lock, tape or dense flash depending on access time and risk. The answer belongs to the data policy, not the training benchmark.

Separate the workload classes

| Workload | Dominant concern | Architecture response | | --- | --- | --- | | Dataset ingest | Sequential transfer, object operations, validation | Scale gateways and staging separately from training | | Multi-epoch training | Concurrent reads and cache behaviour | Hot shared tier plus managed local caching | | Distributed checkpoint | Burst writes, synchronisation and commit | Test checkpoint format and completion deadline | | Research workspace | Metadata, permissions, quotas, small files | Size namespace services and governance controls | | Model serving | Model-load time, replica placement, logs | Local replica cache plus durable model repository | | RAG and analytics | Mixed object, database and file access | Do not force every component through one filesystem | | Backup and archive | Retention, immutability and restore | Independent protection path |

The table prevents a common design error: using the hardest workload to justify premium storage for every byte.

Open-source building blocks

Lustre for parallel POSIX work

Lustre divides metadata and object storage roles, allowing clients to access striped file data in parallel. It fits active training, scratch and checkpoint workloads that need a shared POSIX namespace. Hardware choice remains open, and experienced HPC teams can tune the system around their jobs.

Operations are part of the architecture. Metadata Targets, Object Storage Targets, failover, client versions, quotas, monitoring and upgrades all need care. Community Lustre is a strong component when the organisation accepts that ownership.

DAOS for native distributed data access

DAOS presents a native distributed object model and integrations for MPI-IO and HDF5, with POSIX access through dfuse. It uses user-space I/O and NVMe-oriented storage engines over an OpenFabrics Interface network. Current DAOS documentation describes both persistent-memory and Metadata-on-SSD options for the metadata role.

DAOS can suit scientific and AI workflows prepared to use its interfaces. It should not be treated as a transparent replacement for every POSIX application. Validate framework behaviour, client support and team skills.

Ceph for the broader data estate

Ceph unifies object, block and file services over RADOS. This makes it useful for dataset repositories, Kubernetes or OpenStack storage, shared project data and capacity tiers. CephFS clients access file data in RADOS while Metadata Servers manage namespace state.

The same broad platform can become a large failure and maintenance boundary. Separate pools and media classes deliberately, leave room for recovery, and decide whether active training deserves its own filesystem. Ceph versus commercial AI storage covers that choice in more detail.

Commercial routes

DDN EXAScaler

DDN EXAScaler packages Lustre into a supported platform with qualified hardware, management and vendor engineering. It suits large training and HPC buyers who want Lustre's parallel filesystem model without qualifying the entire system alone.

The procurement case rests on support scope, operating tooling and tested configuration as much as throughput. Validate the exact client, fabric and failure target; vendor-scale claims do not replace acceptance work.

WEKA

WEKA supplies a distributed NVMe platform with POSIX, NFS, SMB, S3 and GPUDirect Storage access in supported configurations. Its tiering model can keep active data on SSD while moving colder file content into object storage, with metadata remaining on SSD.

WEKA's performance network uses dedicated resources and has specific NIC, DPDK, RDMA and compatibility requirements. Those cores, memory and ports belong in the architecture and cost model. The product can combine hot and warm data in one visible namespace, but the object tier still needs capacity, protection and performance planning.

PEAK:AIO

PEAK:AIO takes a software-defined route based on partner server hardware and standard access methods including NFS, RDMA, NVMe-oF, GPUDirect Storage and pNFS work. Its compact AI Data Server approach can reduce the starting footprint for teams that do not need a large traditional parallel-filesystem estate.

Fit depends on scale. Published single-node material should not be extended to hundreds of clients without a reference design. Ask for metadata, multi-node, failover, supported-kernel and network evidence for the proposed deployment.

Commercial and open source can coexist

Architecture does not require one licensing philosophy from ingest to archive. A commercial high-performance tier may stage from open-source Ceph. A community Lustre cluster may protect released outputs in a supported object platform. DAOS may serve scientific applications while ordinary user shares remain elsewhere.

Mixed systems cost more to integrate, and data movement can fail. They also let each tier match its workload. The decision rests on whether the organisation can automate and observe transitions.

Use one authoritative catalogue to record dataset location, version and readiness. Without it, users copy data between filesystems and nobody knows which copy trained the model.

Design the storage network as its own service

Storage is a network role. Count client and server endpoints, port speeds, redundant paths, optics and uplinks before choosing switches. If storage shares a physical fabric with compute, preserve separate traffic classes and test simultaneous collective communication, reads and checkpoints.

Ethernet may suit a familiar operational model and multiprotocol services. InfiniBand may align with an HPC environment and low-latency clients. Both can carry poor designs. Oversubscription, bad routing, inconsistent MTU, weak failover or one under-connected storage server can limit either technology.

Client NIC placement matters, especially for GPUDirect Storage. Confirm GPU, NIC and PCIe affinity. A server with enough aggregate PCIe lanes may still route traffic through an unfavourable root complex.

Management and out-of-band access should remain distinct from the data path. A storage incident is not the moment to discover that management traffic depended on the failed fabric.

Capacity is more than raw drive total

Calculate usable capacity after protection, filesystem reserve, snapshots, rebuild headroom, metadata and tiering policy. Include endurance and write amplification for flash. For HDD or QLC capacity layers, model rebuild time and bandwidth rather than relying only on cost per terabyte.

Active capacity follows the working set, checkpoint retention and staging overlap. A team that trains on 200 TB at a time may not need its full multi-petabyte repository on premium NVMe. It does need a promotion path that completes before the reservation begins.

Growth planning should identify the unit of expansion. Adding one server may increase capacity, performance, failure domains and software cost together. Understand which resource the next unit actually fixes.

Metadata deserves its own budget

AI storage conversations often begin with terabytes per second because the GPU bill is visible. Research estates also create huge namespace populations. Dataset extraction, experiment tracking and checkpoint retention can make metadata the user-facing bottleneck.

Measure creates, opens, stats, renames, directory listings and deletes. Test the real directory shape and client count. Place metadata on suitable media and size the service processes or targets independently where the platform allows.

Packing small files into shards may improve training, but it changes randomisation and failure recovery. Treat data format as an application decision with storage consequences.

Multi-tenancy and scheduler integration

A shared cluster needs project quotas, access controls, priority and fair service. Bulk ingest should not delay a production checkpoint. One user's recursive scan should not ruin job setup for everyone else.

Connect the scheduler to data staging and storage health. A job should wait in a cheap queue until its dataset is on the required tier. Expose cache and promotion state, then clean local and shared scratch after the retention window.

For Kubernetes, review CSI behaviour, mount propagation, security contexts and upgrade compatibility. For Slurm, integrate prolog and epilog scripts carefully and make failures visible rather than silently starting a job without its data.

Acceptance testing

Build tests from the application and operating model:

Cold and warm training reads with the intended number of clients.
Checkpoint write, commit and restart using the real framework format.
Metadata-heavy project creation and cleanup.
Concurrent ingest, training and background protection traffic.
Drive, storage-node, client-path and switch failures.
Expansion and rebalancing while the service remains active.
Restore from the independent protection copy.

Use MLPerf Storage, gdsio, fio and filesystem tools where they answer a specific question. Preserve the versions, parameters, cache state, topology and protection settings. A result without configuration context cannot support procurement.

Four practical architecture patterns

Compact research cluster

Use durable object storage, a small shared all-NVMe service and local cache. PEAK:AIO, a compact WEKA configuration or a modest open-source filesystem may fit, depending on support and protocol needs. Avoid building multi-petabyte complexity for a working set measured in tens of terabytes.

HPC and AI convergence

Use community Lustre or DDN EXAScaler for active POSIX data and checkpoints, with a capacity repository for retained datasets. Existing HPC operations and Slurm integration can carry much of the service model.

Private-cloud data platform

Use Ceph for object, block and general file services. Add a focused training tier only when CephFS tests show that the hot workload needs one. This keeps broad infrastructure services on the platform the team already understands.

NVMe-led enterprise platform

Use WEKA for a high-performance shared namespace with object tiering, or evaluate DAOS where applications can use native and HPC data interfaces. The former buys packaged operations; the latter keeps an open software model with more integration responsibility.

Who should avoid a multi-tier design?

Very small teams may create more failure points than value by operating several storage systems. A single supported shared platform plus independent backup can be the better answer. The architecture should not become more complicated than the team that runs it.

Conversely, a large organisation should not force every workload into one product merely to reduce supplier count. Consolidation that makes checkpoints slower or restores harder is an accounting win and an engineering loss.

Buying through GPUMachines

GPUMachines can source the storage servers, enterprise SSDs, capacity media, NICs, switches, optics and rack equipment for open-source Lustre, DAOS and Ceph environments. It can also develop commercial WEKA, DDN and PEAK:AIO configurations with their required hardware and support routes, subject to supplier and regional availability.

That design can extend to GPU Cloud, Buy & Host or a hybrid system. Data gravity, WAN staging time and facility power often settle deployment location before the storage product shortlist.

FAQ

Do GPU clusters always need a parallel filesystem?

No. Small inference or independent-workload clusters can use object storage plus local NVMe. A parallel shared tier earns its place when many clients need concurrent access, common checkpoints or one POSIX namespace.

Can object storage feed training directly?

Yes, when the framework uses object APIs efficiently and the data layout avoids excessive request overhead. Other pipelines should stage prepared data into a faster shared tier. Test first-read and repeated-epoch behaviour.

Should local NVMe be RAID protected?

That depends on its role. Disposable cache may favour capacity and speed over protection. Local checkpoints or unique data need protection or immediate replication. Define what a node loss is allowed to remove.

Can Ceph be both the object tier and training filesystem?

It can, but CephFS training performance and recovery interference need proof. Shared hardware also increases the blast radius between services. Separate pools, media or platforms where service targets conflict.

Where does DAOS fit beside Lustre?

DAOS can serve applications using native object, MPI-IO, HDF5 or dfuse paths, while Lustre supplies familiar parallel POSIX access. Some environments use both and move datasets according to application needs.

Is a commercial platform easier to run?

Usually it narrows supported combinations and provides an escalation path. It still needs network design, capacity planning, monitoring, upgrades and workload testing. Read the division of responsibility in the support agreement.

How do we prevent storage tiers becoming data silos?

Use an authoritative catalogue, automated movement, versioned datasets and scheduler-visible readiness. Assign ownership for failed transfers and retention. Manual copying will eventually lose provenance.

Verdict

Good GPU cluster storage is a pipeline with named roles. Put hot, shared work on the service that meets its latency, metadata and checkpoint targets. Keep durable data on a tier built for retention, use local NVMe deliberately, and protect the estate outside the live platform.

Lustre, DAOS and Ceph give capable open building blocks. WEKA, DDN and PEAK:AIO package different commercial approaches. Select each for a job in the architecture rather than asking one product to solve every data state.

Design an open-source, commercial or mixed GPU storage platform with GPUMachines.

Sources and further reading

Treat named vendor performance and scale figures as vendor claims until the proposed configuration reproduces the buyer's workload and failure targets.

Designing Storage for GPU Clusters: Open-Source and Commercial Options