Best Storage for AI Training

A storage array can pass a bandwidth test and still starve training jobs. The failure usually sits in metadata, preprocessing, client paths or checkpoint bursts.

GPU utilisation drops, the storage dashboard looks quiet, and everybody blames the array. Sometimes they are right. Often the delay is a CPU decoder, an empty data-loader queue, a cold object fetch, a metadata lock or one slow worker holding back the other ranks. "GPU starvation" describes the symptom; it does not identify the failed layer.

The best storage for AI training is therefore the system that meets the application's service targets through normal operation and failure, with an operating model the team can sustain. Peak sequential bandwidth is only one part of that result.

This guide explains how to find the actual wait, size the shared and local tiers, test candidate platforms and stop expensive GPU nodes idling behind data work. It applies to open-source Lustre, DAOS and Ceph designs as well as commercial WEKA, DDN and PEAK:AIO routes.

Executive diagnosis

Start with the training process, not the storage quotation. Measure GPU activity beside data-loader queue depth, CPU utilisation, file operations, network counters, client latency and storage-server load. If GPUs wait while storage is idle, adding drives will not help. If every client reaches the same bandwidth ceiling or checkpoint time rises with node count, the shared path deserves attention.

A dependable AI training design usually combines local NVMe cache, a high-performance shared namespace, durable object or capacity storage and an independent protection path. The balance changes with dataset reuse, checkpoint volume, file size, concurrency and recovery objectives.

GPUMachines can review the storage servers and client fabric as part of a scale-out storage configuration. The review should include the framework and data format; otherwise, it stops at hardware.

What GPU starvation looks like

Training does not need constant storage traffic. Compute-heavy steps may leave the data path quiet, then preprocessing or checkpoint phases produce bursts. Diagnose the timeline rather than treating low average GPU utilisation as a storage verdict.

Common signs include:

GPU activity falling at epoch boundaries while new shards open.
Data-loader queues emptying before the next batch reaches device memory.
One rank reporting longer I/O time and delaying a synchronous step.
Checkpoint duration increasing sharply as more workers join the job.
Restarts taking far longer than steady training reads.
Storage clients showing retries, fallback paths or uneven NIC use.

These signs narrow the investigation. They do not prove which component needs replacement.

Trace the path end to end

A sample may travel through object storage, a staging service, a shared filesystem, a client page cache, CPU preprocessing, pinned host memory and a GPU copy. GPUDirect Storage can change part of that route for supported applications, but most pipelines still contain several queues and transformations.

Instrument each boundary. NVIDIA DCGM and nvidia-smi help show GPU and PCIe activity. Linux tools expose CPU, disk and network behaviour. Filesystem tools reveal client and server latency. Framework profiling shows whether the training process waits for data.

Keep clocks aligned across the systems. A five-second gap between storage and GPU telemetry can make cause and effect look reversed.

Eight causes that storage hardware alone cannot fix

1. Slow preprocessing

Image decode, tokenisation, augmentation and decompression consume CPU and memory bandwidth. A loader that cannot prepare batches fast enough leaves the GPU waiting even when the files arrived promptly. Increase workers only after checking core count, NUMA placement and memory pressure; too many workers can create contention.

Precompute expensive transformations where repeatability and capacity allow. Otherwise, size CPU resources alongside GPU count.

2. Millions of tiny files

A directory full of small samples turns the job into metadata and latency work. Packing samples into larger shards can reduce opens and improve read-ahead. The correct shard size depends on randomisation, retry behaviour and how much data a failed job must reread.

Do not change format purely for a benchmark. Confirm that the loader can preserve the required sample distribution and recover partial work.

3. A cold data tier

Object-tiered or hierarchical filesystems may need to promote data before the first read. If a reservation begins before staging finishes, premium GPUs wait for cheaper media. Connect the scheduler to data readiness, pre-fetch the working set and expose promotion progress to users.

Cold-start time belongs in the service target. Hiding it from the storage benchmark does not remove it from the training run.

4. Client path imbalance

One NIC, route, mount option or PCIe root complex can limit a server while its neighbours run well. Synchronous distributed training feels the slowest rank. Compare per-client throughput and latency; an aggregate graph can conceal the outlier.

For GDS, inspect whether traffic uses the intended direct path or compatibility mode. NVIDIA's guidance makes GPU/NIC affinity and PCIe configuration part of the result.

5. Shared-fabric congestion

Compute collectives, storage reads and checkpoints may share switches or links. A fabric that passes storage tests in isolation can still queue badly during all-reduce. Model compute and storage as separate network roles even when they use common physical equipment, then test concurrent traffic.

6. Cache policy working against the job

Repeated epochs may benefit from client, local NVMe or filesystem cache. Another workload can evict that data just before use. Conversely, a warm-cache benchmark can claim storage performance without touching the backend.

Run cold and warm tests, document the cache size and decide who controls eviction.

7. Protection and recovery traffic

Replication, erasure coding, snapshots, tiering and rebuilds consume resources. A system tuned only for healthy foreground reads may fall behind after a drive loss. Reserve network and media headroom, then test degraded operation.

8. Checkpoint implementation

Frameworks can write one file per rank, aggregate state, write asynchronously or distribute checkpoints over several objects. The storage platform sees very different requests. Update the checkpoint library or format when it reduces synchronisation and metadata cost; buying more flash is sometimes the slower fix.

Set service targets instead of "fast storage"

Useful targets describe the application outcome.

| Training phase | Better target | Weak substitute | | --- | --- | --- | | Dataset read | Epoch or step time with a defined client count and cache state | Maximum array read bandwidth | | Checkpoint | Committed checkpoint within a stated pause | Drive write total | | Restart | Model state loaded and training resumed within a deadline | Warm-cache file read | | Metadata | Job setup and sample enumeration at production file count | Single-client ls time | | Failure | Job impact during a named drive, node or path loss | Healthy-state availability claim | | Multi-tenancy | Priority job meets target beside background work | One isolated benchmark |

Add a percentile, not only an average. Training stalls often come from the long tail.

Calculate bandwidth from the workload

For steady reads, start with measured bytes consumed per client and the number of active clients:

application read target = sustained bytes per client x concurrent clients

For checkpoints:

application write target = checkpoint bytes / allowed completion time

These are application-level values. The storage system needs additional headroom for protection, protocol overhead, metadata, concurrent jobs and failure recovery. The factor cannot be copied from a generic table because replication, erasure coding, compression and network topology change it.

Record actual delivered bytes at the client and committed data at the storage tier. A gap between the two helps identify copies, caching or write-back behaviour.

Choose the right storage layers

Local NVMe cache

Local NVMe is often the cheapest way to remove repeated reads from the shared fabric. It suits disposable copies, prepared shards, container layers and per-node scratch. A cache needs admission, eviction and refill rules. Without them, users fill drives manually and jobs become tied to particular nodes.

Parallel shared filesystem

Lustre, DDN EXAScaler, WEKA and other parallel platforms can serve many clients through one namespace. They suit active datasets and checkpoints that must be visible across nodes. Metadata architecture, client software and failure handling matter as much as media speed.

DAOS

DAOS can expose native object operations, HPC middleware interfaces and POSIX access through dfuse. It is worth testing when applications can use its object model or when an NVMe and low-latency-fabric design fits the research environment. Check the supported server platform and exact application path.

Ceph

Ceph can hold object, block and file data in one distributed system. It is a sensible durable layer for datasets, model artefacts and private-cloud services. CephFS may also serve training, but recovery interference and metadata behaviour must meet the hot-tier target.

Object storage

Object storage works well as the authoritative repository for large immutable datasets and model versions. It may feed applications directly when the framework uses sharded objects efficiently, or it may stage material into the active filesystem. The right answer depends on API behaviour and first-read latency.

How the named platforms fit

Community Lustre gives an experienced team direct control over a parallel POSIX service. DDN EXAScaler packages Lustre with commercial hardware, tooling and support. WEKA provides a commercial NVMe-led namespace with GDS and object-tiering options. PEAK:AIO uses standard NFS, RDMA, NVMe-oF, GDS and pNFS approaches, with a simpler starting footprint in its published designs.

DAOS and Ceph address broader data models. DAOS rewards native or middleware integration; Ceph can consolidate durable object, block and file services. Neither should be forced into a Lustre-shaped evaluation.

Read the parallel filesystem comparison for AI GPU clusters before creating a vendor shortlist.

GPUDirect Storage: useful, conditional and measurable

GDS lets supported software move data between storage and GPU memory without the usual CPU bounce buffer. NVIDIA positions it for cases where storage-to-GPU transfer limits the pipeline, particularly coarse-grained streaming I/O.

The words "GDS compatible" do not establish that a job uses the direct path. Validate the filesystem and mount, driver stack, cuFile configuration, alignment, NIC and PCIe topology. Watch for fallback. Use NVIDIA's gdsio and counters as a path test, then measure the real application.

GDS does not remove object promotion, metadata lookup, decompression, tokenisation or checkpoint synchronisation. It fixes a particular data movement cost.

Benchmark without fooling yourself

Reproduce the data format

Use the real number and size distribution of files or objects. If data is sharded, use the actual shard range. Synthetic one-megabyte reads against a large test file cannot represent a tiny-file corpus.

Match concurrency

Run the intended client count or a justified scale model. Confirm that each client receives a fair share and identify the slowest. Increase load until a specific layer saturates.

Control cache state

Label results cold, warm or mixed. Flush or bypass caches only when that reflects the question being tested. Preserve the cache configuration with the result.

Add competing work

Run ingest, checkpoint, backup or tiering traffic beside training. Production is rarely one clean stream.

Break something

Remove a storage path, fail a node or trigger a rebuild. Record job behaviour and recovery time. This is where a reliable storage design separates itself from a fast demonstration.

MLPerf Storage can help compare training-data supply through published workload definitions. gdsio, fio, metadata tools and filesystem-specific benchmarks answer narrower questions. Application traces decide whether the result matters.

Operational controls that preserve GPU time

Connect storage telemetry to the scheduler and GPU monitoring. Alert on client latency, metadata queueing, failed paths, tier promotion and usable-capacity thresholds before users report slow jobs. Track checkpoint duration as an application metric.

Quotas and project boundaries prevent one experiment from consuming the service. Rate limits or quality-of-service controls can protect priority training from bulk ingest. Maintenance needs a drain procedure so that storage failover tests do not surprise a multi-day job.

Keep a known-good client image. Filesystem modules, kernel versions, RDMA packages and cuFile components can drift across hundreds of nodes; one inconsistent client is enough to create the slow rank.

Who should buy a commercial platform?

A commercial system makes sense when the value of GPU uptime and staff time exceeds the licence premium, when the team wants a qualified hardware/software matrix, or when a single escalation path matters. WEKA, DDN and PEAK:AIO offer different answers, so support does not remove the need for workload testing.

Community Lustre, DAOS or Ceph can be the better choice for organisations with strong distributed-storage engineers, existing operational tooling and a reason to control hardware or software lifecycle. Price that team honestly.

Who should not build a large shared tier?

Small inference or development environments may load models from object storage into local NVMe and avoid a parallel filesystem entirely. Independent workstation jobs may need ordinary project shares. A large shared tier adds cost and an operating surface that must earn its place.

Hosted options can also remove facility and on-call work. Compare GPU Cloud and Buy & Host when power, cooling or data-centre staffing is the limiting resource.

Buying through GPUMachines

GPUMachines can review the entire path: storage-node CPUs, memory, NVMe and HDD layout, client NIC placement, switch ports, optics, rack power, cooling and management separation. It can source hardware for community Lustre, DAOS and Ceph builds and scope commercial WEKA, DDN or PEAK:AIO routes, subject to project and regional availability.

The acceptance plan should sit beside the bill of materials. A quote without a test for training reads, checkpoints, failure and restart is only a parts list.

FAQ

How can I tell whether storage is starving the GPUs?

Correlate GPU activity with data-loader queues, CPU work, client latency, network counters and storage-server load. If storage is busy and client wait rises with concurrency, investigate the data path. If storage is idle, look at preprocessing and framework queues first.

Is more sequential bandwidth the usual fix?

No. Small files, metadata, one slow client, cold-tier promotion and checkpoint format can dominate. Add bandwidth only after identifying a saturated throughput path.

Should every training node cache data locally?

Local caching helps repeated reads when the scheduler can manage placement and refill. It is less useful for constantly changing datasets or jobs that land on arbitrary nodes without enough staging time.

Does GPUDirect Storage guarantee high GPU utilisation?

No. GDS can improve a supported storage-to-GPU transfer path. It does not fix preprocessing, metadata, object staging, synchronisation or an undersized network.

Which filesystem is most reliable?

Reliability depends on the proposed hardware, protection, failure domains, software version, support and operating discipline. Test a named failure while the target workload runs; brand-level claims are not enough.

How much headroom should storage have?

Enough to meet the foreground service target during the failures and background work the organisation accepts. Derive it through degraded-mode tests rather than applying one universal percentage.

Can GPUMachines benchmark the proposed design?

GPUMachines can define workload and infrastructure acceptance criteria and coordinate configuration review. Any claim of measured performance should come from an agreed test on the proposed or representative system.

Verdict

Preventing GPU starvation starts with finding the wait. Storage may be the culprit, but "faster storage" is not a diagnosis. Measure the training path, change the data format or loader where needed, and size shared storage from read demand, checkpoint deadlines and failure behaviour.

The best platform is the one that keeps the application queue full without creating an operating burden the team cannot carry. Prove that under load and while something is broken.

Design high-performance storage that keeps GPU training jobs supplied with data.

Sources and further reading

Vendor performance and utilisation figures need reproduction with the intended framework, clients, fabric, protection policy and failure state.

Best Storage for AI Training: How to Prevent GPU Starvation