Best Parallel File System for AI GPU Clusters

Lustre, DAOS and Ceph do not solve the same problem, and neither do WEKA, DDN or PEAK:AIO. The right shortlist starts with the data path and its owner.

Ask six storage suppliers for the "best parallel filesystem" and you will receive six different definitions of the problem. One will optimise a POSIX training namespace. Another will treat object data as the centre of the system. A third will sell operational certainty around an open-source filesystem; a fourth will begin with familiar NFS clients and a compact appliance. All may be credible, yet only one or two may match the way your jobs actually read, checkpoint and recover.

The names in this comparison do not occupy one technical category. Lustre is an open-source parallel filesystem. DAOS is an open-source distributed object store with native, middleware and POSIX access. Ceph combines object, block and file services. DDN EXAScaler is a supported commercial Lustre platform. WEKA supplies a commercial NVMe-led data platform with several protocols, while PEAK:AIO takes a standards-based NFS, RDMA, NVMe-oF and pNFS route.

That unevenness is useful. It forces the buyer to decide what the storage service must be before comparing quotations.

Executive answer

Choose community Lustre when parallel POSIX throughput is the main job and the organisation already knows how to operate Lustre. Choose DDN EXAScaler when the same workload needs a qualified platform and a commercial escalation path. WEKA belongs on the shortlist when an NVMe-led shared namespace, GPUDirect Storage, multiprotocol access and object tiering are worth the licence and host-resource requirements.

DAOS is compelling for teams prepared to use its native object model or validated middleware integrations; its POSIX layer needs application testing. Ceph is the broadest open platform of the group and often the strongest durable object, block and shared-file foundation, although it should not inherit the hottest training tier by default. PEAK:AIO offers a simpler standards-based route that can suit smaller or mid-sized GPU clusters, while large deployments need a vendor-backed pNFS scale design.

There is no universal winner. GPUMachines can build a filesystem shortlist from the workload, storage servers and fabric, then test the deciding paths rather than relying on product-category labels.

First, correct the taxonomy

Search results often put these products in one list because each can move data to GPU clients. Their data models and operating assumptions differ.

| Platform | Software model | Typical access paths | Commercial model | | --- | --- | --- | --- | | Lustre | Parallel filesystem | Native Lustre POSIX client | Open source; support available from third parties | | DAOS | Distributed object store | Native API, MPI-IO, HDF5, dfuse POSIX | Open source; support scope depends on provider | | Ceph | Distributed object storage base with file and block services | RGW/S3, RBD, CephFS | Open source; supported distributions available | | DDN EXAScaler | Engineered Lustre platform | Lustre POSIX client and DDN tooling | Commercial hardware/software/support | | WEKA | Distributed NVMe data platform | POSIX, NFS, SMB, S3, GDS | Commercial software and appliance routes | | PEAK:AIO | Software-defined AI storage using standard protocols | NFS, RDMA, NVMe-oF, GDS and pNFS options | Commercial software with partner hardware |

Calling all six "parallel filesystems" makes the article easier to title and harder to trust. The buyer should compare the access path presented to each application, the service behind it and the people required to keep it working.

What an AI filesystem must do

"AI workload" is too broad for procurement. Separate at least these behaviours.

Feed training workers

Large sequential shards can produce high aggregate reads without severe metadata pressure. Random samples spread across millions of small files produce the opposite shape. Two systems with similar large-file bandwidth may deliver very different epoch times.

Record the number of readers, file-size distribution, read-ahead behaviour, cache state and preprocessing cost. A training job can wait on CPU decoding or a Python data loader while the filesystem remains idle.

Complete checkpoints on time

Checkpoints are synchronisation events. The service target is not "high write bandwidth"; it is a committed and recoverable checkpoint inside a defined time. Writer count, file count, protection policy and flush semantics belong in the test.

The restart is equally important. A platform that writes quickly but reloads poorly during a cold restart still extends the recovery window.

Survive a shared research estate

Researchers create environments, small configuration files, logs and many model versions. Quotas, snapshots, permissions, namespace scans and accidental deletion matter beside throughput. A storage platform may excel at one large training job and frustrate a hundred users doing ordinary work.

Recover without taking the cluster with it

Drive and server failures trigger rebuild, recovery or failover activity. Test foreground I/O during those events. Healthy-state benchmark charts say nothing about whether a checkpoint still finishes while redundancy is restored.

Lustre: the direct open-source parallel filesystem answer

Lustre separates metadata from file data. Metadata Servers make Metadata Targets available, while Object Storage Servers expose Object Storage Targets. Clients can stripe files over multiple targets and move data directly to the relevant servers.

This architecture suits sustained parallel POSIX I/O. It also lets the design scale metadata and bulk data roles independently. For distributed training, large dataset shards and coordinated checkpoints can map well to Lustre when stripe settings, client behaviour and target layout are understood.

The difficulty sits in ownership. High availability, target placement, client compatibility, tuning, monitoring, upgrades and recovery procedures require storage engineers. An open-source licence does not include an on-call team. Organisations with an HPC practice may already have those skills; a new AI platform group often does not.

Lustre fits when

POSIX applications dominate and parallel throughput is the first requirement.
The team already runs Slurm, HPC clients and Lustre operational practices.
Hardware choice and software control matter more than an appliance experience.
The buyer can maintain a staging environment for upgrades and failure tests.

Lustre is a poor fit when

The storage team is one busy Linux administrator, the cluster has an immediate production deadline, or the business expects one supplier to own every performance and recovery issue. That buyer should price DDN EXAScaler or another supported implementation rather than pretending labour is free.

DDN EXAScaler: Lustre with packaged engineering

DDN EXAScaler keeps Lustre at the centre but changes the delivery model. DDN supplies hardware designs, software, management, tuning knowledge and support around the filesystem. Buyers still get a parallel POSIX service; they take on fewer unqualified combinations.

That distinction matters during upgrades and incidents. With community Lustre, the internal team reconciles firmware, clients, kernel versions, target state and the surrounding fabric. With EXAScaler, the support contract and qualified configuration narrow responsibility. Read the contract, because response terms and supported integrations carry as much value as the array.

DDN publishes ambitious throughput, checkpoint and GPU-utilisation statements. Treat each as a vendor claim. The proposed EXAScaler configuration still needs to meet the buyer's file distribution, protection settings and degraded-mode target.

EXAScaler tends to make sense for large training or HPC estates that want Lustre behaviour but cannot justify becoming their own storage vendor.

DAOS: a different data model with a POSIX option

DAOS was designed for distributed non-volatile memory and user-space I/O. Storage nodes run DAOS engines, data is divided into targets, and the service uses the OpenFabrics Interfaces for network communication. Applications can use native key-array-value objects, integrations such as MPI-IO and HDF5, or a POSIX interface through dfuse.

Version 2.6 supports two approaches for the byte-addressable metadata tier: persistent memory, or Metadata-on-SSD using DRAM with logging to NVMe. That matters because an older assumption that every new DAOS deployment must source Intel Optane is no longer a complete description.

DAOS can reward applications that use its model rather than forcing every operation through classic file semantics. It can also fit research environments that already use MPI and HDF5. The penalty is integration work. A framework expecting ordinary POSIX behaviour may run through dfuse, but that path should be tested for compatibility, caching and operational tooling.

Do not select DAOS because it sounds newer than Lustre. Select it when the application interfaces and engineering team can use what makes DAOS different.

Ceph: broad infrastructure, not a default scratch winner

Ceph's RADOS layer supports object, block and file services. RGW presents object APIs, RBD supplies block volumes, and CephFS adds a POSIX namespace with Metadata Servers while clients access RADOS for file data. This breadth can replace several separate infrastructure services.

For private clouds, Kubernetes estates, durable dataset repositories and project storage, Ceph may be the most useful platform in the comparison. It gives the organisation hardware freedom and a large open-source ecosystem. A team already operating Ceph also avoids introducing another control plane for every data service.

Ceph's flexibility does not guarantee the best hot training tier. Object Storage Daemons perform replication, recovery and rebalancing; those tasks use CPU, drives and network capacity. CephFS metadata behaviour, data placement and client performance need proof under the intended training and failure conditions.

A common design uses Ceph for durable object and shared data, then stages active training material to Lustre, DAOS, WEKA or DDN. That adds data movement, but it stops the capacity tier and checkpoint tier from compromising each other.

WEKA: commercial NVMe performance with a broad namespace

WEKA distributes functions across backend servers and uses dedicated processes for frontend access, filesystem compute, drives and management. Its performance-oriented network path uses DPDK, with RDMA and GPUDirect Storage available in supported configurations. WEKA documentation also describes POSIX, NFS, SMB and S3 access.

The object tier changes the capacity discussion. Hot file data can stay on SSD, colder content can move to object storage, and filesystem metadata remains on the SSD tier. Administrators can pre-fetch data before a job or release it after use. This suits estates that want a fast namespace without placing every retained byte on NVMe.

Nothing here is free of design work. WEKA reserves host cores and memory, relies on specific networking behaviour, and has compatibility conditions for GDS, encryption and NIC modes. Include those resources in the comparison. A licence quotation without client CPU, network and object-tier costs is incomplete.

WEKA is strongest when the buyer wants commercial ownership of an NVMe-led multiprotocol platform and values integrated data lifecycle features.

PEAK:AIO: standard protocols and a smaller starting point

PEAK:AIO's AI Data Server material focuses on software-defined storage over partner server hardware, using NFS, RDMA, NVMe-oF and GPUDirect Storage. Its recent Open pNFS material describes a route to distribute metadata and data services across nodes. The proposition differs from a traditional Lustre build: start with familiar client access and a compact footprint, then scale according to the validated design.

That can suit departmental clusters, inference farms, research groups and mid-sized training estates that do not want a large parallel-filesystem operations burden. It may also fit buyers who prefer kernel-standard access over a proprietary client.

For large GPU counts, ask harder questions. What client scale has the proposed pNFS version demonstrated? How does metadata fail over? What happens during a data-server loss? Which NICs, fabrics and kernels form the supported configuration? Vendor material from a single 2U appliance cannot answer a multi-rack design by multiplication.

PEAK:AIO belongs in the comparison, but its best-fit scale must come from a reference architecture and acceptance test.

Open source against commercial storage

The licence is the visible difference. Operational packaging is the expensive one.

An open-source build gives the buyer control over servers, drives, release timing and many tuning decisions. It also assigns that buyer responsibility for qualification, monitoring, upgrade rehearsals, security response and difficult incidents. Commercial platforms charge for software and support while reducing the combinations the internal team must own.

Neither model wins automatically. A university with a seasoned Lustre group can operate community software more confidently than a general enterprise can operate an unfamiliar appliance. Conversely, a small platform team may save far more in GPU time and staff attention by buying a supported system.

Calculate total cost over the intended life:

Storage servers, media, NICs, switches, optics, racks, power and cooling.
Software subscriptions, support levels and renewal assumptions.
Engineering time for integration, on-call work, upgrades and capacity planning.
Spare capacity and equipment reserved for failure recovery.
Data migration, staging and eventual platform exit.

The cheapest usable terabyte is not necessarily attached to the cheapest storage service.

GPUDirect Storage should not decide the tender alone

NVIDIA GPUDirect Storage can establish a direct DMA path between storage and GPU memory, avoiding a CPU bounce buffer. It can reduce CPU load and improve transfer behaviour where storage-to-GPU movement limits the application.

GDS depends on the application, supported filesystem path, NIC, GPU, driver and PCIe topology. NVIDIA's documentation calls out affinity, ACS, IOMMU and alignment. Compatibility mode can also fall back to a standard path. A proposal that lists "GDS support" without proving the active path is a checkbox, not evidence.

Test GDS using gdsio, inspect counters, and compare it with the application's normal I/O. Some pipelines spend more time decoding data or opening files than copying large aligned buffers. Fix the dominant wait.

How to run the comparison

1. Freeze the client contract

Record GPU server models, client count, operating system, kernel, NICs, ports, PCIe topology and scheduler. A 64-node eight-GPU cluster is not the same client problem as 128 four-GPU nodes.

2. Capture real workload traces

Measure reads, writes, file sizes, metadata operations, queue depth, checkpoint bursts and restart behaviour. Include cold and warm cache. Separate storage wait from preprocessing and framework wait.

3. Test healthy and degraded states

Fail a drive, storage server and network path. Run rebuild or recovery while training and checkpoint traffic continues. Record tail latency and job completion, not only average bandwidth.

4. Score the operating model

Name who patches clients, replaces media, manages quotas, rehearses recovery, handles vendor cases and plans capacity. If no name appears beside a task, the architecture is unfinished.

Decision matrix

| Buyer profile | First shortlist | Why | | --- | --- | --- | | HPC team wanting open parallel POSIX storage | Lustre | Direct fit and existing operating skills | | Large AI estate wanting supported Lustre | DDN EXAScaler | Packaged Lustre engineering and support | | Application team able to use native object or HPC middleware APIs | DAOS | Data model and user-space NVMe path may fit | | Private cloud needing object, block and shared files | Ceph | Broad open service portfolio | | Enterprise seeking NVMe, GDS, multiple protocols and object tiering | WEKA | Integrated commercial data platform | | Mid-sized team seeking standard NFS/RDMA access and a compact entry point | PEAK:AIO | Simpler client model, subject to scale proof |

The matrix identifies where to start. Real results may move a buyer to another row.

Alternatives and mixed designs

These six are not the entire market. BeeGFS, IBM Storage Scale, VAST Data and other products may deserve a place depending on the workload and procurement route. More importantly, a mixed architecture can beat a single-product mandate.

Use object storage for retained datasets, a fast filesystem for active training and local NVMe for disposable cache. The workflow promotes data before the allocation, writes checkpoints to the shared tier and moves completed artefacts to durable storage. The cost is orchestration and duplicate capacity; the benefit is paying flash prices only for hot material.

Related GPUMachines comparisons cover Ceph versus WEKA, Lustre versus WEKA, Ceph versus DDN EXAScaler and community Lustre versus DDN EXAScaler in more detail.

Who should not buy a large parallel filesystem?

A small inference service loading a few models may run better from local NVMe with a durable object source. A workstation team with independent jobs may need a normal shared project volume, not an HPC filesystem. And an organisation that cannot name a storage owner should not begin with community software simply to remove the licence line.

Hosted GPU infrastructure can also be the better choice when the on-premise site lacks power, cooling, fabric skills or a realistic data-ingest path. Compare GPU Cloud and Buy & Host with the complete on-premise service.

Buying through GPUMachines

GPUMachines can source the storage servers, enterprise NVMe, network adapters, switches and rack infrastructure for open-source Lustre, DAOS and Ceph designs. It can also scope commercial WEKA, DDN and PEAK:AIO routes with the corresponding hardware and deployment requirements, subject to supplier and regional availability.

The useful deliverable is not a brand recommendation. It is a testable configuration with client paths, storage roles, failure domains and acceptance criteria.

FAQ

Is Ceph a parallel filesystem?

CephFS is a distributed POSIX filesystem over RADOS, with clients accessing data objects directly. Its architecture and broad service role differ from Lustre, so buyers should not assume identical behaviour for parallel training.

Is DAOS a filesystem?

DAOS is a distributed object store. It offers native APIs and middleware integrations, plus POSIX access through dfuse. The interface used by the application changes the result and must be part of testing.

Is DDN EXAScaler proprietary Lustre?

EXAScaler is a commercial platform built around Lustre, adding DDN hardware, software, management and support. Compare it with community Lustre on total operating responsibility, not only filesystem ancestry.

Can WEKA replace object storage?

WEKA can present S3 and can tier file data to an object store, but the long-term capacity and protection architecture still needs design. Some estates use a separate object platform behind or beside WEKA.

Can PEAK:AIO serve a large distributed training cluster?

Potentially, through a validated multi-node and pNFS design. Do not extrapolate a single-node result. Require client-scale, metadata, failure and network evidence for the proposed deployment.

Which option is cheapest?

That depends on hardware, usable capacity, people, support, failure headroom and migration. Community software removes a proprietary licence but does not remove engineering or downtime cost.

Which benchmark should we use?

Use MLPerf Storage where its workloads map to the project, NVIDIA gdsio for GDS-path testing, and application-level tests for training, checkpoint and restart. No single benchmark covers the whole service.

Verdict

Lustre is the clearest open-source parallel filesystem in this group, and DDN EXAScaler is the clearest supported Lustre route. That does not make either the automatic winner. DAOS can offer more to applications that use its object model, Ceph can consolidate a broader data estate, WEKA packages an NVMe-led multiprotocol platform, and PEAK:AIO offers a standards-based commercial path with a smaller starting footprint.

Choose the service model first. Then make the vendors prove it with your files, clients, network and failure conditions.

Compare parallel filesystem and AI storage options with GPUMachines.

Sources and further reading

Vendor performance, scale and utilisation statements remain vendor claims until reproduced with the buyer's application, client, network and protection settings.

Best Parallel File System for AI GPU Clusters? Six Storage Approaches Compared