Exploring DAOS interfaces and performance

POSIX distributed file systems such as Lustre and IBM Storage Scale are currently the most widespread software approach to HPC storage. However, due to fundamental design aspects, POSIX file systems can manifest performance limitations at scale for certain I/O workloads.

In recent years there has been a proliferation of object storage solutions such as DAOS. These break away from the POSIX standards and implementations and offer new APIs and characteristics with the potential to address the limitations of POSIX I/O.

DAOS was designed from the ground up with the limitations of POSIX in mind and with the aim of providing the highest performance the underlying storage and network hardware allow. Prior work by the same authors [1], described in the paper "Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS", demonstrated DAOS' performance potential when used natively via its bespoke object storage API. 

However, porting existing POSIX I/O applications to this API can be a daunting task. To mitigate this, DAOS provides a libdfs library which maps the POSIX I/O APIs, such as directories and files, onto native DAOS objects, but this requires some modification of existing POSIX I/O applications to integrate them with libdfs before running on DAOS. A DAOS File System in User Space (DFUSE) module is also provided such that applications can mount a DAOS system in the directory tree and transparently use DAOS through the POSIX I/O infrastructure without modification.

This paper [2] deploys DAOS on Google Cloud Virtual Machines (VMs) with NVMe SSDs, and runs a range of I/O benchmarks to measure and compare its performance and scalability when accessed natively as well as via the libdfs and DFUSE interfaces. Some of the results, included here, show that DAOS can also provide excellent performance for simple POSIX I/O workloads using the libdfs and DFUSE interfaces.

^{Figure 1. Write scalability (top) and read scalability (bottom) of the DAOS APIs and applications with no redundancies.}

The paper also tests DAOS with a benchmark which uses the HDF5 storage library to perform more complex I/O workloads. HDF5 provides backends for operation on POSIX file systems and DAOS, enabling the benchmark to run natively against both storage systems. The results show that, in this particular case, the benchmark runs that use HDF5's POSIX I/O backend on top of DFUSE reach better performance than equivalent runs using the native DAOS backend. The reasons why this happens are discussed in the paper.

Finally, Lustre and Ceph (a popular, open-source object storage solution) are deployed on the same hardware as DAOS, and the performance delivered by the three storage systems is tested using fdb-hamer, a Numerical Weather Prediction I/O benchmark. fdb-hammer stores and indexes many small meteorological data objects, and has backends for native operation on the three systems. 

The comparative results, included here, show that DAOS achieves significantly better performance overall than the other two. Lustre struggles when reading across many files, and Ceph, although it provides reasonable performance, reaches lower bandwidths than the two other both for write and read.

^{Figure 2. Results for fdb-hammer runs on 32 client nodes against deployments of DAOS on 16 nodes, Ceph on 16+1 nodes, and Lustre on 16+1 nodes.}

This work demonstrated that DAOS is effectively designed for high performance, and can enable new I/O patterns at scale that were not possible with POSIX file systems. 

This paper resulted from a collaboration between the EPCC at the University of Edinburgh, the European Centre for Medium-Range Weather Forecasts, and Google Cloud.

[1] Manubens Gil, N, Smart, S, Danovaro, E, Quintino, T & Jackson, WA 2024, 'Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS', Paper presented at The Platform for Advanced Scientific Computing (PASC) Conference, Zurich, Switzerland, 3/06/24 - 5/06/24 pp. 1-12. https://doi.org/10.1145/3659914.3659926

[2] Nicolau Manubens, Johann Lombardi, Simon D. Smart, Emanuele Danovaro, Tiago Quintino, Dean Hildebrand, and Adrian Jackson. 2025. Exploring DAOS Interfaces and Performance. In Proceedings of the SC '24 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W '24). IEEE Press, 1340–1348. https://dl.acm.org/doi/10.1109/SCW63240.2024.00175

Author

Nicolau Manubens Gil, EPCC PhD candidate and software engineer at the European Centre for Medium-Range Weather Forecasts (ECMWF).

Study at EPCC

Interested in studying with us? See our Education and Training webpages for full details.