Improving data flows for weather forecasting
24 July 2024
EPCC PhD student Nicolau Manubens, in collaboration with supervisor Adrian Jackson and the European Centre for Medium-Range Weather Forecasts (ECMWF), presented work on utilising object storage technology for weather and climate simulation applications at the PASC24 conference in Zurich, Switzerland, last month.
This novel research is moving large-scale data storage approaches for production weather forecasting from filesystems to newer software approaches to data storage, such as object stores. We have achieved a close to 2x reduction in data movement costs for ECMWF's production data storage approaches by using DAOS, a high performance object store.
PASC24
The PASC Conference series is an international and interdisciplinary platform for the exchange of knowledge in scientific computing and computational science with a strong focus on methods, tools, algorithms, workflows, application challenges, and novel techniques in the context of scientific usage of high performance computing.
Speaking about the experience of presenting at the event, Nicolau said: "PASC24 was a great conference with lots of insight on the challenges being faced by various types of HPC applications and state-of-the-art approaches to achieve better and faster results, both from scientific and technical standpoints. Machine learning and GPU techniques were the overarching theme, and weather and climate prediction applications had a strong presence. PASC24 was an excellent opportunity to get an updated overview on these matters as well as to present and discuss the object storage approach for HPC applications."
Further information
Paper: Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS. https://www.research.ed.ac.uk/en/publications/reducing-the-impact-of-io-contention-in-numerical-weather-predict
PASC24 website: https://pasc24.pasc-conference.org
