PhD research: enhancing Sparse Linear Algebra with AI and HPC

^{Figure 1: The optimal sparse matrix storage format distribution among six different formats (COO, CSR, DIA, ELL, HYB and HDC) for 2100 real-life sparse matrices across various HPC systems architectures (Intel, AMD, ARM CPUs and NVIDIA/AMD GPUs).}

Sparse matrices, which contain a significant number of zero values, are ubiquitous in simulations ranging from material sciences to artificial intelligence. However, their effective manipulation requires complex data structures that align well with the hardware architecture being used. My PhD research focuses on automating the selection of the optimal sparse matrix storage format for different hardware architectures using artificial intelligence (AI). It is worth noting that today more than 70 sparse matrix storage formats are available. This not only speeds up computations but also simplifies the development process for users who might not be specialists in high-performance computing (HPC).

Morpheus ecosystem

The Morpheus library-based ecosystem, developed during my PhD at EPCC, is central to my work. It abstracts sparse matrix formats into a unified interface, allowing for seamless transitions between different storage options at runtime. This dynamic flexibility is crucial in a field where a single format rarely suffices due to the diversity of computational environments and hardware resources on modern supercomputers. One of the standout contributions of my research is the development of an AI-driven auto-tuner within Morpheus that intelligently selects the most appropriate format for any given matrix and computation scenario. This tool leverages machine learning to analyse the characteristics of the data and predict the optimal format, significantly enhancing performance without the user needing to manually intervene, across multiple hardware environments.

Why is this research important? For starters, it has profound implications for computational efficiency and resource utilisation in HPC systems, which are increasingly governed by energy constraints and the need to maximise throughput. My work ensures that researchers and engineers can focus more on their scientific inquiries rather than the intricacies of the computational tools they employ. Moreover, the performance portability aspect of my research— ensured through frameworks like Kokkos — means that the optimisations and tools developed are not just confined to a single type of hardware but can be deployed across the diverse landscape of modern computing resources. This universality is vital for the sustainability and future-proofing of scientific software.

At EPCC, we are uniquely positioned to drive forward these innovations. The centre provides access to state-of-the-art computing facilities and a collaborative environment that brings together experts from various disciplines, a synergy integral to tackling complex problems like optimising sparse matrix operations.

Research impact

Looking forward, the impact of my research is twofold. Firstly, it contributes to the scientific community by providing robust tools that enhance the efficiency of computational research. Secondly, it lays the groundwork for future explorations into how AI can be further integrated into HPC to tackle emerging challenges in data-intensive domains. As I continue to refine the auto-tuning mechanisms and explore new frontiers in performance portability, the support from EPCC is invaluable. Together, we are setting the stage for the next generation of HPC applications, making them more adaptable, efficient, and accessible to a broader range of users.

In conclusion, my work at EPCC involves harnessing the power of AI to accelerate Sparse Linear Algebra operations, which are critical for a wide range of scientific computations. By automating the selection of optimal data storage formats, we enhance the efficiency and speed of these operations, directly benefiting users by reducing the complexity and improving the performance of their computational tasks. This research not only advances our understanding of matrix operations in high-performance environments but also simplifies the process for researchers and engineers who rely on these tools for their innovative work. Through continued development and refinement, this project aims to make high-level computational tools more accessible and effective, paving the way for more advanced research and applications in various scientific fields.

Presentations and publications

For anyone interested in diving deeper into the specifics of my work and its broader implications, I encourage you to explore two of my recent publications: 

1. “Exploiting Dynamic Sparse Matrices for Performance Portable Linear Algebra Operations”, presented at the 2022 International Workshop on Performance, Portability & Productivity in HPC (P3HPC) in Dallas, Texas (publication, presentation).
2. "Optimizing Sparse Linear Algebra Through Automatic Format Selection and Machine Learning", presented at The Eighteenth International Workshop on Automatic Performance Tuning (iWAPT) in St. Petersburg, Florida, in 2023 (publication, presentation).

These publications provide a comprehensive overview of the methods and technologies we’ve developed, showcasing the practical applications and the theoretical underpinnings of our approach to enhancing high-performance computing with AI, specifically in the area of Sparse Linear Algebra.

If you are interested in staying up-to-date with my research developments you can follow my work from:

• Personal website: https://cstyl.github.io
• LinkedIn: https://www.linkedin.com/in/cstyl/

Acknowledgments

My work is part of the EPSRC project ASiMoV (EP/S005072/1). Throughout my research I used the ARCHER2 UK National Supercomputing Service and the Cirrus UK National Tier-2 HPC Service at EPCC, funded by the University of Edinburgh and EPSRC (EP/P020267/1), and Isambard 2 UK National Tier-2 HPC Service operated by GW4 and the UK Met Office, and funded by EPSRC (EP/T022078/1).

Author

Christodoulos Stylianou

PhD candidate and Research Engineer at the Computation-based Science and Technology Research Center (CaSToRC) of The Cyprus Institute (CyI).

c.stylianou@cyi.ac.cy