ExaHyPE-DSL project: integrating the ExaHyPE engine with the LLVM compiler ecosystem

The ARCHER2 Embedded CSE (eCSE) ExaHyPE-DSL project, led by Professor Tobias Weinzierl (Durham University) and Dr Nick Brown (EPCC), started in April this year. This collaboration intends to build a new software stack, including a new stencil-like [1] domain specific language (DSL), that integrates the existing ExaHyPE engine [2] with the LLVM compiler ecosystem [3]. 

This will be achieved by leveraging the xDSL framework [4] (developed by the ExCALIBUR xDSL project) to enable developers to express the ExaHyPE C++ compute kernels in the new DSL within Python. The use of the MLIR [5] / LLVM ecosystem will enable much wider community access to the capabilities of ExaHyPE and allay the fears of ‘DSL lock-in’ [6] that a proprietary DSL compiler framework would create.

^{Figure 1: Simulation of seismic waves around Mount Zugspitze with curvilinear meshes. The internal Cartesian mesh (left) is used to simulate a complex topography (right). [2]}

ExaHype engine

Professor Weinzierl and his team at Durham University are the creators and developers of the ExaHype engine and the underlying Peano framework [7]. The latter provides solvers on dynamically adaptive Cartesian meshes ‎(Figure 1), with support for Finite Volume, Runge-Kutta Discontinuous Galerkin and ADER-DG discretisations. EPCC is contributing the extensive xDSL development expertise gained from our current involvement with the development of xDSL as part of the ExCALIBUR project. 

The use of MLIR by xDSL enables ExaHyPE-DSL to leverage existing support for GPUs and distributed memory parallelism (DMP) [8]. For example, ‎Figure 2 below shows how we can create an xDSL-based pipeline that progressively lowers MLIR dialects to generate MPI calls, for distributed memory parallelism, from high-level stencil definitions generated by ExaHyPE-DSL. The resulting code is finally lowered to LLVM-IR to generate the executable binary. 

The ability to apply arbitrary transformations on the generated code provides exciting opportunities for novel optimisations of the ExaHyPE-DSL code. Indeed, the project will implement a machine learning (ML) component to automatically tune/select kernels and set their hyper-parameters at runtime. 

ExaHyPE-DSL demonstrates the synergies between projects in the ExCALIBUR programme and the catalyst that ARCHER2 eCSE funding provides to realise such collaborations. 

^{Figure 2: Progressive ‘lowering’ of MLIR dialects to generate MPI calls [8].}

Further information

The ExaHyPE-DSL project is funded by the ARCHER2 Embedded CSE programme.

The xDSL work is supported by the Engineering and Physical Sciences Research Council (EPSRC) grant EP/W007940/1.

[1] N. Brown, B. Echols, J. Zarins, and T. Grosser, ‘Exploring the Suitability of the Cerebras Wafer Scale Engine for Stencil-Based Computation Codes’, in Euro-Par 2022: Parallel Processing Workshops, J. Singer, Y. Elkhatib, D. Blanco Heras, P. Diehl, N. Brown, and A. Ilic, Eds., Cham: Springer Nature Switzerland, 2023, pp. 51–65. doi: 10.1007/978-3-031-31209-0_4.

[2] A. Reinarz et al., ‘ExaHyPE: An engine for parallel dynamically adaptive simulations of wave problems’, Comput. Phys. Commun., vol. 254, p. 107251, Sep. 2020, doi: 10.1016/j.cpc.2020.107251.

[3] ‘The LLVM Compiler Infrastructure Project’. Accessed: Nov. 13, 2020. [Online]. Available: https://llvm.org/

[4] xDSL project, ‘xDSL’, xDSL. Accessed: Apr. 02, 2024. [Online]. Available: https://xdsl.dev/

[5] C. Lattner et al., ‘MLIR: Scaling Compiler Infrastructure for Domain Specific Computation’, in 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), Feb. 2021, pp. 2–14. doi: 10.1109/CGO51591.2021.9370308.

[6] F. Tomassetti and V. Zaytsev, ‘Reflections on the Lack of Adoption of Domain Specific Languages’, in STAF 2020 Workshop Proceedings: 4th Workshop on Model-Driven Engineering for the Internet-of-Things, 1st International Workshop on Modeling Smart Cities, and 5th International Workshop on Open and Original Problems in Software Language Engineering co-located with Software Technologies: Applications and Foundations federation of conferences (STAF 2020, ) Bergen, Norway, June 22-26, 2020, CEUR, Oct. 2020, pp. 85–94. Accessed: Jun. 28, 2024. [Online]. Available: https://research.utwente.nl/en/publications/reflections-on-the-lack-of-adoption-of-domain-specific-languages

[7] T. Weinzierl, ‘Peano 4’, Tobias Weinzierl. Accessed: Jun. 28, 2024. [Online]. Available: https://tobiasweinzierl.webspace.durham.ac.uk/software/peano/

[8] G. Bisbas et al., ‘A shared compilation stack for distributed-memory parallelism in stencil DSLs’, in 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Association for Computing Machinery, Apr. 2024, pp. 38–56. doi: 10.1145/3620666.3651344.