Supporting heterogeneous architectures in ASiMoV-CCS

Project description

ASiMoV-CCS is a new Computation Fluid Dynamics (CFD) and Combustion modelling code that is being developed at EPCC as part of the ASiMoV Strategic Prosperity Partnership project with Rolls-Royce . The aim of the ASiMoV project is to achieve the world's first high fidelity simulation of a complete gas-turbine engine during operation, simultaneously including the effects of thermo-mechanics, electro-magnetics and CFD. The new ASiMoV-CCS code, which focusses on modelling the combustor part of the engine, is designed to be flexible, efficient and scalable from the start in order to meet the requirements of the ASiMoV project.

ASiMoV-CCS is written in modern Fortran (using features up to Fortran 2018 that include polymorphism and object orientation) and uses a "separation of concerns" design approach that separates the implementation of the parallelism away from the implementation of the scientific functionality. In addition, ASiMoV-CCS abstracts interfaces away from implementations; this makes the code highly customisable and extensible, allowing new implementations of features to be plugged in without requiring changes to the overarching code. The decision to develop ASiMoV-CCS in Fortran is a deliberate one: is it a language that still dominates in many scientific computing domains that rely on high-performance software (including engineering, aerospace, and weather forecasting), because it is fast and efficient, and lends itself extremely well to expressing mathematical concepts in code. Fortran applications represent a large share of all applications run on the world's largest supercomputers every day, and with more and more of these systems moving to heterogeneous architectures, ensuring that Fortran codes can exploit these architectures optimally is an important and timely research area.

Overview of research area:

The main platform that is used for simulations as part of the ASiMoV project is ARCHER2, the UK's national HPC service. ARCHER2 is a CPU-only platform, and therefore the development effort for ASiMoV-CCS focuses on CPU-only code first and foremost. However, it is clear that future large-scale platforms in the UK will be heterogeneous (according to the Top500 list from June 2021, 6 of the top 10 most powerful supercomputers use GPU acceleration) and it is therefore important that ASiMoV-CCS supports heterogenous platforms. In this project, the student will investigate how to make ASiMoV-CCS ready for the heterogeneous supercomputing era. Many new programming models for heterogeneous architectures that have recently emerged focus on C++ code to start, however the importance of Fortran in the high-performance computing world means that more and more support for Fortran programs is being developed. For instance, Nvidia provide CUDA Fortran to build GPU-accelerated Fortran applications for Nvidia GPUs, and similarly AMD's offering HIPfort provides interfaces to core HIP and ROCm libraries to target AMD GPUs. OpenMP and OpenACC are platform-agnostic solutions that have well established support for Fortran.

ASiMoV-CCS has many different components (such as CFD, combustion, particle tracking, spray modelling) which all have different performance characteristics and different demands on computing resources. Not all the components will for instance lend themselves equally well to GPU acceleration. In addition, the student will investigate what programming models are best suited to make ASiMoV-CCS ready for heterogeneous models to give the best balance between performance, portability and usability. Because ASiMoV-CCS is designed in a modular fashion, the student will be able to implement different code components using a range of models and evaluate the performance on large-scale test cases. 

Potential research question(s)

• What programming models are best suited to make a modern Fortran code ready for large-scale heterogeneous platforms? OpenMP5.0 seems the obvious choice, but is this true for this particular code?
• Which components of the ASiMoV-CCS code will benefit from acceleration, and how can the computation of the different components be scheduled to make best use of a heterogenous system (i.e. is there parallelism between the components that can be exploited)?
• Does the fact the ASiMoV-CCS is written in Fortran pose any limitations when making the code ready for heterogeneous platforms? If so, what how can these limitations be overcome?

Student Requirements:

Note that these are the minimum requirements to be considered for admission.

A UK 2:1 honours degree, or its international equivalent, in a relevant subject such as computer science and informatics, physics, mathematics, engineering, biology, chemistry and geosciences.

You must be an experienced programmer in at least one of Fortran, C or C++ and should be familiar with numerical software and mathematical concepts such as algebra, linear algebra and probability and statistics.

English Language requirements as set by University of Edinburgh

Student Recommended/Desirable Skills and Experience

• Experience with modern Fortran
• Experience with GPU programming
• Experience using HPC systems
• Understanding of performance analysis and benchmarking 

References

[1] https://gow.epsrc.ukri.org/NGBOViewGrant.aspx?GrantRef=EP/S005072/1

[2] A. Hsu, D. N. Asanza, J. A. Schoonover, Z. Jibben, N. N. Carlson and R. Robey, "Performance Portability Challenges for Fortran Applications," 2018 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), 2018, pp. 47-58, doi: 10.1109/P3HPC.2018.00008.

[3] Müller M., Aoki T. (2018) Hybrid Fortran: High Productivity GPU Porting Framework Applied to Japanese Weather Prediction Model. In: Chandrasekaran S., Juckeland G. (eds) Accelerator Programming Using Directives. WACCPD 2017. Lecture Notes in Computer Science, vol 10732. Springer, Cham. doi.org/10.1007/978-3-319-74896-2_2

[4] Reuben D. Budiardja, Christian Y. Cardall, "Targeting GPUs with OpenMP directives on Summit: A simple and effective Fortran experience", Parallel Computing, Volume 88, 2019,

102544, ISSN 0167-8191, doi.org/10.1016/j.parco.2019.102544.

[5] L. Luo et al., "Pre-exascale accelerated application development: The ORNL Summit experience," in IBM Journal of Research and Development, vol. 64, no. 3/4, pp. 11:1-11:21, 1 May-July 2020, doi: 10.1147/JRD.2020.2965881.

[6] Max P. Katz, Ann Almgren, Maria Barrios Sazo, Kiran Eiden, Kevin Gott, Alice Harpole, Jean M. Sexton, Don E. Willcox, Weiqun Zhang, and Michael Zingale. 2020. Preparing nuclear astrophysics for exascale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(SC '20). IEEE Press, Article 91, 1–12.

[7] Eggels R.L.G.M. (2018) The Application of Combustion LES Within Industry. In: Grigoriadis D., Geurts B., Kuerten H., Fröhlich J., Armenio V. (eds) Direct and Large-Eddy Simulation X. ERCOFTAC Series, vol 24. Springer, Cham. https://doi.org/10.1007/978-3-319-63212-4_1