Title: Multi-level parallelization of computational fluid dynamics codes using CUDA, MPI and OpenMP
Abstract: Efficient use of current HPC platforms, with very large numbers of CPU and GPUs,
provides a challenge not only in computational fluid dynamics (CFD) but in many applications.
I will discuss methods to augment an existing hybrid MPI-OpenMP CFD scheme with a CUDA-based fine
grain parallelization. This allows for simultaneous computation in CPUs and GPUs, using threads
to balance workload between CPU cores, and performing operations in GPUs when convenient. The
basics of the multi-level parallelization scheme will be discussed, with heuristics that show the
benefit of the CUDA implementation. CUDA streams are implemented to overlap data transfer with
computation. Testing with a baseline solver demonstrates significant aggregate speed-up over the
hybrid MPI-OpenMP solver by offloading to GPUs on an NVLink-based system. While the batch streamed
approach provides little benefit with NVLink, we see a performance gain of 30% when tuned for the
optimal number of streams on PCI-based systems. The method achieves strong GPU scaling, and
provides an example of how to manage methods that require all-to-all communications using multiple
GPUs. Finally, I will briefly discuss how the output of large simulations can be visualized and
analyzed by combining HPC approaches together with interactive desktop tools.
Bio: I received my diploma in 1999 and my doctoral degree in 2003, both in physics and from the
University of Buenos Aires (UBA) in Argentina, under the supervision of Daniel Gomez. From 2004 to 2007
I was a postdoc and later a staff scientist at National Center for Atmospheric Research (NCAR), in
Boulder, CO, USA, working under the supervision of Annick Pouquet, David Montgomery, and Darryl Holm.
I continued working for NCAR as a part-time scientist from 2007 to 2012. Since 2007 I am a professor
at the Physics Department at UBA, where I was also the head of the department from 2011 to 2015. From
2017 to 2020 I was the Project Manager for all grants in physics, mathematics and astronomy of the
National Fund for Scientific and Technological Research in Argentina. I received the Houssay prize
(Argentina) in 2010, and the ICTP prize (UNESCO/Italy) in 2012. I work on the numerical and theoretical
study of turbulent flows, with applications in astrophysics, geophysics, and atmospheric sciences. In
the field of fluid dynamics, my expertise includes parallelization methods for computational fluid
dynamics, the application of statistical methods for the characterization and analysis of turbulent
flows, spectral analysis of multi-scale and multi-physics phenomena, and sub-grid modeling for turbulent
flows. Applications include the solar cycle and turbulent dynamos, magnetic reconnection, rotating and
stratified turbulence, and superfluid turbulence.