Constructing a Supercomputing Framework by Using Python for Hybrid Parallelism and GPU Cluster

This paper reports the construction of a new software framework, SOLVCON, for high-fidelity solutions of linear and nonlinear hyperbolic partial differential equations (PDEs). Applications of SOLVCON include gas dynamics, aero-acoustics, stress waves in solids, electromagnetic waves, etc. The default numerical scheme employed in SOLVCON is the space-time Conservation Element and Solution Element (CESE) method. SOLVCON is organized based on the two-loop structure found in all finite-volume methods for time-marching solutions: (i) the outer temporal loop for time-marching, and (ii) the inner spatial loops for calculating flux conservation. Built as a software framework that enforces inversion of control, solver kernels in SOLVCON are completely segregated from supportive functionalities and are made pluggable to the framework. As such, SOLVCON is inherently a multi-physics software package. SOLVCON also aims at reducing turnaround time for high-resolution calculations by incorporating hybrid parallelism, i.e., simultaneously using shared- and distributed-memory parallel computing. The code for General-Purpose Graphic Processing Unit (GPGPU) and multi-threaded computing are modularized as optional parts of the solver kernels. Based on the two-loop algorithm, the software structure of SOLVCON has been organized into 5 layers of Python modules. To handle large data sets associated with high-fidelity solutions of PDEs, special modules were developed for parallel input and output (I/O) and in situ visualization. SOLVCON is developed by using Python programming language because it is efficient to be coded and maintained and its unique capability of interfacing with other programming languages for high-performance computing (HPC). By using SOLVCON for typical computational tasks, 99% of the execution time is used in spatial loops, which are implemented by using C with multi-threading computing, and/or CUDA for GPGPU computing. The use of Python does not impair the performance. To model complex geometry, SOLVCON uses unstructured mesh consisting of mixed elements, i.e., triangles and quadrilaterals for two-dimensional problems, and tetrahedra, prisms, and pyramids for three-dimensional problems. SOLVCON has been routinely used for computational tasks running on clusters composed of more than a thousand CPUs. For a supersonic cross flows passing a jet, a benchmarking CFD problem, with 1.3 billion degrees of freedom, SOLVCON delivers the full transient results, including visualized results, in 3 days. SOLVCON has been open-sourced and released (http://solvcon.net) under the term of GNU General Public License.