Scalable GPU Communication with Code Generation on Stencil Applications

Clusters with GPUs are mainstream in HPC as shown by the last edition of the Top500 list, increasing the demand for GPU capable scientific computing software. Programming large scale GPU systems in an efficient and future-proof way present numerous challenges, such as optimizations for a variety of GPUs and interconnect hardware, hiding communication overhead with computation and efficient domain partitioning. We present an improvement to the CUDA-based communication of stencil applications in the WALBERLA framework, achieving scalability while supporting different GPUs and communication infrastructures. We utilize the lattice Boltzmann Method for fluid flows as a representative of stencil-based scientific computing and implement a communication hiding strategy that is capable of adjusting to a system's computing and communication capabilities. We compare the use of CUDAMemCopy with the use of customized pack/unpack kernels and show that packing achieves almost linear weak scaling behavior in the Santos Dumont supercomputer with up to 128 GPUs. We also show that the proposed approach is not sensitive to the direction of the domain partitioning, one of the biggest challenges when communicating 3D domains in GPU-based stencil simulations.