Research on Parallel Task Optimization of High Performance Computing Cluster

The application of cluster system has gone deep into all aspects of production and life. The use of the cluster system's cost-effective parallel computing capabilities to solve complex model calculations and mass data processing issues has become an important branch of high-performance computing research. Spark-based parallel computing platforms can greatly improve computational efficiency. However, if the spark task does not perform reasonable parameter settings, it will still cause a huge waste of resources and resource consumption. In this paper, high-performance computing cluster is used as the experimental hardware environment, and based on the Spark platform, the parameters of the parallel task are adjusted, which greatly reduces the hardware resources and improves the utilization of the cluster. Through experimental comparison, it provides a direct reference for the calculation of high-performance computing clusters.