PerfExpert: An Automated HPC Performance Measurement and Analysis Tool with Optimization Recommendations

HPC systems are notorious for operating at a small fraction of their peak performance, and the ongoing migration to multi-core and multi-socket compute nodes further increases the already high complexity of performance optimization. The readily available performance evaluation tools require considerable effort to learn and utilize. Hence, most HPC application writers do not use them. As remedy, we have developed PerfExpert, a tool that combines a simple user interface with a sophisticated engine to automatically detect probable core, socket, and node-level performance bottlenecks in each important procedure and loop. For each bottleneck, PerfExpert provides a concise performance assessment and suggests steps that can be taken by the application developer to improve performance. These steps include optimization strategies, code examples, and compiler switches. We have applied PerfExpert to several HPC production codes on Ranger. In all cases, it automatically identified the critical code sections and provided accurate assessments of their performance.