How Empirical Research Supports Tool Development: A Retrospective Analysis and new Horizons

Empirical research provides two-fold support to the development of approaches and tools aimed at supporting software engineers. On the one hand, empirical studies help to understand a phenomenon or context of interest. On the other hand, studies compare approaches and evaluate how software engineers could benefit from them. Over the past decades, there has been a tangible evolution in how empirical evaluation is conducted in software engineering. This is due to multiple reasons. First, the research community has matured a lot thanks also to guidelines developed by several researchers. Second, the large availability of data and artifacts, mainly from the open-source, has made it possible to conduct larger evaluations, and in some cases to reach study participants. This keynote will first overview how empirical research has been used over the past decades to evaluate tools, and how this is changing over the years. Then, we will focus on the importance of combining quantitative and qualitative evaluations, and how sometimes "depth" turns out to be more useful than just "breadth". We will also emphasize how research is not a straightforward path, and negative results are often an essential component for future advances. Last, but not least, we will discuss how the role of empirical evaluation is changing with the pervasiveness of artificial intelligence methods in software engineering research.