Support mechanisms to conduct empirical studies in software engineering: a systematic mapping study

Context: Empirical studies are gaining recognition in the Software Engineering (SE) research community, allowing improved quality of research and accelerating the adoption of new technologies in the software market. However, empirical studies in this area are still limited. In order to foster empirical research in SE, it is essential to understand the resources available to aid these studies. Goal: Identify support mechanisms (methodology, tool, guideline, process, etc.) used to conduct empirical studies in the Empirical Software Engineering (ESE) community. Method: We performed a systematic mapping study that included all full papers published at EASE, ESEM and ESEJ since their first editions. Were selected 891 studies between 1996 and 2013. Results: A total of 375 support mechanisms were identified. We provide the full list of mechanisms and the strategies that uses them. Despite this, we identified a high number of studies that do not cite any mechanism to support their empirical strategies: 433 studies (48%). Experiment is the strategy that has more resources to support their activities. And guideline was the most used type of mechanism. Moreover we observed that the most mechanisms used as reference to empirical studies are not specific to SE area. And some mechanisms were used only in specific activities of empirical research, such as statistical and qualitative data analysis. Experiment and case studies are the strategies most applied. Conclusions: The use of empirical methods in SE has increased over the years. Despite this, many studies did not apply these methods and do not cite any resource to guide their research. Therefore, the list of support mechanisms, where and how they were applied is a major asset to the SE community. Such asset can encourage empirical studies aiding the choice regarding which strategies and mechanisms to use in a research, as well as pointing out examples where they were used, mainly to novice researchers. We also identified new perspectives and gaps that foster other research for the improvement of empirical research in this area.