Assessing the State of Software in a Large Enterprise: A 12-Year Retrospective

To be relevant to the goals of an enterprise, an industrial software engineering research organization must identify problems of interest to, and find solutions that have an impact on, software development within the company. Using a systematic measurement program both to identify the problems and assess the impact of solutions is important to satisfying this need. Avaya has had such a program in place for about 12 years. Every year we produce an annual report known as the State of Software in Avaya that describes software development trends throughout the company and that contains prioritized recommendations for improving Avaya’s software development capabilities. The process of identifying trends and recommending actions for improvement starts with identifying the goals of the enterprise and uses the goal-question-metric (GQM) approach to identify relevant measures [1,4,5, 15]. There are three primary results: ∞ Insight into the enterprise’s problems in software development, ∞ Recommendations for improving the development process, and ∞ Identification of problems that require research to solve. This chapter focuses on the first two. Our process for collecting software development data and analyzing it has undergone considerable evolution over time, and has had continuing impact, from both internal and external viewpoints. We use both qualitative measures (interviews, surveys) and quantitative measures (financial, organizational, code repository, defect repository, quality) to assess our impact. Our purpose in this chapter is two-fold. It provides a model for assessment, based on twelve years of Avaya experience, which others may emulate. Such assessment leads to continuing improvement and substantial impact. In addition it spotlights analyses and conclusions that we feel are common to software development today. Note that there are other organizations that conduct such assessments, but with few exceptions, such as [13,26], they usually don’t publish details about their results or methods