Challenges in Using Comparison Data in Child Welfare Evaluations

Objective:This article describes the evaluation of permanency roundtables, an intervention to help youth in foster care achieve legal permanency and the challenges evaluators faced in finding and using appropriate comparison data.Method:In 2009, permanency roundtables were conducted for 496 children in Georgia, most of whom had spent extended time in care. Half (50%) achieved legal permanency within 24 months of the roundtables.Results:Five potential sources of comparison data, none of which provide an ideal comparison, are presented and discussed.Conclusions:Ideally, evaluators should plan for one or more comparison groups at the very beginning of the evaluation. However, post hoc comparisons should not be dismissed altogether. The challenge for evaluators is to generate valid comparison groups to determine whether an intervention made a significant improvement, despite challenges such as limited funding, tight timelines, and unmeasured contextual differences between groups.