Room for Rigor: Designs and Methods in Informal Science Education Evaluation

ABSTRACTThe authors examine methodological characteristics of summative evaluations in informal science education (ISE), asking: What are the major types of designs used in summative evaluations, and what kinds of questions can they answer? What are the types of data collection methods and measures used, and how many are self-reports or direct measures? They reviewed all summative reports from the year 2012 on informalscience.org, the online resources portal from the Center for Advancement of Informal Science Education. They found reliance on nonexperimental evaluation designs and heavy use of self-report instruments. If a primary function of summative evaluations is to assess impact, and impact is a causal question, then these findings are problematic; the field needs to move beyond the mostly descriptive studies found in the sample. Interviews with nine leaders in ISE and ISE evaluation help explain evaluation challenges in ISE and generate ideas for advancing the field.