Strategic Outcome Progressions Conference: Exploring a Framework for Measuring Informal Education Outcomes and Institutional Impact

Various measures exist to assess learning outcomes from informal science, technology, engineering, or math (STEM) education programming, especially for youth audiences. Although there has been growing interest in the field among practitioners, researchers and evaluators for the development and implementation of shared, common STEM measures and frameworks, informal STEM learning (ISL) institutions (e.g., museums, zoos, aquaria, botanic gardens, and the like) face a unique challenge when they design and evaluate educational programs. By nature of the broad-ranging topic areas these institutions emphasize, programming seeks to reach audiences that vary in age, ability, cultural background, interest, and more. Participation also varies--for example, participants choose which in a series of exhibits they attend and how much time they spend at each. Given this planned-for variance, how do educators target appropriate learning outcomes? To what extent do outcomes map to activity complexity? How does programming success link to institutional goals? One solution may be a framework of outcome progressions. Progressions are within outcome categories (e.g., interest, attitude, knowledge, skills, behavior, science capital, etc.), the ordinal series of effects that can be mapped to programming that ranges from short duration with simple content (e.g. a hallway cart demonstration can trigger situational interest) to extended duration with complex content (e.g. a youth internship can generate well-developed personal interest). STEM educators, evaluators and researchers working in or with ISL institutions will come together to consider the utility and feasibility of an outcome progressions framework. With this framework STEM learning institution educators may be better equipped to understand the impact of their programming and how to design for it. This work will be undertaken by the COSI Center for Research and Evaluation in conjunction with the Office of Education, Outreach, and Visitor Services of the National Museum of Natural History and the PEAR Institute.

The primary aim of this multi-day virtual convening is to inform future development of the framework for outcome progressions. These progressions will reflect and accommodate outcomes targeted by existing and shared evaluation measures. The framework will provide informal educators and evaluators with (1) a way of aligning outcome expectations with programming depth and dosage; (2) outcome choices achievable with various depths of programming; (3) a strategic tool for planning institution-wide targeted outcomes across a full portfolio of institution educational programming; and (4) a tool for documenting an education department's varied outcomes across the full range of possible outcomes. To create rich and informed critical discussion, the conference will involve research consultant experts representing each outcome category; practitioners representing diverse programming; and evaluation theorists familiar with the broad needs of informal learning institutions. A conference subcommittee with expertise in culturally responsive and equitable program development and evaluation will ensure the agenda contains flexibility to generate emergent equitable and inclusive conversations. Conference participants will review and contribute to the soundness and utility of these categorizations as well as the progressions within them. These insights and inputs will inform future development of the framework as a valid and reliable tool. Deliverables include a paper summarizing the findings from the convenings and a related webinar that describes the framework and recommendations for its use. These will be shared broadly via the informalscience.org website and other professional venues.

This conference is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.