Summative Evaluation: Why Engineering? Program

The National Building Museum (NBM) contracted RK&A, Inc. to conduct a summative evaluation of the Why Engineering? distance learning program.  The goal of the evaluation was to assess program operations and explore the extent to which the program achieved its intended outcomes for students and teachers.

How did we approach this study?

RK&A used three methodologies for the study: online program observations; student assessments administered immediately after the program; and telephone interviews with teachers.  Observations were primarily used to gain a holistic understanding of how the program was implemented and to identify program strengths and weaknesses.  Written student assessments were designed to measure achievement of student outcomes, using mostly closed-ended questions and one open-ended question.  Teacher interviews were designed to measure achievement of teacher outcomes and to contextualize student achievement.

What did we learn?

Student assessments found that nearly all students learned something new from the program, most notably that engineers work together to plan, design, and build, indicating high achievement of two student outcomes.  Similarly, the pre- and post-program materials provided by NBM helped teachers feel prepared to use the program in their class.  However, technology remains a major obstacle to the program experience.  Both teacher interviews and student assessment responses indicated the poor audio quality affected students’ ability to understand the information and instructions provided during the program.  Still, the distance learning program is highly valued by teachers as a unique resource that provides an experience not available elsewhere—particularly, the video interviews with engineers describing their work and inspiration.