Summative :: Koepfler, Jes A. (2011). STEPS Summative Evaluation Report. [UXR Consulting] University of Colorado, Denver.

last updated: 2011-04-11 15:09:59

Project Name	STEPS - Science Theater Education Programming System
Project Lead	University of Colorado, Denver
Project Website	http://stepsproject.org/
Associated Grant	NSF# #1043060
Report Author(s)	Jes A. Koepfler
Evaluation Organization(s)	UXR Consulting
Download Report	STEPS_Summative_Evaluation_Report_FINAL.pdf

Summary

Executive Summary
User Experience Research Consulting (UXR) conducted a summative evaluation of the NSF-ISE funded project, STEPS (Science Theater Education Programming System). The STEPS project brought together a network of informal science educators and contractors to create an interactive museum theater authoring and presentation system to increase educational capacity for small and large museums across the country. The software package includes an authoring tool for the creation of multimedia science theater productions; a presentation player for displaying the shows to audiences in museum theaters, planetariums, and in outreach facilities; a web-based tutorial for the astrobiology content and STEPS software; and three pre-packaged shows varying in length and dramatic effects to showcase the potential uses of STEPS for informal science learners ages 7-12. The professional development of the of informal science educators in the network was the primary goal of the project with all other tangible deliverables stemming from that collaborative effort.

This evaluation report synthesizes key analysis and findings from data based on longitudinal surveys, diary studies, and post-project interviews with the network partners; usability surveys with other museums professionals; and a quasi-experimental pre/post-show survey with students viewing one of the completed astrobiology shows.

Throughout the evaluation, STEPS was shown to be effective in achieving the goals set forth in the original proposal, particularly with respect to the professional development of the information science educators in the network. Year 4 research will build on these findings and identify the long-lasting impacts of a project like STEPS on the professional identity and self-efficacy of informal science educators. The following is a list of the most salient findings with respect to the associated audiences and evaluation questions addressed by this summative evaluation:

Primary audience: the museum network that created STEPS
>To what extent is the STEPS product used by individuals in the STEPS network at their institutions?

STEPS participants found a unique way to use STEPS to connect with their visitors to support the sustainability of the project beyond the grant. For example, a large museum is working with its teen program to provide the STEPS shows to audiences of all ages. A small museum took the opportunity to reach out to its local theater company to solicit actor volunteers who will sustain the STEPS program.
 

>To what extent was the process of developing these products recommendable for creating a collaboration/network?

The use of Team Leadership Theory, which was used to develop and maintain a bottom-up distributed leadership structure, was particularly effective for achieving the professional development and self-efficacy goals of the project. STEPS provided opportunities for every individual to learn a new skill, try something outside of his/her comfort zone, and take leadership roles based on individual self-interests. These benefits were exchanged for challenges with decision-making throughout the project and some project members who wanted more top-down structure, particularly at major milestone points. See section 5.0 for a detailed reflection on the benefits and challenges of the Team Leadership Theory framework.

Secondary audience: other science museum educators that will use STEPS and its associated products
>Was the process for developing these products recommendable for creating a more usable final product for science museum educators outside of the STEPS project?

Results from the system usability survey and diary study suggest that with the addition of the finalized software tutorial, the STEPS software will serve the unique needs of science museum educators in performing science theater in their institutions. Unique features, like timeline forking, which allows for multiple endings to a story, and the use of museum theater language in the labeling of buttons (e.g. “cues” and “scenes”), are a direct result of museum educator input.

Tertiary audience: museum visitors who will experience a STEPS show while visiting a museum institution
>To what extent are the show's learning goals and experiential outcomes with respect to museum theater achieved?

The results of the evaluation of the shortest and least theatrical of the three museum educator-developed astrobiology shows produced significant differences with large statistical effect sizes in a pre/post-unmatched pairs study. The target audience of 5th-7th graders who participated in the evaluation of the shortest and least theatrical of the three museum educator-developed astrobiology shows performed approximately 20% better on a test covering the shows content than a comparable group of 5th-7th graders who had not participated in the show. Similar results were obtained for 3rd graders. These findings demonstrate the accessibility of the show’s content to these audiences and the broad appeal of this science content area through a digital and theatrical approach.

>To what extent is the STEPS software system and associated storylines and components a successful product?

The content and delivery of the show used for evaluation (Planet Hunter) was appropriate for the full age range of the intended target audience of upper elementary and middle school students (7-12 years old). The three key features of the STEPS show that were cited as the things that students liked best were: the live actors, the multimedia elements, and the addition of multimedia characters. It is noteworthy that students in each group provided unsolicited positive feedback about the pre-recorded digital actors and asked about the pre-recorded digital actors; which speaks to the fact that presenting science content in a theatrical way is engaging and convincing.