PocketMacro App Summative Evaluation

PocketMacro is a mobile app designed by the Learning Media Design Center at Carnegie Mellon University in collaboration with Stroud Water Research Center, Carnegie Museum of Natural History and Clemson University, and stakeholder input. The PocketMacro app aims to help users better identify benthic macroinvertebrates commonly found in streams and other waterways. Rockman et al Cooperative (REA), an independent educational evaluation group, designed a summative study to explore the effectiveness of the app in supporting users’ aquatic macroinvertebrate identification. The purpose of the summative study was to compare the affordances and impacts of digital and print materials on participants’ knowledge, accuracy, and confidence related to aquatic macroinvertebrate identification. Sixty-four study participants with varying levels of ID experience were randomly assigned to one of three conditions: 1) Those who only practiced aquatic macroinvertebrate identification using the PocketMacro app, 2) those who only practiced using a printed dichotomous key and flashcard deck, and 3) those who practiced using both of the above digital and print materials. Study participants completed a pre-survey before receiving the materials, practiced with the materials they were given for a least 90 minutes, and then completed a resource usage log and post-survey. Ten participants were interviewed about their experiences afterwards.

The study found that practicing aquatic macroinvertebrate identification, regardless of whether it occurred with the print or the digital tools, had positive impacts on users':

• Knowledge of aquatic macroinvertebrate ID
• Perceived confidence in identifying aquatic macroinvertebrates
• Accuracy in identifying unknown aquatic macroinvertebrates at the Order and Family levels
• Confidence in identifying unknown aquatic macroinvertebrates at the Order and Family level
• Attempts to ID an unknown aquatic macroinvertebrate at the Family level
• Ability to help others learn about macroinvertebrates in the future

The PocketMacro app was viewed as more effective than the two print resources for seeing Order level characteristics during practice and in the future.

The app was more useful than the flashcards for:

• IDing unknown specimens in the future 
• Viewing images closely during practice and in the future

The app was better than the printed key for:

• Making aquatic macroinvertebrates more accessible to others
• Engaging students and volunteers with macroinvertebrates

Participants appreciated the app's ability to zoom in closely to see specific features, the flashcards as a user-friendly tool for studying Orders, and the printed key for viewing and comparing multiple Orders at once. Thus, each of the tools has difference affordances that suggest that using a combination of the three resources to practice and improve one’s identification skills may yield the most positive outcomes.