Evaluation Tools and Instruments
Most evaluations require the use of a data collection tool—a survey or other data collection instrument. Evaluators either need to adopt or adapt tools “off the shelf” or create new ones. Either method can pose challenges: Tools that have been developed for one evaluation may not prove suitable for another, at least not without careful modification. At the same time, creating new tools requires expertise in measurement and instrument design.
How do you know if an off-the-shelf instrument is appropriate for your needs?
Good question! When considering the use of an instrument, keep in mind the following:
- What is the instrument measuring? Review how the instrument developers define what it is they are measuring. Does it match exactly what you want to measure? Also look for validity evidence that the instrument measures what it proposes to measure. Validity evidence can be from expert reviews, think-aloud interviews, factor analysis, and other validation techniques.
- What audience was the instrument created for and tested with? Instruments are created for a particular audience. If your audience matches the one that an instrument was designed for, great. If not, you’ll need to do some testing to see if the instrument works for your audience before you use it for an evaluation. For instance, a survey created for adults may or may not be appropriate for children. You won’t know until you test it.
- What context or setting is the instrument meant for? An instrument meant for one setting may not work well in a different one. For instance, a survey developed to measure an experience kids have in a school classroom may not be valid for evaluating an experience they have within a museum. Again, testing is required if an instrument is to be used in a new setting.
- Do I have the expertise to be able to judge the appropriateness and quality of the instrument? Experience with evaluation and instrument design are necessary to successfully choose and use an "off-the-shelf" instrument. If you don't have this experience, be sure to call on someone who does.
List of Evaluation Tools & Instruments
We’ve gathered a collection of tools and instruments that can be used for evaluating outcomes of informal STEM education projects or that can serve as starting points for modification. These sites provide access to tools and instruments useful for evaluating the wide range of outcomes addressed by informal STEM education experiences.
Research and Evaluation Instruments on InformalScience.org: This section of InformalScience.org allows users to search a collection of evaluation reports, articles, and other resources that include evaluation instruments and tools. It also includes assessment tools from Assessment Tools in Informal Science (ATIS). Use the metadata on the left side of the page to filter your search.
Assessment Tools in Informal Science (ATIS): ATIS is a searchable website of assessment tools for STEM learning in educational, especially out-of-school time, environments. The goal of the site is to provide practitioners, evaluators, researchers, and policy makers with a range of tools appropriate for assessing program quality and outcomes for children and youth.
STEM Learning and Research Center (STELAR) Instruments: STELAR supports the NSF’s Innovative Technology Experiences for Students and Teachers (ITEST) program. The STELAR website includes a searchable database of data collection instruments that have been used by various ITEST projects.
DEVISE: Developing, Validating, and Implementing Situated Evaluation Instruments (DEVISE) is a NSF-funded set of constructs and associated instruments designed to measure individual learning outcomes from participation in citizen science such as interest, motivation, self-efficacy, and skills. Each instrument comes with instructions for use and scoring.
Building Informal Science Education (BISE): BISE is a NSF-funded project that coded and synthesized 520 evaluation reports on informalscience.org to see what could be learned about evaluation in the informal science education field. As part of this work, the BISE team coded the evaluation reports for the presence of data collection instruments. On the BISE website, search the BISE NVivo Database or Excel File of BISE Report Level Codes to locate reports with attached surveys, observation instruments, focus group and interview protocols, timing and tracking forms, and much more.
The Learning Activation Lab: The Learning Activation Lab is a national research and design effort to learn and demonstrate how to activate children in ways that ignite persistent engagement in science, technology, engineering, art, and mathematics learning and innovation. As part of this effort, they have developed and tested a variety of instruments to measure constructs such as science learning activation, engagement, and scientific sensemaking.
Online Evaluation Resource Library (OERL): OERL is a NSF-funded project developed for professional evaluators and program developers. Although targeted for those who work in school environments, it provides extensive evaluation resources and samples — instruments, plans, and reports that can be modeled, adapted, or used as is.
From Soft Skills to Hard Data: Measuring Youth Program Outcomes: This document from the Forum for Youth Investment reviews ten youth outcome measurement tools appropriate for use in after-school and other settings to measure soft skills such as communication, collaboration, critical thinking, decision making, initiative, and self-direction. For each tool, it provides sample items and information about usability, cost, and evidence of reliability and validity.