Operationalizing science literacy: an experimental analysis of measurement

Inequalities in scientific knowledge are the subject of increasing attention, so how factual science knowledge is measured, and any inconsistencies in said measurement, is extremely relevant to the field of science communication. Different operationalizations of factual science knowledge are used interchangeably in research, potentially resulting in artificially comparable knowledge levels among respondents. Here, we present data from an experiment embedded in an online survey conducted in the United States (N = 1,530) that examined the distribution of factual science knowledge responses on a 3- vs. 5-point response scale. Though the scale did not impact a summative knowledge index, significant differences emerged when knowledge items were analyzed individually or grouped based on whether the correct response was “true” or “false.” Our findings emphasize the necessity for communicators to consider the goals of knowledge assessment when making operationalization decisions.