Design and Implementation of Data Collection in a Large-Scale, Multi-Year Pre-College Engineering Study: A Retrospective

The data collection procedure and process is one of the most critical components in a research study that affects the findings. Problems in data collection may directly influence the findings, and consequently, may lead to questionable inferences. Despite the challenges in data collection, this study provides insights for STEM education researchers and practitioners on effective data collection, in order to ensure that the data is useful for answering questions posed by research. Our engineering education research study was a part of a three-year, NSF funded project implemented in the Midwest region of the US. The project has engaged more than 60 teachers from 15 different public elementary schools and one private elementary school from five different school districts, as well as homeschool educators. More than 1,000 students, ages kindergarten to second grade, have been involved. Through this project, children engaged in integrated STEM + literacy +computational thinking activities in formal, informal, and homeschool settings. For this multi-faceted project, data collection was complex. The primary data collected for this project was video-recordings of K-2nd grade-aged children as they engaged in curriculum activities in both classroom and homeschool settings, as well as in activities designed for and set in a science center setting. Video recordings allow us to examine the ways that the children engage in engineering design and computational thinking, as well as in mathematics, science, and literacy. Video recordings also allow us to examine the interactions between children, as well as interactions between children and teachers/parents. Additional data included: copies of student work (e.g. worksheets, engineering design prototypes); field notes collected during classroom observation and science center visits; post-implementation interviews with teachers and parents; and surveys. In addition, a new approach, referred to as the 1+2 technique, in video data collection was developed to record the targeted data. Overall, the main aim of this paper is to provide critical insights for researchers who anticipate implementing more successful, purposeful and effective data collection in elementary schools, specifically in K-2 grade levels. We also anticipate that this paper will help practitioners and professional developers consider how they might collect video recordings: whether for allowing practitioners to reflect on their teaching practices; allowing teachers to share with families the in-class activities that children engage in; or assisting professional developers in developing video-based training materials.