FINAL EVALUATION REPORT: EarthSky en Español

Summary of Findings Summative Evaluation Cielo y Tierra Noticias del Mundo Robert L. Russell, Ph.D. Learning Experience Design October 2011 Learning Experience Design conducted summative evaluation of the project entitled: Cielo y Tierra Noticias del Mundo (Sky & Earth News of the World). A series of three evaluation sessions were conducted with radio listeners to look at the impacts of the programs on their interest in the topics presented, their understanding of the content presented, and their motivation to take further action. The first two studies used focus group sessions. The third and final study combined written surveys (using both rating-scales and open-ended responses) and focus group discussions. For the focus group sessions, the listeners, who were Spanish speakers and who were regular users of Spanish-language media, were recruited through community listserves and community organizations. A total of ten sessions were conducted in Washington, DC and San Antonio, Texas. An online survey was also conducted with radio station managers to look at factors influencing their decision to air the program and their assessment of the programs' relevance to their audiences. The key findings from these evaluation studies are presented below: Reach: The programs and web content reach a large Spanish-speaking audience, as evidenced by the strong national radio affiliate network, audience ratings data indicating over 2 million impressions daily, and a strong web presence, with over 2 million page views per month. Radio Station Managers (RSMs): In an online survey, radio station managers (10 responded out of 20 who were asked to return the survey) rated EarthSky en Espa programs very highly, finding the program topics and content very relevant for their listeners and the format very engaging. RSM's rated EarthSky en Espa separately on a series of items. They were asked to respond to the question, On a scale of 1 to 7, where 1 is very dissatisfied and 7 is very satisfied, how would you rate your overall satisfaction with EarthSky en Espa in terms of (item rated). The items (and mean rating in parentheses) were: Quality of the Content (6.62); Program Format (6.38); Easy to Understand Content (6.54); Accuracy of Information (6.58); Current Topics (6.62); and Range of Geographic Regions Covered (6.08). Thus, RSM's rated the programs at nearly the highest level across all items. In their comments, RSMs cited the high quality of the production of the programs, the relevance of the topics to the Latino community. Managers also contributed numerous positive comments about quality of the programs, the relevance and importance of the topics to the Latino community. Audience Interest: Radio listeners found the programs presented in the focus group sessions highly engaging, as indicated by a qualitative analysis of the content of focus group discussions and by the results of listener ratings in the third study. In the third listener study, three-fourths (75%) of listeners responded that they would not flip the channel and listen to another station, if they heard the programs on the radio. In the third study, listeners were also asked to rate how interesting they found each program on a four-point scale, with categories boring, not interesting, interesting, and very interesting. Across the five short-form programs, 70% or more of listeners rated each program as interesting or very interesting. In the focus group discussions, factors that contributed to listeners' engagement included: topics that they are familiar or that they identified as having relevance to their lives; interview segments with scientists embedded in the programs; natural sounds; and music. Listeners Understanding: A substantial majority of listeners exhibited recall and comprehension of the content of the programs, as indicated by the content of focus group discussions and an analysis of written responses to open-ended questions in the third study. In that study, across all of the five short-form (60 second) programs, over three-fifths (61%) of listeners were found to have a good understanding the concepts presented in the programs, as indicated by a classification of their responses indicating a General Idea or higher level of understanding (i.e., the sum of Clearly Understood,13%; Understood,19%; and General Idea, 29%.) A significant proportion, 28%, Grasped the Main Idea. Only 11% showed little understanding or provided no response. Motivation: Some listeners expressed an interest in hearing more in the programs about what can be done (if the program presented a problem), such as how to address an environmental issue. It should be noted, however, that none of the programs presented a call to action.