The Why Files Research

Introduction  

The primary objectives of the CMA Team are to promote public understanding of SMET-related concepts and issues and to understand how the new communication channel of the World Wide Web can be used most effectively to communicate SMET concepts to mass audiences. During the first three years the CMA Team’s primary effort was designing and producing The Why Files (http://whyfiles.news.wisc.edu), the award winning World Wide Web science magazine created by NISE. At the end of year 3, funding for The Why Files site was shifted from NISE to the Graduate School of the University of Wisconsin-Madison. Since year 2, the team has also been conducting evaluative research specifically on The Why Files and more general research on how science learning takes place on the World Wide Web. This latter work has now become the primary focus of the team.

Our research program began with a descriptive focus during years 2 and 3 and, in line with our initial plan, moved into a more explanatory phase for years 4 and 5. Our early descriptive research enabled us to make recommendations for the redesign of The Why Files to make it more effective and efficient as a learning tool. We also now better understand who uses the Web for SMET information and how they use it. As our work completes its explanatory phase during year 5 we will be in a better position to make broader statements about SMET learning on the Web as well as more general theoretical contributions to the study of learning via media.  

Accomplishments to Date  

The accomplishments of the CMA Team during year 4 (previous accomplishments are detailed in previous NISE Overviews) can be classified into three primary categories: (1) continued empirical research on how individuals learn from the World Wide Web; (2) dissemination of data and findings; and (3) consulting for NISE Teams on Web site design.  

Empirical Research  

The research component of the CMA Team began midway through year 2. The CMA Team’s long-term goal developed at that time was to study the uses and effects of SMET information on the World Wide Web in two phases: (1) by describing who was using SMET information on the Web, how they were using it, and why they were using it; and (2) by determining the effectiveness of communicating SMET information via the World Wide Web and determining what individual-level and information-design factors were most strongly related to learning with this new medium. During years 2 and 3 the research conducted by the CMA Team focused on describing the users of The Why Files and how they used the site. We were able to describe the characteristics of and number of users of The Why Files, what parts of the site were most frequently viewed, what temporal patterns of navigation were evident, and how users processed the information provided in the site. This descriptive information provided the foundation for the second phase of this research.  

The work we have conducted in year 4 and will continue during year 5 requires that we experimentally assess the effectiveness of learning via the World Wide Web compared to more traditional sources like print magazines, as well as determine what individual and structural variables can increase the effectiveness of learning via the Web. Specifically, during year 4 we (1) concluded a second, more extensive survey of repeat users of The Why Files, tapping their exposure to science information carried in other media, their motivations for use of The Why Files, their trust in The Why Files relative to other sources of science information, and their strategies for processing science information; (2) conducted a series of experiments examining the impact of cognitive load (as manipulated by complexity of the structure of the information), motivation (learning vs. entertainment goals for SMET information exposure), and learning strategy (high vs. low levels of cognitive elaboration) on self-efficacy for learning about science and actual learning about science; and (3) conducted an experiment examining the impact of medium of delivery (Web vs. paper) and degree of cognitive elaboration on interest in the content, self-efficacy for learning about science, and actual learning about science. We are currently integrating the results of this research, using them to clarify and refine our research strategy for year 5 as described below.  

Dissemination  

In addition to empirical research, the CMA Team has endeavored to make more widely available not only the conclusions of our research but also the raw data on which they are based. We hope that this effort will encourage other researchers to tackle the issues in SMET communication that we are studying and will effectively enlist researchers structurally outside the NISE in the efforts of the NISE, ultimately generating more value but without increasing the cost of the CMA Team’s research.  

Our dissemination efforts took place on four fronts during year 4. First, we worked with the University of Wisconsin-Madison’s Data and Program Library Service (DPLS) to develop a Web-accessible archive of the data collected by the team through year 3 regarding the usage patterns for The Why Files and our initial survey data on repeat users of The Why Files. These data will also be used as DPLS’s test case for a new method of data distribution being developed by the University of Michigan’s Inter-University Consortium for Political and Social Research. Already a number of researchers from across the United States have downloaded these data and now have the potential to conduct secondary analyses of this rich data set; as already noted, they will be able to build on the work the CMA Team has already done to further our understanding of the Web as a tool for communicating SMET information.  

The second component of our dissemination efforts also makes use of the Web. We have begun working with NISE’s Project Manager to develop a Web site for the CMA Team to disseminate the results of our research on Web-based SMET learning. We will provide a summary of our current work and future plans on this site once it is completed. In addition, we will provide links to the DPLS archive of our data to increase its accessibility for interested researchers. We will also provide a link to the Web site of the journal Public Understanding of Science, which has made an electronic version of our article published by the journal available at its site. Finally, we are developing a bibliography of research on the uses and effects of new learning technologies that will be part of the team’s Web site. The site will be regularly updated as we learn more from our research.  

The third component of our dissemination efforts during year 4 took a more traditional route. We actively published and presented our work and findings among researchers and professional communicators interested in communicating science to the public. We published findings from our first year of research in the international journal Public Understanding of Science. The results of our cross-disciplinary review of the literature on hypermedia, the technology underlying the World Wide Web, was recently published by NISE as Research Monograph 15. A different version of this manuscript is currently under review at Communication Yearbook, an annual collection of state-of-the-art literature syntheses published by the International Communication Association. The results of one of our year 3 studies is currently under review at the cross-disciplinary journal Media Psychology. We also delivered two invited talks about our NISE research at the 1998 Association for Education in Journalism & Mass Communication conference, one to a combined panel of the Communication Theory and Methodology and Communication Technology and Policy divisions, and the other to a combined panel of the Visual Communication and Science Communication divisions.  

A fourth component of our dissemination efforts involves communicating about our research with more general audiences. This domain of presentations is just getting under way as awareness of our research spreads. For example, in spring 1999 the CMA Team gave a University Roundtable presentation at UW-Madison titled “Surfing the Web for Science.”  This forum attracts a wide variety of university faculty and staff. Similarly, Dunwoody made repeated presentations about The Why Files while conducting a series of science communication workshops for scientists, students, and writers in South Africa in early spring 1999. And the CMA Team has been invited to discuss The Why Files research at a World Wide Web session being proposed for the 2000 meeting of the American Association for the Advancement of Science.  

Consulting for NISE Teams  

In addition to dissemination of our data and results in formal publications and Web sites, we have also worked to disseminate our knowledge informally as the opportunity arises, during year 4 serving as a resource for NISE more generally. For instance, after successfully providing input on the redesign of The Why Files during year 3, during year 4 we consulted with the CL-1 Team regarding the design of their Collaborative Learning Web site. After our thorough review of the site, we were able to make detailed and specific suggestions about site layout and design based on what we have learned from our review of the existing literature as well as our specific experience studying The Why Files.  

We expect to continue this consulting effort for other NISE teams. One major contribution during year 5 will be providing our experience and expertise in evaluating Web sites to the Formative Evaluation Team. The Formative Evaluation Team proposes to study and make recommendations for the CL-1 Team’s Field-tested Learning Assessment Guide (FLAG) Web site during year 5. The College Level One Team plans to incorporate site navigation and survey methods similar to those we used during years 2 and 3.  

Strategic Plan for Year 5  

In our plan for years 4 and 5 in 1998 we proposed several experiments designed to determine the role of a number of individual and site-design factors in learning SMET information from the Web. We identified these factors through an extensive review of the pertinent literature (recently published as NISE Research Monograph 15) and our own research experiences during years 2 and 3. The initial findings of our year 4 experiments have allowed us to narrow and refocus our efforts for year 5, enabling us to more efficiently study what now appear to be the key factors.  

Based on the results of our year 4 work, we will continue to focus much of our effort on the impact of elaborative information processing on learning from the Web. Elaborative processing is the act of making connections between new information in the environment (i.e., in a Web site) and stored knowledge, connecting new information to one’s personal experiences or other information previously stored in memory. The Communicating with Mass Audiences Team’s research, as well as the work of others studying learning from more traditional communication media, indicates that elaborative processing is one of the most important individual-level predictors of learning. We plan to examine both the origins and the impact of elaborative processing of SMET information on the Web by pursuing two interrelated strands of research.  
 

Continuation of Learning Experiments Focusing on Elaborative Processing  

Our first order of business in year 5 is to develop a more thorough understanding of how elaborative processing of Web SMET content encourages learning. Most experimental research on the role of elaborative processing in learning has attempted to manipulate this construct in one of two contexts: (1) the materials to be learned were very focused and narrow, such as word pairs or simple sentences, and verbal instructions or pointed questions were used to encourage elaboration on this very specific content; or (2) the outcome was learning of college course content, and the manipulation of elaboration was accomplished by providing lengthy instruction—typically a full college course teaching the effective use of a number of learning strategies—that could lead to increased elaboration by individuals in a classroom setting.  

By comparison, the focus of the CMA Team is on the use of relatively more complex, lengthy, SMET information than materials in the first context described above, and our manipulation of elaboration (and the corresponding exposure and learning tests) must be accomplished within a very brief period, much less than in the second context and generally the same as research in the first context. The challenge so far has been to design manipulations that are both strong enough to encourage the appropriate cognitive processing throughout the content while at the same time realistic enough to maintain ecological validity. For instance, in our first experiment we found that simply providing written instructions asking participants to engage in activities defined as elaborative processing was insufficient (based on postexperimental self-report manipulation checks) to produce substantially more elaboration in experimental compared to control groups. However, when elaboration was examined as an individual difference, we found that it was indeed correlated with greater learning of the content. If we are successful in developing a reasonably powerful and yet realistic manipulation, our findings will be more meaningfully adapted to actual informal SMET learning contexts using the Web.  

During year 4 we conducted research in which we experimentally manipulated elaborative processing of lengthy SMET articles on topics such as flu viruses, tornadoes, and wildfires, all of which were derived from stories originally appearing in The Why Files. These stories were presented as print magazine articles or as a Web site depending on the condition. During year 5 we plan to continue this line of research, focusing primarily on Web-based delivery of the content, since our research so far indicates few if any differences in learning between Web and print conditions. We propose to refine a questioning technique—adapted from elaboration research examining recall of word pairs—to more precisely manipulate and thus determine elaboration’s effectiveness for encouraging SMET learning from the Web.  

Experiment on Variations in Processing by Site Design Using Think Aloud Protocols  

Our second goal for year 5 is to determine what Web design factors can encourage (or discourage) the natural production of elaborative processing. We expect to manipulate the design of the site structure—for instance by organizing it in a hierarchy versus a narrative, or providing frequent versus infrequent hyperlinks—and then conduct a think-aloud study to measure the effects on information processing. “Think aloud” is the label given to a method designed to measure the processing of information by encouraging individuals engaged in a task—in this case using the Web to learn about SMET—to express all the thoughts going through their head as they occur. These comments are recorded and then quantitatively coded as indicators of different types of information processing, including elaboration. We propose to use this technique to determine how the design of Web sites may encourage or inhibit naturally occurring elaborative processing. Theory suggests that poorly designed Web sites (and potentially even well-designed ones) may produce disorientation, or a feeling of being lost in the virtual environment. This potential and actual disorientation can lead to excessive cognitive load—the mental effort required to orient oneself to the structure of the information in a Web site. Since the cognitive resources of any human are limited, expending substantial effort orienting oneself to the design of a Web site could consequently reduce the available cognitive capacity of the user to engage in elaborative processing of the content. Because elaborative processing is a key means by which information in the environment is interpreted and stored for later recall, poor site design could ultimately reduce meaningful learning.  

Research conducted by the team during year 3 using think-aloud protocols to observe naturally occurring elaboration suggested that some SMET Web sites can encourage greater elaboration than other sites. Specifically, we observed more frequent elaboration when participants were surfing in The Why Files than when they were viewing content elsewhere on the Web. At this time we cannot make strong claims about why we observed these differences because this initial study was not designed to make causal inferences. That is, instead of experimentally manipulating site design or exposure to The Why Files versus other sites in this first study, we were seeking descriptive information on how individuals process SMET information on the Web, specifically in The Why Files. Users made their own choices about what Web content they would view. Thus, our evidence for site design effects is correlational, and we are unable to make strong causal inferences.  

During year 5 we anticipate conducting a think-aloud study in which site design is experimentally manipulated while site content is held constant. The results of this study will allow us to make stronger inferences about how SMET sites can be organized to encourage greater elaboration and thus greater learning. We hope to be able to increase elaboration—and learning to which it is strongly related—simply by altering site design, without the necessity of directly manipulating elaborative processing at the individual level via instructions as we and others have done in past experiments. If we are able to do so, we can disseminate information to SMET Web site designers that will allow them to modify existing site designs to increase the degree of elaborative processing among users. Once enacted, these redesigns should produce more frequent elaboration and thus increase overall learning of SMET information on the Web.  

Deliverables  

¨       Communicating with Mass Audiences Web site including new technology bibliography and review of past and future research (Eveland, Dunwoody)

¨       NISE Research Monograph integrating the results of our research on processing of SMET information on the Web and experiments on the role of elaboration in learning of SMET content on the Web (Eveland, Dunwoody)

¨       “Examining Information Processing on the World Wide Web Using Think Aloud Protocols,” currently being reviewed for publication in Media Psychology (Eveland, Dunwoody)  

¨       “Applying Research on the Uses and Cognitive Effects of Hypermedia to the Study of the World Wide Web,” currently being reviewed for publication in Communication Yearbook 25 (Eveland, Dunwoody)

Staff: Sharon Dunwoody and William Eveland (Team Leaders), Gregory Kelly, and Tom Wiggins.