Interacting with Professional Audiences

The IPA Team is charged with wide dissemination among policymakers and professional organizations concerned with SMET education of research learning, promising products and models, technologies, and educational practices resulting from NISE work. The IPA program also creates occasions for dialogue among NISE researchers and the communities having interest in and a need for NISE products. The IPA team accomplishes both the dissemination and the outreach/interaction missions directly through organizing the NISE Annual Forum, but also assists other NISE teams in in planning and conducting dissemination activities. The Annual Forum convenes representatives of the many NISE audiences to share research and developments in an area of SMET education reform selected for its timeliness and salience. The IPA team also has recruited and is working with some 25 selected professional organizations that are collaborating with the NISE more intensively. These Collaborating Organizations assist in the work of the NISE and in disseminating NISE results in ways that are tailored for reaching the organizations’ members.

Below is a summary of plans for the Institute's interactions with professional audiences. The first three items are continuations of important NISE dissemination activities that have proved effective over the last three years. The last item is conceived as a strategic new opportunity to ensure that the SMET community will be able to take advantage of the Institute's work.

Convene Annual NISE Forums. The Year 4 Forum will focus on the work of Strategies for Evaluating Systemic Reform (SESR) team. The team expects to have the draft of its book on evaluating systemic reform in mathematics and science completed by early winter and the Forum will be built around the findings and recommendations in the book. The Year 5 Forum will focus on a new NISE initiative or new work from a continuing team (e.g., the PD team).

Collaborate with Other Organizations. We will continue to convene the annual meeting of NISE Collaborating Organizations. In addition, we are starting a new bimonthly electronic communication specifically written for them and other key SMET organizations.

Provide Dissemination Assistance to NISE leaders and teams. This includes negotiating commercial book publications and shepherding the subsequent production process, suggesting venues for articles in professional and popular journals, and making connections with the Collaborating Organizations for distribution of NISE work.

Write "spin-off" articles. These will be specially designed for prominent periodicals and news publications of NISE Collaborating Organizations, publications of other relevant SMET organizations, and the general education media, when appropriate. Four such articles are planned for each of Years 4 and 5.

Each of these activities is further described below. We are convinced that launching the spin-off articles at this time of the Institute’s life is highly desirable and could have a potent influence on the SMET community. While refereed publications assist in building academic credibility for the Institute, NISE publication in prominent periodicals of strategic SMET organizations would probably achieve greater impact among practitioners, curriculum supervisors, project directors, and administrators. In this vein, for example, the PD Team's commercially published book probably has had more impact on professionals than any other NISE product; The Why Files is addressed to a more general audience.

There are several reasons for increasing the NISE dissemination effort during Years 4 and 5: (1) A number of results and products now exist or will be coming to fruition as a consequence of the earlier years of NISE work; (2) the Institute's considerable and high quality work merits the widest possible exposure; and (3) dissemination is critical for NISE work to have the desired impact. Regarding the third point, it is a well-known phenomenon that the academic R&D enterprise and the federal agencies that support it budget their investments in exactly the opposite fashion as does the private sector: Investment is high for research, modest for development, and minimal for dissemination (i.e., marketing). To date, this has been characteristic of NISE, and appropriately so, since one has to have results or products before dissemination or marketing can take place. The IPA staff argue that the Institute will have attained reasonable maturity by Years 4 and 5; therefore the balance between research and dissemination needs to be adjusted in favor of getting the Institute's work out in form suitable to its various audiences.

Accomplishments to Date

The Annual NISE Forum

The three annual NISE Forums have been very successful in two ways: (1) creating awareness and a generally favorable perception of NISE and its work and (2) generating state-of-the-art knowledge on pressing topics in the SMET community. Even though the NISE Forum is a new event on the scene, it has been fully subscribed every year by a very rich mix of participants and has attracted supplemental corporate support each year. Individuals from virtually every U.S. state have represented many kinds of institutions (dozens of professional organizations, all types of postsecondary institutions, a variety of research and development projects, and diverse types of school districts). Participants' feedback from the formal evaluation questionnaire as well as anecdotally through subsequent communications has been overwhelmingly positive.

As for generating new knowledge, the caliber of Forum participants has made it more than worthwhile to collect and analyze the perspectives offered in the Forum's small-group discussions and transform them into reports. The top flight experts who have been enlisted to speak also have contributed to the reports through their prepared papers that are circulated in advance of the Forum. The Year 1 Forum discussions led to the commercially published book by Susan Loucks-Horsley and her PD team. With the the Year 2 Forum, the Institute set a new precedent for incorporating the on-site work of participants in the Forum’s report, thereby expanding the knowledge base on systemic reform. The IPA team plans to discuss with commercial publishers their interest in the report on the Year 3 Forum just held on assessment issues in postsecondary education, which similarly will capture front-line research and development in this critical area. Some papers written by the Forum speakers have independently been receiving acclaim. For example, Susan Loucks-Horsley's Year 1 paper was reproduced in Science Educator, the newsletter of the National Science Educational Leadership Association. Many requests for Uri Treisman's Year 2 paper have led to Institute plans to transform it into an NISE Brief. The National Research Council is circulating several papers from the recent Year 3 Forum to participants in one of its upcoming conferences on undergraduate education.

The Forum is a good example of how the Institute is successfully drawing together the work of its teams. While the IPA team coordinates all aspects of the Forum, every team has been integrally responsible for the success of the Forums. The NISE management regularly monitors and facilitates the development of each year's Forum. The research teams have worked hard not only to highlight their NISE work but also to draw in leading experts from other organizations. The Formative Evaluation (FE) team has helped design the details of the Forum program so that speakers' papers and participants' comments can subsequently be analyzed by FE and transformed into a published report.

Collaborating with Other Organizations

The IPA team has initiated relationships with some 25 organizations interested in collaborating with NISE. This somewhat small effort to date has had some important results, for example, (1) several organizations have formally lent their names and/or efforts to putting on one of the annual NISE Forums: AAHE, CCSSO, CPRE, NCTM, NGA, NSDC, and NSTA; (2) the publication divisions of NCTM and NSDC are formally codistributing the PD Team's book and NSTA is considering this as well; (3) senior staff of other organizations have volunteered to provide critical reviews of draft NISE publications and attended working meetings of NISE research teams; (4) NCTM, NSDC, and (particularly) NSTA have invited NISE staff to make featured presentations at their annual and/or regional conferences; and (5) several organizations have asked NISE staff to confer with them on issues of mutual interest; for example, the Council of Scientific Society Presidents (CSSP) asked NISE’s Co-Director, Andy Porter, to write a research agenda for the organization. By the end of NISE Year 3, the IPA team will confer with the most strategic NISE collaborating organizations to determine how to increase these mutually beneficial interactions in Years 4 and 5.

For a list of Collaborating Organization Representatives, click here.

Dissemination Assistance

The most notable success of the IPA team's efforts to work with NISE staff on dissemination activities is its work to negotiate and execute the commercial publication of the PD Team's work by Corwin Press and the secondary distribution of the book by NCTM and NSDC. This book is making far more impact than if it had been self-published by NISE. For example, in its first month, the book already had become one of Corwin's top three all-time sellers.

The team also has produced a paper that describes and organizes all of the Institute’s dissemination activities in an easily grasped manner. We also have put together a comprehensive guide (currently in draft form) on how to effectively organize, run, and report on large-scale conferences and smaller symposia, based largely on our work with the Forum but also other NISE meetings and workshops. When published, this will be the only guide of its kind that specifically addressess conferences and meetings held for educational and research audiences.

Strategic Plan for Year 4 and Implications for Year 5

Annual NISE Forum

For Year 4, the Forum will feature the work of the SESR team on the critical issue of developing new paradigms for evaluating systemic reform initiatives, synthesized in a book the team is planning to have near completion by the time of the Forum. Obvious audiences for the book’s findings and recommendations are researchers from the American Evaluation Association and the evaluation section of AERA; PIs, evaluators, and other project personnel from NSF's systemic initiatives; and staff from NSF and other federal agencies supporting systemic reform efforts. We plan, however, to expand the audience to the broader community concerned with SMET reform and particularly with evaluating its outcomes. Through the interactive processes we have designed for the NISE Forums, we plan to engage this broader audience as well as the participating evaluators and principal investigators in how they might change their conceptions and future designs for evaluating systemic reform efforts as a result of what they learned at the Year 4 Forum.

By Year 5, the Institute is likely to be faced with several candidate topics that have sufficient drawing power for a major Forum, for example, the more recent work of the PD team (as proposed in their current plan), or the work coming from a new initiative, such as the proposed analysis of the recurring failures of reform SMET curricula to find their way into the hands of sufficient teachers to enhance SMET learning of all students. The relative merits of these two candidate ideas and others that will probably arise from NISE work coming to fruition in the next two years will be reviewed toward the end of Year 4.

Collaborating Organizations

The Institute expects to maintain and strengthen its working relationship with these organizations by continuing to hold an annual meeting to provide early awareness of emerging NISE work and to solicit their views on it. Further, the IPA plans to strengthen its effort to contact key organizations on a more regular basis. The Institute largely has focused on establishing its own dissemination mechanisms by means of the Forum and distribution of publications. IPA will interact with the collaborating organizations to enable them to assist us in disseminating NISE work. Such contact will permit IPA staff increasingly to understand the organizations' activities and priorities and, consequently, to make more effective recommendations on how NISE work could benefit them. Continued contact also will result in these organizations thinking of NISE as a partner more often for future substantive, joint projects. Finally, writing "spin-off" articles also in part involves working with the NISE collaborating organizations; that is, the IPA team will actively work with editors of appropriate publications of collaborating organizations to ensure the most mutually beneficial timing and placement of NISE "spin-off" articles to be written in Years 4 and 5.

A new activity for increasing collaboration will be to produce and circulate an electronic "newsletter" for the NISE collaborating organizations and other key SMET organizations, as well as NSF staff and NISE team leaders. In contrast to the existing "NISE News," which mainly serves as an internal update, each edition will have one to three entries highlighting recent and emerging NISE research and results and feature new NISE publications as they come on line. Just as importantly, the communication will invite feedback on planned NISE research and occasionally invite interested parties to serve as external reviewers of draft products. Such a communication will facilitate regular contact between NISE and some 40 organizations. This electronic newsletter has been instituted as a result of requests for such a communication by participants in the February 1998 annual meeting of NISE Collaborating Organizations. CEOs of several of the large organizations attending even said they would want to be the recipient rather than having it addressed to their staff.

Dissemination Assistance

The IPA team will continue to prompt NISE teams to plan effective dissemination and build it into their work. Since progress in NISE work will result in a greater number of dissemination possibilities in Years 4 and 5, the IPA team also will increase its assistance to NISE management in making strategic decisions about deploying dissemination resources.

Individual NISE team members will continue to publish their work in the journals that match their professional affiliations. Over and above that, it is the IPA team’s responsibility to help the other NISE teams think more broadly about dissemination possibilities and, when appropriate, assist in making the needed contacts and shaping relevant articles. In addition, we have found that commercial book publication is another very effective means of bringing the work of the Institute to a large audience; it also makes possible a much more comprehensive treatment of such complex subjects as professional development of teachers, analysis and evaluation of systemic reform, and assessment of student learning in undergraduate SMET courses than is possible in journal articles. Hence, another activity of this small part of the IPA portfolio will be to work with NISE team leaders in negotiating commercial books and facilitating their production, as we successfully did for the PD Team's book. Candidate book projects are the SESR and PASR books, the PD Team's newer work, and the CL-1 team's work on assessment and evaluation (including a synthesis of the Year 3 Forum proceedings). This assistance by IPA generally will consist of (a) identifying prospective publishers and contacting acquisition editors, (b) helping teams create a publishing prospectus, (c) reviewing relative advantages among publishers, (d) helping meet production timelines, and (e) arranging codistribution with other companies and organizations.

"Spin-off" Articles

Our conception of "spin-off" articles is to tailor—in terms of content and communication style—relevant portions of NISE research to the interests of various target audiences. This activity is intended to increase the impact and visibility of NISE work, something advocated by the NISE Advisory Board but typically given limited attention in the work and funding of academic enterprises. The plan we propose for accomplishing such publications is intended to overcome a critical dissemination barrier, namely, how teams can (especially their leaders) accomplish more dissemination activities while at the same time getting the work itself completed. This vexing problem is a valid concern and probably a contributor to the generally anemic dissemination and, therefore, impact of educational research no matter how good.

Raizen and Britton will take the lead in facilitating this activity, drafting the articles, and shepherding their overall production. Their experience and success in doing this for NCISE, combined with the expertise that Porter, Stewart, and White bring to this effort, will ensure that high-quality articles will be produced. The other NISE teams will be integral in deciding which articles to produce and will have input and review functions throughout; in all instances, attribution will place the research members first, followed by IPA staff. There might be some instances when the team itself may prefer to take the lead in producing an article and can find the time to do so on schedule. In those instances, the IPA staff offer to review and provide editing suggestions. The writing process will entail two major steps: identifying candidate "spin-off" articles and achieving consensus on selections among NISE teams, leaders and IPA staff; and leading the production of articles. The following further describes each activity:

Selection of articles. This step seems simple but actually involves considerable effort. The goals are to be comprehensive in identifying possibilities, prompting any given NISE researcher to think "outside the box," and to be strategic in considering the costs and benefits of each proposed article. The IPA team will propose a list of potential "spin-off" articles by matching identified elements of NISE work and their implications with the typical contents and audiences of various periodicals. We estimate that each of 4 or 5 major NISE research efforts could give rise to about 5 or 6 potential articles, for a total of 20-30 candidates. For some of these, we will contact editors directly to explore their level of interest. After circulating the list of all proposed articles to team leaders and NISE management, we will schedule a 2-3 day visit to have a series of 2-3 hour meetings with each team, beginning and ending with NISE management. The NISE management subsequently will make the final determination of which articles to produce, perhaps after consulting with the Management Team and the NISE Advisory Board. The goal is to select topics for producing four articles in each of Years 4 and 5.

Production of articles. Below is a summary of the proposed process:

• Based upon existing NISE writings, IPA writes the first draft for a given spin-off article,
• Appropriate team(s) review and recommend changes,
• IPA revises to produce second draft,
• NISE management and team(s) review second draft,
• IPA creates final draft and submits for publication,
• IPA circulates any publisher requests for major changes, and
• IPA revises per all NISE and accepted publisher feedback.

Raizen and Britton will take responsibility for writing the first drafts for most of the articles, though there may be instances when a team prefers and has the time and people to generate the first draft or some specific elements of any article. It is imperative for teams to review any first draft dealing with their work; NISE management, however, could elect whether or not to review a given first draft. It is a given that management will provide input and feedback on each article at some stage before publication.

Primary Deliverables

• Annual NISE Forums
• Electronic newsletter to NISE Collaborating Organizations
• Increased collaboration with other organizations
• Commercially published NISE books
• 8 "spin-off" articles of major NISE work in prominent publications

Staff: Senta Raizen (Team Leader) and Ted Britton.

Collaborators: The IPA team collaborates with virtually all NISE leaders in effecting and facilitating the Institute’s dissemination activities. Further, the IPA staff confer with dozens of other organizations as an explicit part of conducting their work.

Communicating with Mass Audiences⁴

The Why Files

The primary objectives of the CMA Team are to promote public understanding of SMET-related concepts and issues and to understand how the new communications channel of the World Wide Web can be used most effectively to communicate SMET concepts to mass audiences. The CMA Team's primary effort has been the continuous development of The Why Files (http://whyfiles.news.wisc.edu), a Web site that explores issues of science and technology that underpin the news of the day.

While traditional mass media such as newspapers, television, and radio play a critical role in informing the public about news and issues of science and technology, they are often limited in their resources and ability to provide depth and context. And while traditional media pique public curiosity about science and technology, The Why Files seeks to satisfy that interest by offering in-depth, informed, and critical looks at current topics. For example, among the more than 35 new feature packages produced this past year, The Why Files visited such topics as genetically engineered crops, providing detailed insight into the advantages and risks of a technology that is transforming the farm and the products produced there. When El Nino began to disrupt patterns of weather and climate, The Why Files not only explored its history, cause, and effects, but the technologies for forecasting and coping with the influence of the phenomenon. Last fall, when public health officials in Hong Kong identified a threatening new strain of influenza, The Why Files took a close look at the business of flu research and composed a package that explained both the intricacies of the flu virus and how it changes to confound efforts to prevent infection.

The Why Files routinely incorporates new features and ideas designed to enhance the site and make it easier to use and navigate. A number of those changes have been driven directly by the research conducted on The Why Files site by members of the CMA Team.

Changes in package design and approach to a given topic may be subtle or pronounced, but the basic formula that has made The Why Files a popular success remains. That formula involves a carefully crafted combination of accurate but humorously written text, eye-catching graphics, photos, selected hot links to other Web sites, and occasionally video and sound. That formula is now employed on a weekly basis (up from biweekly in Year Two) to produce a feature package that keys off of events in the news. By keeping to a rigorous production schedule, the site has expanded considerably in the past year, and the dynamic nature of the site is a feature that draws a loyal and growing following. Another key feature of the site is the Cool Science Image gallery, which draws on pictures and images from the front lines of scientific research. The site also hosts an active net forum and the occasional contest, both of which boost the site's drawing power.

Accomplishments to Date

During Year Three, there have been several new developments. One primary change has been the addition of shorter feature packages to run every other week, alternating with our longer biweekly features. The new short features, focusing mostly on new discoveries, means that The Why Files now adds new editorial content on a weekly basis. Another soon-to-be-unveiled change is a complete redesign of the popular Cool Science Image feature. The look and navigational features of the Cool Science Images will more closely mirror the rest of the site. Archived images will be accessible in a format that parallels the way The Why Files feature packages are archived.

Changes in Year 4

The most significant development of Year Three for the CMA Team, however, is an agreement that the production side of The Why Files become an entity of the Graduate School of the University of Wisconsin-Madison. Under the agreement, The Why Files will no longer be an operational part of the National Institute for Science Education. This change will take effect at the start of the new fiscal year (July 1, 1998). The production staff, function, and purpose of The Why Files will remain intact, but the research side of the CMA Team will remain a part of NISE and continue its research to see how users of The Why Files operate within the site in an effort to understand how people learn on the web. As a part of UW-Madison's Graduate School, The Why Files production team will continue to develop features and the site in much the same way as it has done under the auspices of NISE. The Why Files Team hopes to maintain a strong but less formal affiliation with both NISE and the National Science Foundation.

During Year Three, The Why Files continued to maintain a profile as one of the most popular and critically acclaimed science-related sites on the World Wide Web. By providing an in-depth look at the SMET content of current affairs and news, The Why Files has developed an informational niche that attracts a broad audience in terms of both demographics and interest.

Use of the site remains steady. Statistics gathered during one two-week period indicate that people at more than 50,000 individual computers tapped into The Why Files during that time, requesting more than 180,000 pages of The Why Files. People from at least 115 different countries have visited the site.

The Why Files also continued to enjoy critical success. Among more than a score of awards and citations received in the past year: PC World Magazine's Top 101 Best Web Sites; Top 100 Sites, POV Magazine; Recommended Site, The Alchemist (the World Wide Club for the Chemical Community); Recommended Site, the Microsoft Network in the UK; Way Cool Site, Web Scout.

The Why Files continues to build collaborations with other organizations.The team worked closely with the NSF Office of Legislative and Public Affairs to produce a package in concert with the 1998 National Science and Technology Week. One major metropolitan newspaper, the Richmond Times-Dispatch, excerpts The Why Files weekly on its science page. Requests for links are routine, and numerous schools, businesses and other organizations have established direct links to The Why Files over the course of the past year.

In addition to statistical measures and awards, The Why Files receives a steady flow of anecdotal reports, mostly email from readers, which indicate The Why Files is well read and routinely shared by teachers, students, parents, health care professionals, scientists and others.

A sampling of recent comments from readers:

I just wanted to let you know that I think that your site is great. I thoroughly enjoy the variety of topics you cover and the amount of information you put into it! Especially great is the fact that many of the articles are on topics that had been making waves in the newspapers only days before. The site is wonderfully designed—very easy to navigate. It's a great way to learn about science—and since it's becoming increasingly important for the public to be aware of what is going on in the science arena, it's great to have people out there taking the mystery out of science and making it interesting and accessible.

I am currently teaching ninth grade physical science and tenth grade Applied Biology/Chemistry at Danville Area High School in Danville, PA. I have just recently come across your site and have been using it for reference materials to share with my students in the classroom. Your information is presented in a very clear and concise way. It is also very entertaining and current. I find it to be a very valuable educational research tool.

I am a statistics professor at Richland Community College in Decatur, IL. One of my requirements is that students try to find references to statistics and comment on them. One of my students came across your site on Polling. I was so impressed by the material here that I would like to make it suggested reading for my class and place a link to it from my statistics homepage.

With the support of NISE colleagues, The Why Files continued to draw together experts on the news media, science communication, and the scientific community to forge a product that transcends the interests of any single institution or organization. The work of The Why Files will continue in that vein as the program makes the transition from NISE to the University of Wisconsin-Madison. We anticipate that The Why Files will maintain its global perspective and ties to NISE and NSF.

Staff: Susan Trebach (Team Leader), Terry Devitt, Yael Gen, Darrell Schulte, David Tenenbaum, and Amy Toburen.

The Why Files Research

We are working to elucidate how people use science information on the World Wide Web and how learning of this information may be facilitated by the individuals using the Web and the designers of the site structure and content. So far our research has provided primarily descriptive information about how people use science information in one Web science magazine, NISE’s The Why Files <http://whyfiles.news.wisc.edu>. This award-winning site covers the science behind the news headlines, providing in-depth information on a timely basis as well as a searchable archive of SMET stories.

What our work has not done to date is examine the effects of The Why Files or the factors that may make SMET Web sites more or less effective. The research that we propose here will build upon the first two years of descriptive research we have conducted in order to clarify how science may be communicated via the World Wide Web to most effectively increase users’ interest in science, self-efficacy for learning about science, and knowledge about and application of scientific information. These findings can then be disseminated to practitioners to help them more effectively communicate science to the general public.

Accomplishments to Date

As part of our work for the NISE, over the past 18 months we have (1) conducted a large-scale review of the cross-disciplinary hypermedia literature; (2) analyzed detailed computer-collected usage data from The Why Files (representing about 50,000 users) to assess navigational patterns through a complex SMET site; (3) collected and analyzed responses to an on-line questionnaire distributed to frequent users of The Why Files to determine their characteristics and interests; and (4) conducted and content analyzed think-aloud and indepth interviews among a sample of users to observe and probe some of the cognitive processes involved in using Web sites for SMET information.

We are currently in the process of collecting data for two additional studies. The first is a multiwave panel survey of users of The Why Files. The first wave of this data collection effort measures independent variables of interest such as the cognitive effort employed while using the site, the perceived credibility of the site, and use of other sources of science information. Subsequent waves will test recall and application of information presented in stories in The Why Files that were read by the individual respondents. These data will not only provide us with a more detailed description of users than the first survey described above, but will also allow us to assess the influence of individual-level characteristics on learning from The Why Files in a naturalistic context.

The second study for which we are currently collecting data is a pilot study for the research proposed for Years 4 and 5. In this 5 x 2 x 2 design, we use information processing theory to predict the interactive influence of different learning strategies, motivations for using The Why Files, and different levels of cognitive load (as operationalized by the complexity of site design) on self-efficacy for learning about science and actual learning. Subjects for this pilot study are undergraduate students at the University of Wisconsin-Madison.

Academically, to date our work funded by the NISE has led to (1) a synthesis of the hypermedia literature that was named a "Top 3 Faculty Paper" by the Communication Technology & Policy division of the 1997 meeting of the Association for Education in Journalism and Mass Communication and is now under review for publication in Journalism & Mass Communication Monographs; (2) a paper presenting detailed information from our initial site navigation and survey findings presented at the 1997 meeting of the International Conference on the Public Understanding of Science and Technology that is currently under review for publication in the international journal Public Understanding of Science; and (3) an NISE Brief describing the resources needed to create and maintain a site like The Why Files and early data on how the site is used and by whom. We will also be discussing different aspects of our research at two invited panel sessions at the 1998 meeting of the Association for Education in Journalism and Mass Communication, one cosponsored by the Communication Theory & Methodology and Communication Technology & Policy divisions and the other cosponsored by the Visual Communication and Science Communication divisions.

Our initial research has also laid the foundation for our descriptive (and to a lesser extent associational) understanding of the uses and effects of The Why Files as a means to communicate science to the public. Pragmatically, our research has resulted in several revisions to the structure and content of The Why Files, the most prominent of which was a substantial redesign of the home page and site information section during fall 1997. Finally, publicity of our research through presentations at academic conferences, colloquia at which our Why Files research was the focus (University of Wisconsin-Madison, University of Washington-Seattle, Cornell University, North Carolina State University, and Kent State University) or at least a significant part (University of California-Santa Barbara and University of Connecticut-Storrs), and news media coverage has produced the beginning of fruitful outreach activities. For example, during fall 1997 and winter 1997/1998 we were contacted for advice on Web design and evaluation issues by representatives of organizations such as the Cornell Theory Center (Cornell University), the National Center for Genome Resources (Santa Fe, NM), and the Integrated Microscopy Resource (University of Wisconsin-Madison).

Strategic Plans for Year 4 and Implications for Year 5

Overview

For Years 4 and 5 we intend to make overt linkages between characteristics of individuals and the Web site on the one hand and learning about science on the other using a formal experimental design. We expect to learn about better designs for the structure and content of science Web sites that can then be used to economically communicate up-to-date and archived science information to the general public.

Based on our review of the hypermedia literature and our first three studies, we have identified variables that appear to have important influences on the uses and effects of this new medium to communicate science. Of these variables, two groups—the motivations that people have for using hypermedia systems and the design and structure of such systems and their content—appear to be among the most important factors influencing the way people use hypermedia and the cognitive effects of that use. In our proposed research, we would examine the impact of these two key groups of variables on how people use SMET Web sites, science interest, self-efficacy for learning about science, and science learning, from the well-grounded theoretical perspective of human information processing.

1. Research Questions/Purposes

Motivations for SMET Web Site Use. In general, motivations influencing hypermedia use may be characterized as either internal (self-generated) or external (other generated). By far the more common type of motivation examined in the hypermedia literature has been an external, formal learning motivation. In the typical study, motivations are externally imposed by subjects being randomly assigned to either a search-based (closed) task, in which they are asked to find discrete units of information situated in specific locations in the hypermedia system, or a browse-based (open) task, in which they are asked to integrate loosely connected units of information located in different places in the hypermedia system. These studies suggest that the type of task has considerable impact on whether hypermedia will be more effective and/or efficient than other media such as traditional paper text. Specifically, hypermedia is more effective for open learning tasks requiring browsing and integration of information from several locations.

The manipulation of intentional learning tasks in past research was appropriate because the goal usually was to evaluate the effectiveness of a hypermedia system for classroom learning or other formal instruction. However, studies of learning from the mass media in real-world settings—a type of informal education—assume that many, if not most, of the users will not have specific, externally defined, intentional learning goals. To accommodate this more general scenario, we believe that Web research should move from comparing two or more external, intentional learning goals to the common comparison in psychology of intentional versus incidental learning conditions. This comparison would allow the research to apply to both formal classroom or distance education situations (intentional learning) and informal SMET education (incidental learning).

While learning goals are important, we suspect that they are unlikely to be immediate causes of the quantity or quality of learning. We concur with the prevailing assumption in psychology and education that goals serve as antecedents of different types and quantities of information processing. Information processing, in turn, is the direct cause of differences in learning. Therefore, although goals may be manipulated by the researcher (intentional) or left to the participant to formulate (incidental), in either case the amount and type of information processing of content in which the individual engages should be an important (measured) mediating variable in the process of SMET learning from the Web.

Design Factor: Hypermedia Structure. Hypermedia structure has also been extensively studied to determine how to most effectively design systems to facilitate use and to increase desirable outcomes such as SMET learning. Factors such as maps of system content and linking strategies have been identified as important variables that may influence the uses and effects of hypermedia systems like the World Wide Web. How might the influence of these small design factors be examined within a more broad-based theoretical perspective? As suggested by some past research, the notion of learner control and/or information processing theory may provide a suitable framework within which to study these effects.

The learner control hypothesis studied in educational psychology predicts that, when individuals are able to determine the content, pace, and/or order of instruction, they should be more likely to learn from the content. Similarly, the users should feel more efficacious and should be more motivated to continue their efforts to learn in the future. However, research to date has lent only intermittent support to the learner control hypothesis; in fact, some scholars have concluded that learner control may be beneficial only for those who already have a solid grasp of the content domain. (This finding may be particularly important in the case of SMET information, which may require more extensive background knowledge.) Full learner control offers too many decisions for the novice, they contend, who is unable to handle the freedom (or, depending on one’s viewpoint, chaos); thus, it has been suggested that designers add "instructional advisement" to computer-based instructional systems to reduce the decision-making demands on novices.

Similarly, the research on hypermedia systems has indicated a consistent problem for users that has alternatively been described as being "lost in hyperspace" or simply "disorientation." In effect, because of the limited number of cues available for spatial orientation within the virtual space of the hypermedia system, users—particularly novices—have difficulty finding the desired information and often become "lost" and frustrated with the system. Psychologists remind us that humans are by nature cognitive misers, and thus the cognitive "load" or "overhead"—defined here as the amount of cognitive effort needed to successfully navigate or orient oneself in the virtual space—may cut into the cognitive effort employed to comprehend or learn the content itself. Our think-aloud interviews from Year 3 indicate just this: those who expressed confusion while surfing the Web were less likely to make connections between the information they encountered and their past experiences and stored knowledge. Since the type and amount of information processing of content is a direct determinant of learning of that content, the cognitive load induced by the structure of the hypermedia system is likely to be negatively related to its effectiveness for learning of the content. In addition, the unsettling feelings of being lost in hyperspace may reduce users’ sense of self-efficacy and, consequently, their cognitive effort and potentially even their future use of the site.

Design Factor: Narrative Structure. Another design factor that has been suggested to influence how and what is learned from popular informal science media like newspapers and television is the narrative structure—the form of the story and the style in which it is conveyed. Narrative structure may be particularly important in the World Wide Web context, where narrative may be the only clear connection to novices between nodes (pages) of a site, but little research has examined this issue.

The think-aloud interviews we conducted in summer 1997 revealed that subjects were more likely to make connections between the site content and their own knowledge or past experiences when they were in The Why Files than when they were in other (primarily) science-related Web sites. We suspect, but cannot formally demonstrate, that this is in part due to the unique narrative structure of the stories in The Why Files. In our proposed research we hope to isolate—through systematic manipulation—the type of narrative structure that encourages greater elaboration and integration of science information, holding things like motivations and site structure more generally constant. The implications of identifying a narrative style that can encourage more and deeper processing of information—and thus learning—would be an important and practical finding for science communicators generally and SMET Web site designers specifically.

Elaborative Processing of Site Content. Above we have described three variables—motivations, cognitive load, and narrative structure—that, when manipulated, should influence the amount of learning accomplished by users of The Why Files. However, as alluded to above, we believe that the influence of these three manipulated variables is indirect through the amount and type of information processing in which users engage. Specifically, a large and generally consistent body of research in cognitive and educational psychology has indicated that one instrumental factor in learning is the process of elaboration—making connections with stored knowledge and past experience—on the information provided. We believe that our manipulated independent variables will indirectly influence learning through their impact on elaborative processing. Those who are more highly motivated to learn, who have the available cognitive capacity (due to low cognitive load), and who are presented with a familiar and clear narrative structure are more likely to engage in elaborative processing, and thus will be more likely to learn from science Web sites like The Why Files. Most importantly, if we are correct, at least in the case of the two site design variables, Web designers can increase learning from their sites by encouraging greater amounts of elaboration by their visitors through these design factors.

2. Rationale

We seek to understand the effects of the variables of motivation, cognitive load (as induced by site structure), and narrative structure on information processing and on SMET learning from the Web. We propose to examine these variables from a human information processing perspective to assess their influence on the uses and effects of SMET World Wide Web sites. Our work to date studying the uses and effects of The Why Files SMET Web site has led us to develop a preliminary model of the process of learning from the Web, within which the variables described above fit. Briefly, we believe that variables such as demographic factors, cognitive styles and abilities, and expertise in the medium or content area serve as important exogenous variables that can influence variables that come into play later in the learning process. Motivation and characteristics of the medium (such as hypermedia structure or, more specifically, the cognitive load induced by this structure) and content (narrative structure) should also serve as important determinants of learning through their direct impact on a user’s perceptions of self-efficacy and the amount and type of cognitive effort invested in learning.

This proposed research specifically, and our research more generally, fits in well with the purpose of the NISE. Since much of the learning that takes place in any individual’s life is through informal media, our research using informal channels provides breadth for NISE’s main focus on formal learning about science. Even more so, our intentional versus incidental learning comparison will provide clues to any potential differences in the effectiveness of the Web as a learning tool in formal versus informal contexts. One question we hope to answer is, Is the Web equally effective in formal and informal science learning contexts, and if not, why not?

Finally, the research that we are doing would not be able to be communicated nearly as effectively to professional audiences and educators without the assistance of the NISE. For our work to move beyond theoretical importance, it must be brought to the attention of those who can best use it to make changes in policy and practice. Through NISE publications like the NISE Brief and NISE Research Monograph, as well as connections we’ve made with the Interacting with Professional Audiences team, we can make our research have broader impact.

3. Importance to Theory and Practice

As a result of the research we have conducted to date and the proposed research for the next two years, we will better understand (1) differences between how visitors use SMET Web sites and the relative effectiveness of these sites in formal versus informal learning settings; (2) the effectiveness of information processing behaviors like elaborative processing for learning about science from the Web as compared to print sources; and (3) the influence of cognitive load on learning from the Web. Armed with this information, three groups of people with more pragmatic interests will be able to make more informed decisions: policymakers can make more informed decisions about the investment of funds for communicating science via the Web; educators can make decisions about how to most effectively use the Web in their classroom and better instruct students in the most effective strategies to use to learn from the World Wide Web; and science communicators can make decisions about the design of their Web sites to allow greater learning of the primary content.

Sketch of Research Procedures

Based on this theoretical and empirical foundation, we will utilize the variables in our proposed model in a series of experiments. These experiments will be designed to examine the possibly interactive influence of the exogenous variables as well as motivations, cognitive load, and narrative structure on mediating or criterion variables such as self-efficacy beliefs, invested cognitive effort and elaboration, and content recall, recognition, application of the information to real world decisions or "knowledge in use."

Stimulus Material. Our experiments will use SMET content derived from The Why Files Web site. In some experiments, stories covering different science topics will be used to provide a degree of generalizability across different SMET topics.

Manipulated Independent Variables. As suggested above, we intend to operationalize motivations as a comparison between intentional learning contexts and incidental learning contexts as has been done in years of psychological research concerning basic learning processes. The self-reported amount and type of cognitive effort invested, the strength of learning motivations, and perceived cognitive load will also be measured to serve as manipulation checks and as mediating variables.

Cognitive load would be operationalized by the complexity of choices provided while holding constant the basic scientific content and time available to subjects to use the content. For example, in the first experiment in the series, SMET information contained in a Web site like The Why Files will be transferred to paper text, changing the links and other structural features to traditional print structure (i.e., removing those features that facilitate learner control of the order and content) but maintaining the color, text, and informational graphics as in the Web site. This paper version will serve as the stimulus material in the low cognitive load condition, while the same information delivered via the Web with different levels of learner control features left intact will be used in the moderate and high cognitive load (or Web) conditions. Later experiments will include comparisons of different linking strategies, the use of different types of content maps, and other variables in the Web condition that have been suggested to influence cognitive load within hypermedia system use.

Narrative structure will be manipulated by comparing the traditional newspaper format (i.e., inverted pyramid), print magazine format (e.g., traditional print narrative, Why Files narrative style), and other, less common formats while holding the Web context and the informational content constant. Based on the preliminary findings of our think-aloud protocols, we expect that the narrative style of The Why Files, and potentially some other styles, will generate greater cognitive effort and more frequent elaborations on the site content and thus greater learning.

Measured Mediating and Criterion Variables. We expect that our manipulated independent variables will influence several concepts of interest: perceptions of self-efficacy, amount and type of cognitive processing, and learning. We have developed our own measure of self-efficacy for learning about SMET information through a pilot test, and the amount and type of cognitive effort will be measured using variations on validated self-report indices of the amount of invested mental effort in learning and elaborative processing of information. We will also measure elaboration and factors contributing to cognitive load (i.e., time spent orienting oneself to the content or structure of the site) via a content analysis of think-aloud protocols conducted in conjunction with some of the experiments, extending the fruitful work we did during Year 3 using this method. The ultimate criterion variable for our studies will be learning of the stimulus content, which should be influenced directly by the mediating variables. To measure learning, we intend to develop content-based items to tap recognition memory, free recall, conceptual understanding, and application of the content to real world decisions.

Participants. Instead of using only young and relatively Web-literate undergraduate students as subjects in these experiments, we propose to draw our participants from the general population of computer users in the area surrounding the university. This strategy is consistent with the design and implementation of our Year 3 think-aloud research. This decision, while increasing the cost for recruitment and participant remuneration, will address one of the largest deficiencies in the experimental hypermedia literature—the use of extremely homogeneous participants not representative of the target audience of Web sites like The Why Files. We understand that nonrepresentative participants do not necessarily harm the ability of experiments to assess causality. However, if individual characteristics of participants (such as age, experience, or education) interact with manipulated independent variables as some past research has suggested, we will not be able to detect those interactions in a sample that essentially holds these characteristics constant. Thus, while we do not seek a probability sample for our experiments, we do seek a group of participants with meaningful variability in demographic characteristics.

Timeline

We expect to complete three experiments during Year 4, with several more planned for Year 5. In addition to the deliverables described below, we would continue to present our research at mass communication and science communication conferences and to publish in these fields.

Primary Deliverables

During Year 4 we expect to generate a Research Monograph describing the cross-disciplinary theoretical and empirical literature on the uses of and learning from hypermedia systems, integrating work done in the fields of educational technology, computer science, library and information studies, cognitive and environmental psychology, and geography. Further, we will link the hypermedia literature to the study of the World Wide Web—the largest hypermedia system in existence—and generate a framework for the study of informal science education via the World Wide Web.

Staff: William Eveland and Sharon Dunwoody (Team Leaders).