NISE Year 4 Accomplishments and Year 5 Activities

Information Resource Coordination Team

The Information Resource Coordination Team (IRCT) was formed in 1999 to examine issues in information technology use and knowledge sharing, including design of technology to support improved information flow. There has been a growing interest both among NISE projects and nationally regarding the needs for identifying and using online information resources to support an Educational Community of Practice (ECOP). The objective of the IRCT is to define and create improved electronic environments, based on the IRCT group’s combined expertise in human factors, information science, and engineering design of information and communication technology systems. The IRCT effort also emphasizes the role of design experiments combining systems engineering research, engineering design practice, and education applications.

Accomplishments to Date

Two interim reports were completed in May 1999. One report focused on hardware and software needs of students in an educational psychology course (the Derry STEP project); the second report described designs of a database for collecting and sorting ECOP resources contributed by community members. IRCT work in the July 1, 1999 – June 30, 2000 (Year 5) time frame will build on these efforts.

Strategic Plan for Year 5

The IRCT will focus on three major tasks for the Year 5 period. Each task has connections with broader ECOP projects, both internal to NISE (Webb Infonet, Derry STEP), and externally. EHR funded research groups (including CILT projects at UC, Berkeley and SRI, and the Math Forum at Swarthmore), and the National

Partnership for Advanced Computing Infrastructure (NPACI) at the UC, San Diego will partner with IRCT for these project tasks. A secondary school in Carbondale, Colorado, and local middle and secondary schools in Wisconsin, will be the sites of the design experiments conducted to demonstrate IRCT product feasibility. These sites were selected on the basis of three criteria: (1) interested teachers working in a socially isolated school setting, (2) planned changes in information technology infrastructure at each site during the 1999-2000 academic year, and (3) previous links to UW-Madison. (The principal at the Colorado school is a graduate of the UW-Madison School of Education; one teacher there is a professional colleague of Prof. Caldwell.) Project travel to Colorado and California is intended to support these relationships.

Project tasks are proposed as follows:

Task 1: Define information metadata needs for organizing online resources. Database systems such as PsychLit and ERIC are based on metadata (information about each resource in the printed collection) that combine both general library catalog information and subject information specific to the field. Currently, no consistent metadata structures exist for online collections of cognitive development theories, classroom practice and assessment techniques, or sites that can be used by teachers in the classroom. IRCT will define classification structures for online information specific to secondary mathematics and science topics. The Task focus is on the information metadata needs of teachers for organizing online classroom and professional development resources.

July 1 – August 31, 1999: Collect examples of online sites suggested by practicing teachers to focus on secondary school math and science topics. We already have a list of several hundred sites. Others will be collected at a TAPPED IN online “summer carnival” where teachers will be encouraged to submit sites with a description of each site. An extensive set of examples will allow us to ensure that our metadata description covers the variety of materials available online.

July 1 – August 15, 1999: Compare ERIC and other thesaurus terms with STEPWeb resource descriptions. This work will help us to ensure that the metadata that we develop will be compatible with terms already in use in the education and psychology subject domains.

September 1, 1999 – January 31, 2000: Complete report describing metadata needs assessments. Our focus will be on classroom-relevant sites and subject material, both for secondary school teachers and for educational theories related to the teaching of secondary school math and science.

Task 2: Distinguish metadata for academic (e.g., classroom teaching) school information from administrative (e.g., records and reporting) school information. Numerous tools exist for administrative management of school-level information; however, almost no tools are designed to link on line resources for classroom learning and teaching to online student, class, or school records. A consistent metadata language for information flow between academic and administrative data (and the software tools that manage that data) must be developed. Strong collaboration with the Webb Infonet project will continue with experiments to ensure that proposed designs are useful in school contexts. We hope to combine tools to help teachers become more effective and enable administrators to make better decisions. Task results will emphasize school design experiment implementation.

July 1 – September 30, 1999: Identify existing tools to integrate classroom and administrative information for schools. The results of this search will help to define the gaps between current practice and teacher needs.

October 1, 1999 – January 31, 2000: Describe initial requirements for metadata definitions based on NPACI schema and XML criteria. NPACI Storage Broker tools define how schema (metadata language description formats) can be described and coded in software. We will emphasize the difference between data relevant to classroom learning processes and data relevant to school administration. One project trip may be needed during this period to help coordinate with SDSC researchers.

February 1 – June 30, 2000: Provide a report of draft standards for academic/administrative metadata coordination. This report will be written for NISE target audiences, with a description of the schema being submitted to the SDSC Storage Broker. Project trips to California and Colorado for preliminary responses from SDSC and the Carbondale design experiment are planned for this period.

Task 3: Identify information technology implementation and services need to support ECOP effectiveness. Simply making software tools available to potential users, or providing information resources on the web, is not enough to assure their successful use. Caldwell’s previous research has shown that network availability and download times, timely and useful responses to user questions, and ability of the site to meet situational needs of visitors greatly determine user acceptance of online resources. A complete systems engineering evaluation of an information technology system includes analysis of these variables in a dynamic task environment. Two ECOP online communities have already responded with enthusiasm to offers of collaboration on this issue, the TAPPED IN community hosted at SRI and the Math Forum community based at Swarthmore College. The STEP project offers an opportunity to evaluate how a new online ECOP is developed.

September 1, 1999 – December 31, 1999: Evaluate the effectiveness of a centralized teacher information resources repository. Already, the number of NSF-, Dept. of Education-, and other-sponsored math and science web sites exceeds the capabilities of most inservice teachers to evaluate what sites best suit their needs. We will collect, index, and organize existing online teacher resources and will develop a “one-stop” web site that indexes secondary math and science web sites.

January 1 – June 30, 2000: Analyze non-software resources needed to support ECOP activities. To ensure the continuing success of web sites and to answer questions about the repeatability and scalability of online ECOP sites, data on hardware performance, network availability, and use patterns of both support staff and participating teachers are needed. Staff will coordinate with TAPPED IN and Math Forum to obtain and analyze data and summarize their experiences to guide future efforts. Additional information about ECOP development will be gathered from the Educational Psychology 301 course, developing as part of the Derry STEP project.

Primary Deliverables

¨ “STEP Project Information Flow and Web Site Design,” working paper under development.

¨ “Human Factors of Educational Information Technology Knowledge Webs” to be submitted to Human Factors and Ergonomics Society/International Ergonomics Association joint meeting (July 2000).

¨ Project report on improving NISE electronic information exchange and electronic access to NISE publications.

¨ Project report on integrating information technology tools to support preservice and inservice SMET teachers in electronic access to educational resource materials and professional community support networks.

Staff: Barrett Caldwell (Team Leader), Christine Lee, Enlie Wang, and Armand Ardika.

Fellow: Susan Zeyher.

Collaborators: ConneMara Bazley, Calvin Chan, William Clune, Sharon Derry, Judi Fusco, Pat Rossman, Mark Schlager, Christopher Thorn, Jeff Watson, Norman Webb, and Paula White.

NISE Formative Evaluation

The objectives of the NISE Formative Evaluation Team have evolved over the course of the NISE’s history. The first objective was to provide the NISE leadership with real-time information about the development of the Institute as an organization to help the NISE make mid-course corrections and thus maximize its goal achievement. By the middle of year 3, the NISE team leaders agreed that the NISE had evolved to the point where the value of additional formative evaluation had reached the point of diminishing returns. The formative evaluation objective was completed with the production during year 4 of a paper synthesizing the key themes that emerged from the formative evaluation work. A second objective of the team was to provide formative and summative evaluation of the NISE Forums. The team pursued this objective by evaluating the First, Second and Third Annual Forums, and the Graduate Education Forum. The prototype for Forum evaluation having been established, the NISE Central Office assumed responsibility for this activity as of year 4. During year 2, the team began pursuing a third objective, which was to draw on its social science expertise to produce syntheses and proceedings of the Second and Third Annual Forums, and the Graduate Education Forum. During year 5, the team proposes to shift to a fourth objective—conduct an impact study of one of the key products of the year 4 NISE effort, the College Level One Team’s Field-tested Learning Assessment Guide (FLAG).

Accomplishments to Date

The feedback that the Formative Evaluation Team provided during the first three years helped Institute leaders understand how various NISE constituents were experiencing and viewing the NISE teams and the NISE overall (Reports 1, 2, 4, 5, and 6, of the Internal Documents series, listed below). The Management Team, Team Leaders, and most team members used these reports to make organizational improvements. The value of this formative evaluation process is described in “Formative Evaluation: A Key to Fostering a Cross-Disciplinary Knowledge-Building Community of Practice within a Research University.” This paper, written during year 4, explores the processes and emerging principles by which one complex, cross-disciplinary organization—the NISE—developed. In addition, it locates the development of the NISE within the context of the research literature on other cross-disciplinary organizations, and makes a case for the utility of formative evaluation for the development of academic centers. The primary purpose of this paper is to serve the NISE, NSF leaders and program officers, and others seeking to foster productive multi-disciplinary academic research centers. Such understandings are of value as we move with ever increasing momentum into a period when key problems in science, mathematics and engineering education can no longer be solved by people from a single field of expertise, and few experts are trained to work in effective cross-disciplinary teams.

The Formative Evaluation Team evaluated the first three annual Forums and the Graduate Education Forum. These findings appear as Reports 2, 7, 8, and 9 of the Internal Document series. These reports enabled the organizers of each successive Forum to design national events that have improved with over time, and have established a reputation for excellence with an increasingly wide national audience.

Starting in the second year, the Formative Evaluation Team began supporting the NISE by helping with the planning and analysis of the annual Forums. During 1996-97, the team worked closely with the Strategies for Evaluating Systemic Reform, Policy Analysis of Systemic Reform, and Interacting with Professional Audiences teams to plan and organize the Second Annual Forum. In addition, the Formative Evaluation Team took major responsibility for managing the production of the forum proceedings and produced a forum synthesis based on participant think pieces. The synthesis comprises a key component of the proceedings and was very positively reviewed. During fall 1997, the team joined the College Level One Team’s Forum planning group. Because the members of the Formative Evaluation Team are researchers at the UW-Madison Learning through Evaluation, Adaptation and Dissemination (LEAD) Center, they brought expertise developed through their SMET evaluation research work. From February 1998 through the beginning of year 4 (1998-99), the team produced the year 3 Forum Proceedings, working closely with a CL-1 Fellow, Elaine Seymour. In addition, throughout year 3, the Formative Evaluation Team participated on the Graduate Education Forum planning group. During year 4, the team worked with the leader of the Graduate Education Forum to produce the Proceedings of this event.

Strategic Plan

For year 5 the Formative Evaluation Team plans to explore the ways in which the College Level One (CL-1) Team’s Field-tested Learning Assessment Guide (FLAG) Web site fits the needs of instructors who teach introductory science, math, engineering and technology (SMET) courses. The team will also explore the ways in which these instructors report using the assessment methods in their courses as well as any new understanding of assessment that they develop as a result of using the FLAG.

The FLAG Web site was originally developed by Steve Kosciuk through the NSF-funded Chemistry New Traditions grant and Elaine Seymour, as a year 3 CL-1 Fellow, and was subsequently restructured by the NISE CL-1 Assessment Institute in year 4. The FLAG Web site is intended to be a core resource of field-tested assessment materials for SMET instructors from all types of institutions. To this end, the Web site provides its visitors with an introduction to assessment, a description of a dozen or so alternative classroom assessment techniques developed by experts, and various examples of these techniques that can be downloaded and used (with some revision) in a course. Dissemination of this Web site began in May 1999.

¨ Does the FLAG Web site offer assessment materials that interest instructors of introductory SMET classes who visit the site and does it offer them in a format that these instructors find useful?

¨ Do these instructors believe that the FLAG Web site increased their understanding of assessment and the different assessment methods?

¨ Do these instructors report adopting, or plans to adopt, any of the assessment techniques into their classes?

¨ Besides the dissemination strategy employed in this investigation (see below), how else did these instructors learn about FLAG?

¨ Once these SMET instructors visit the site, where do they go, how long do they stay, and how easily do they:

find what they are looking for; understand what they've found; and connect what they find with what they can implement in their course?

¨ Do the SMET instructors who have visited the site plan to return? If so, why? If not, why not?

¨ How do the answers to above questions change when considering instructor variables?

A study of whether the FLAG is serving the needs of its targeted audience will first provide insight into the needs that SMET instructors have for learning about new assessment practices and, second, provide insight into the ways in which a Web-based resource can fulfill those needs. This will be helpful primarily to the year 5 CL-1 Team in refining the FLAG and developing a learning technology Web site for the same SMET audience. Additionally, the team expects that the published results of this study will also help others in science education as they develop Web sites that provide resources for faculty.

Methods

The team expects to interview 50 instructors who will be chosen to represent the various disciplines and institutions types of the intended audience. The one-hour structured open-ended interviews will explore the issues presented in the above research questions: the ways in which the FLAG fits the instructors' needs, what they learned about assessment, and what they have used or plan to use from the FLAG. In addition, the interviews will also cover issues of: the instructors’experience with alternative assessments or educational innovation, their access to instructional resources, and their experience with the web and technology. These last variables correlate with the way in which users learn from Web sites, since sophistication with the subject matter and technology can minimize the “cognitive load” that many first-time Web users can experience. The team chooses to use interview-based self-report data to explore these research questions for three reasons. First, the release of the CL-1 Team's FLAG is too recent for any significant behavioral changes to occur; self-report data may be the only reliable measure at this time. Second, the team believes it is important to use a broad sample from the target audience of the FLAG: the analysis of resulting 50 interviews will require as much researcher time as is warranted given the newness of the Web site. Finally, the use of open-ended interviews will enable the team to develop a richer understanding of the answers to the complex research questions that other methods of self-report (such as open-ended or closed-ended surveys) may not be able to capture.

The Formative Evaluation Team will obtain a substantively representative sample of the audience for which the FLAG is designed by disseminating information about the site to five SMET departments at five local postsecondary institutions and selecting instructors from the targeted departments who subsequently visit the site. These SMET departments comprise a subset of those targeted by the FLAG: biology, chemistry, engineering, mathematics, and physics. The five institutions are diverse in terms of institutional type and type of control: a public research institution of national reputation (UW–Madison), a private doctoral-granting institution serving its local region (Marquette University), a public comprehensive institution that offers Master's programs (UW–Whitewater), a private liberal arts institution (Edgewood College), and a public community college (Madison Area Technical College). The Formative Evaluation Team will hire UW–Madison graduate students from the five selected disciplines to give demonstrations of the FLAG to their respective departments at each of the five institutions. These demonstrations will be advertised to all members of the department through email or fliers and presented in a way appropriate for each discipline. A member of the FE Team will call people in the department who teach introductory courses to see whether they have visited the FLAG. Instructors who have visited the site will, with consent, be included in a list of faculty from which our interview sample will be selected. Instructors who have not visited the site will not be included in the interview sample, but will be queried on the telephone about why, as this information also is useful.

The team believes that, while its method for obtaining an interview sample will not necessarily produce a truly representative sample of users nationally, the method is among the most reliable available, given the resources and that it is no longer feasible to locate users through electronic tracers, or “cookies.”[1] In addition, during their interviews and in their analysis, they will attempt to take into account the effect of using graduate student presentations to inform their sample about the FLAG in order to improve the validity of their findings.

The team spent June and July 1999 identifying the five graduate students, introducing them to the FLAG, and then working with them to prepare presentations. The team anticipates that the CL-1 Team will have produced an outline of a presentation from which these students could work. In September, the students will give presentations to the appropriate departments at the five institutions, after preparing announcements through email and posters. In October, members of the FE Team will contact the instructors of the introductory level courses of these disciplines for interviews and will conduct these interviews through the end of December 1999. The team will then analyze the interview data and produce reports and other products from January through June of 2000.

Primary Deliverables

No. 1 Formative Feedback Report #1.1, Baseline Report on the Team Leaders Team and Management Team, October 1995.

No. 2 Formative Feedback Report #1.2 First Annual NISE Forum Evaluation Report, April 1996.

No. 3 Formative Feedback Report #1.3, Baseline Report on NISE "Intermediaries," April 1996.

No. 4 Formative Feedback Report #1.4, Report on Non-Team Leader Team Management Team Participants, April 1996.

No. 5 Formative Feedback Report #2.1 Report on Perspectives of the Co-Directors, Team Leaders Team, and Management Team, September 1996.

No. 6 Formative Feedback Report #2.2, Report on Perspectives of the NISE Membership, April 1997.

Clune, W. H., Millar, S. B., Raizen, S. A., Webb, N. L., Bowcock, D. C., Britton, E. D., Gunter, R. L., & Mesquita, R. (1997). Research on systemic reform: What have we learned? What do we need to know? Synthesis of the Second Annual NISE Forum, Volume 1: Analysis; Volume 2: Proceedings (Workshop Report No. 4). Madison: University of Wisconsin–Madison, National Institute for Science Education.

Millar, S. B. (Ed.). (1998). Indicators of success in postsecondary SMET education: Shapes of the future. Synthesis and proceedings of the Third Annual NISE Forum (Workshop Report No. 6). Madison: University of Wisconsin–Madison, National Institute for Science Education.

Millar, T. S., Mason, S. A., Gunter, R. L., & Millar, S. B. (1999). Synthesis of the NISE Graduate Education Forum. Madison: University of Wisconsin–Madison, National Institute for Science Education. (This document may also be distributed as a book.)

Millar, S. B. (1999). Formative evaluation: A key to fostering a cross-disciplinary knowledge-building community of practice within a research university. (to be submitted to Review of Higher Education and to appear as an NISE publication)

The team will provide continuous feedback to the CL-1 Team about its findings, as well as formative and summative reports of its findings. A formative report will be written midyear, and a summative report will be provided at the end of year 5. Depending on the nature of the findings, the team will submit an article to an appropriate journal. In addition, the team members plan to present their findings at an NISE brown bag seminar and at a professional organization meeting during the year.

[1] Eveland and Dunwoody (1998) used cookies in their study of the use of The Why Files. A cookie is a small text file that is modified on the user's hard drive at first access of an application on a Web server through some browser software. When a user visits the site again with this same browser software, the Web server searches for this file, reads it, and identifies the user's computer. Through the use of cookies, Eveland and Dunwoody were able to identify repeat users of the site and request that they complete a survey. However, the process of identifying a representative sample of users in this manner is becoming less reliable as users become more savvy with their browsers and don’t allow cookies to be saved on their machine.