A Short-Term Impact Study of the National Science Foundation’s Instrumentation and Laboratory Improvement Program Table of Contents Prepared under Contract RED 94-52966 by Kenneth Burgdorf Tom Ewing Michael Feinstein Paul Tuss Stephen Roey Diane Ward Westat, Inc. with the assistance of Phillip Tavel NSF NSF Program Officers Mary Sladek, Division of Research, Evaluation & Communication Duncan McBride, Division of Undergraduate Education Directorate for Education and Human Resources [Image] Division of Research Evaluation & Communication NOTE: The views expressed in this document are those of Westat, Inc. and do not necessarily reflect the views of the National Science Foundation. October 1996 ------------------------------------------------------------------------ Table of Contents Executive Summary I. The ILI Program * History of ILI in the NSF Portfolio * Program Objective II. Signs Of Progress * The Early Years * Success in reaching target audiences o Institution Level o Students o Disciplines o Faculty o Other Coverage Dimensions + Consortia + Women + Underrepresented Minorities o Summary * Recent Feedback from Grantees o Resource Investment o Educational Impacts o Dissemination o Research Gains * Some Grantees Voice Concerns III. Looking Ahead * Evaluation Objectives ------------------------------------------------------------------------ EXECUTIVE SUMMARY ------------------------------------------------------------------------ The Directorate for Education and Human Resources (EHR) of the National Science Foundation (NSF), which periodically assesses all of its programs, is currently in the process of reviewing its Instrumentation and Laboratory Improvement (ILI) program. Under the direction of EHR’s Division of Research, Evaluation and Communication (REC) and Division of Undergraduate Education (DUE), the current study is scheduled for completion by the end of 1996. This three-section report is an initial product of the evaluation now in progress. The first section(The ILI Program) briefly describes the history, objectives, and structure of this annual, peer-reviewed competition, which provides funds for the purchase of laboratory equipment that will be used in projects aimed at improving undergraduate science, mathematics, engineering, and technology curricula. The second section(Signs of Progress) summarizes what the Foundation has learned thus far about the program’s success and shortcomings since its establishment in 1985. Information in this section is based on previous evaluations, the analysis of existing NSF record systems, and direct contacts with selected ILI grantee institutions as well as individual Principal Investigators (PIs) whose projects have incorporated laboratory equipment acquired under an ILI grant. Among the previous evaluations referred to in this section is a 1988-89 ILI study that was based on mail surveys of grantees and site visits to recipient institutions. This earlier evaluation revealed, for example, that in the first 3 years of ILI’s existence, projects funded by the program had provided students with previously unattainable opportunities for beneficial “hands-on” learning experiences; that ILI-supported projects were spurring progress among faculty members in improving their undergraduate science and mathematics curricula; and that many recipient institutions were successfully parlaying their financial and material support from NSF into broader recognition and support from other outside sources. Preliminary findings presented in the current evaluation--which focuses to a great extent on grantees from 1990 and 1992--affirm many of the positive conclusions reached in the earlier study, but go beyond those findings to answer a number of other very important evaluation questions. For example, the more recent findings presented here indicate that ILI has been generally successful in reaching its target audience; that many students have pursued advanced science/technology studies or careers following experiences in ILI-enabled laboratories; and that the presence of ILI-funded laboratory equipment has attracted new faculty at some institutions. At the same time, preliminary findings of the current study suggest some areas where further refinements of the program may be warranted--for example, to address gaps or unevenness of coverage, to reduce time burdens or delays sometimes associated with project start-up, or to facilitate more widespread dissemination of ILI-initiated innovations. The third and concluding section of this report (Looking Ahead) discusses the approach that REC is taking to further evaluate ILI’s impacts and to identify specific steps that might be taken to enhance the program. Table of Contents ------------------------------------------------------------------------ THE ILI PROGRAM ------------------------------------------------------------------------ Thanks largely to funding support from the National Science Foundation (NSF), Priscilla Laws, a physics faculty member at Dickinson College in Carlisle, Pennsylvania, was able to acquire a sophisticated microcomputer system for use in her innovative work to improve the teaching of undergraduate physics. Another NSF grantee—chemist Benjamin DeGraff, a professor at James Madison University in Harrisonburg, Virginia—teaches challenging courses on the use of lasers and advanced measuring devices to perform thermal analysis. The courses depend heavily on the presence in his laboratory of costly state-of-the-art laser units and measuring devices. The support he has received from the Foundation has made acquisition of these instruments possible. Likewise, NSF funding has allowed mathematician Eugene Herman of Grinnell College in Iowa to purchase the powerful software he needs to nurture his students' insights into linear algebra, differential equations, and calculus by representing mathematical ideas and objects in onscreen forms that permit manipulation and experimentation. Despite the diversity of their disciplines and pedagogical pursuits, Drs. Laws, DeGraff, and Herman have in common that they and their institutions are all beneficiaries of NSF's Instrumentation and Laboratory Improvement (ILI) program. ILI is an annual, peer-reviewed grant competition that provides funds for curriculum improvement through the purchase of laboratory equipment for use in projects aimed at improving undergraduate education in science, mathematics, engineering, and technology. ILI grants are in the $5,000-$100,000 range and must be matched with nonfederal funds; the grant and matching funds must be used solely for purchase of instructional laboratory equipment. ILI grants are awarded for projects that show promise of advancing the state of the technology in a given field: ILI projects are expected to produce models for the improvement of undergraduate laboratory instruction nationally, as well as at the project site. ILI has become one of the Foundation's most popular programs, attracting more than 2,000 proposals annually and awarding some $23 million a year to successful applicants. In its first decade, from the program’s inception in 1985 through 1994, ILI has considered nearly 18,000 proposals from nearly 1,800 different institutions (Table 1). It has made a total of more than 4,700 grants to applicants from approximately 1,200 different institutions and has awarded more than $158 million. Since grantees are required to match ILI funding dollar for dollar, the program has stimulated investment of at least $316 million in the acquisition of undergraduate laboratory equipment. ------------------------------------------------------------------------ Table 1. Cummulative ILI proposal and award data, 1985-94 ProposalsAwards Award amount ($ million) Total number/amount 17,793 4,704 $158.6 Number of institutions 1,777 1,185 -- Table of Contents ------------------------------------------------------------------------ History of ILI in the NSF Portfolio ------------------------------------------------------------------------ Established in 1985 as the College Science Instrumentation Program (CSIP), the ILI program bolsters the NSF mission of providing leadership for the Nation's efforts toward stimulating student interest in science, mathematics, engineering, and technology. Operated by the Division of Undergraduate Education (DUE)—a unit of NSF’s Directorate for Education and Human Resources (EHR)—ILI complements other programs aimed at enhancing the quality of instruction in institutions of higher education, including 2-year as well as 4-year colleges and universities. In addition to ILI, DUE's program portfolio includes Course and Curriculum Development, a program aimed at improving the learning environment, content, and experience of instruction; Undergraduate Faculty Enhancement, which supports projects enabling faculty members to gain experience with technological advances and new experimental techniques in order to incorporate them effectively into undergraduate instruction; and Collaboratives for Excellence in Teacher Preparation, which supports large-scale efforts to achieve comprehensive change in the undergraduate education of future teachers and to increase substantially the quality and number of teachers well prepared in science and mathematics, especially members of underrepresented population groups; and Advanced Technological Education, a program (administered jointly with EHR’s Division of Elementary, Secondary and Informal Education) that promotes curriculum improvement and teacher development in undergraduate and secondary technological education through national and regional alliances between associate’s-degree-granting institutions and other secondary and postsecondary institutions, business, industry, and government. During its first 3 years of operation (1985-87), ILI had annual budgets in the $5-$8 million range, with grantees—then restricted to non-doctorate-granting 4-year colleges and universities—receiving awards ranging from $5,000 to $50,000. Beginning in 1988, the program was given its current name and was expanded in several respects. Most notably, the range of eligible institutions was expanded to include 2-year institutions and 4-year doctorate-granting colleges and universities; and the upper limit of individual awards was increased to $100,000. Between 1988 and 1994, the program was flat-funded at $23 million annually. Table of Contents ------------------------------------------------------------------------ Program Objectives ------------------------------------------------------------------------ The principal objective of the ILI program is “...to support the development of experiments and laboratory curricula which improve the science, mathematics, engineering, and technology education of undergraduate students, both science majors and non science majors, including pre-service teachers. Proposals are sought for the development of innovative methods for using laboratory exercises to improve student understanding of basic principles and for use of modern instrumentation, new technologies, or new applications for instruments that extend the instructional capability of the equipment” (1996 ILI Program Announcement; NSF 96-10). In addition to its explicit, primary focus on laboratory curriculum innovation, ILI implicitly has several other program objectives. Many of these relate to the programs and personnel of the recipient institutions: * Encouraging and assisting individual institutions and departments to upgrade and expand their instructional laboratory equipment, facilities, and curricula by enabling them to acquire modern equipment they would not otherwise be able to afford; * Spotlighting, recognizing, and rewarding individual faculty, departments, and institutions that develop high-quality educational programs in science, engineering, and technology, thereby encouraging and attracting similar recognition and support from others; * Stimulating, encouraging, and enabling faculty members to pursue their interests in improving undergraduate laboratory curricula, and retaining and reinvigorating where necessary their enthusiasm for undergraduate teaching; * Encouraging institutions and departments to provide increased recognition and reward for faculty accomplishments in improving undergraduate science education; and * Helping undergraduate science, mathematics, engineering, and technology courses to become more up to date, attractive, understandable, stimulating, and relevant to modern life, thereby generating increased student interest and enrollment in science- and technology-related courses and degree programs. In addition, the ILI program has from its inception sought to do more than support improvements in the quality of instruction at individual institutions. It has gone beyond this with an interest in innovative curriculum development projects that show promise for improving the quality of undergraduate science, mathematics, engineering, and technology education nationwide. According to a recent DUE program announcement (NSF 96-10), “The ILI program seeks projects that will improve laboratory instruction nationally as well as at the project site and that will produce and adapt national models for the improvement of undergraduate laboratory instruction.” Given this emphasis, the program actively discourages proposals that request equipment solely on the basis of an institution's peculiar financial need or increased enrollments. Furthermore, throughout its existence ILI has particularly encouraged proposals that involve provisions for equipment sharing through consortia or centers, or that would strengthen undergraduate education by increasing the participation of women, underrepresented minorities, and persons with disabilities, especially if the projects present models for increasing the numbers who successfully pursue careers in science, mathematics, engineering, and technology. Table of Contents ------------------------------------------------------------------------ SIGNS OF PROGRESS ------------------------------------------------------------------------ What has been discovered so far about the effectiveness of its ILI program? What patterns have emerged over the years that shed light on the institutions, individuals, and scientific disciplines benefiting from it? And what issues must it try to resolve as the program enters its second decade? Presented below is an initial overview of findings. The main sources of information are an extensive evaluation of the program that was conducted in 1988-89 (3 years after ILI’s inception); an analysis of ILI records covering the program’s first 10 years (1985-94); and preliminary findings from a series of site visits to grantee institutions and interviews with individual awardees. Table of Contents ------------------------------------------------------------------------ The Early Years ------------------------------------------------------------------------ The 1988-89 evaluation of ILI’s first 3 years—relying on data from mail surveys of grantees and from site visits—revealed the following observations: * Projects afforded students previously unavailable opportunities for direct, hands-on learning that significantly stimulated, enriched, and enlivened their undergraduate experiences. In many cases, ILI-supported projects were viewed as having produced increased department enrollments, increased student interest in science and technology careers, and increased postcollege employment opportunities for students in science-related fields. * ILI-supported projects consistently had substantial positive effects on faculty and departments, both by providing a prestigious form of recognition and by encouraging and enabling faculty to pursue their ideas for updating and improving undergraduate curricula. As well as having beneficial effects on individual faculty members' careers, many of these projects were said to be responsible for reinvigorating faculty morale and enthusiasm for teaching. * From the initial impetus provided by ILI grants, many projects had attracted additional financial and other support enabling them to obtain additional instructional equipment and to grow in scope well beyond what had initially been contemplated. (On average, 1985-87 projects attracted total financial support at least 4.5 times the size of the NSF grant amount.) A particularly telling indication of the program's value was yielded by a hypothetical question put by evaluators to the For 85 percent of awardees, without NSF support projects would have been significantly scaled back or not implemented at all. grantees being surveyed. What, they were asked, would have happened to the projects they envisioned if NSF funding had been denied?For 85 percent of the awardees, the grants were of critical importance; without the Foundation's support, they said, their projects would either have been significantly scaled back or not implemented at all. Commenting at the time of the 1988-89 evaluation, Thomas Tucker, a Colgate University mathematics professor, reflected the enthusiasm often expressed by grantees. He cited a Colgate project supported by an ILI grant as an important factor in the Hamilton, New York, institution’s receiving a $100,000 state grant to interest inner-city youth in science. Tucker said: “The program is now a flagship. It would be difficult to imagine a project more successful than this one.” Table of Contents ------------------------------------------------------------------------ Success in Reaching Target Audiences ------------------------------------------------------------------------ In the initial phases of the 1994-97 evaluation, ILI proposal and award data residing in NSF’s mainframe database were compiled for the program's first 10 years (1985-94). These data reveal an abundance of information about the program's effectiveness thus far in reaching its target audiences of institutions, disciplines, students, faculty members, and others. Table of Contents ------------------------------------------------------------------------ Institution Level¹ ------------------------------------------------------------------------ Approximately 1,500 colleges and universities in the United States award bachelor’s degrees in NSF-supported fields (science, mathematics, engineering, and technology). The great majority of them (79 percent) have sought support from ILI by submitting one or more proposals during the 1985-94 period (Chart 1). Nearly all doctorate-level colleges and universities (95 percent) have requested ILI support,² as have 88 percent of master’s-level institutions and 68 percent of bachelor’s-level colleges. The lowest rate of participation has been from 2-year institutions that award associate’s degrees in science-related fields: 40 percent of the 1,160 such institutions have submitted proposals to ILI during the 7-year period that they have been eligible to participate in the program. ----------------------- ¹In this discussion, institutions that award undergraduate degrees in NSF-supported fields are classified according to the highest degree they award in these fields: doctorate, master's, bachelor's, or associate's. All institutions in the first three categories are 4-year;all in the last category are 2- year.Return ²Most of the exceptions are seminaries or other institutions whose only science-related doctoral programs are in psychology or the social sciences.Return ------------------------------------------------------------------------ Chart 1. ILI coverage of institutions, by highest degree awarded in science, mathematics, engineering, and technology, 1985-94 [Image] *Base = 4-year colleges and institutions that awarded bachelor’s degrees in science, mathematics, engineering, or technology fields in 1993, and 2-year colleges that awarded associate’s degrees in these fields in 1993. ------------------------------------------------------------------------ Over half (61 percent) of all 4-year colleges and universities with undergraduate degree programs in NSF-supported fields have received one or more ILI awards, including 87 percent of doctorate-level institutions, 74 percent of master’s-level institutions, and 45 percent of bachelor’s-level institutions. Nineteen percent of relevant 2-year institutions have received ILI awards. This last figure, while low, indicates that nearly half (47 percent) of the 2-year institutions that have submitted one or more ILI proposals have been successful in winning one or more awards. Among institutions that participated in ILI during its first decade, most submitted multiple proposals. The most proposals submitted by any single institution was 126. The overall average was 10 proposals per institution, but this varied considerably by institution size and level, ranging from an average of 24 proposals per institution for doctorate-level institutions to 2.4 proposals per institution for 2-year, associate’s-level institutions. Most institutions that have won any ILI awards during the 1985-94 period have won multiple awards. The highest number of awards for any single institution was 36. The overall average was 4 awards per institution, generally spread across several disciplines. Doctorate-level and master’s-level institutions have been most successful (both averaging about 5.5 awards per institution), followed by bachelor’s-level institutions (3.4 awards per institution) and 2-year institutions (1.4 awards per institution). Table of Contents ------------------------------------------------------------------------ Students³ ------------------------------------------------------------------------ In general, institutions that serve relatively large numbers of undergraduate science, mathematics, engineering, and technology students have been most active and most successful in the ILI program. Consequently, program coverage statistics are even higher when expressed in terms of coverage of students than when calculated with institutions as the unit. The extent of knowledge about and interest in ILI in the academic community is indicated by the finding that 96 percent of all students who received bachelor’s degrees in 1993 in NSF-supported fields got their degrees from institutions that have requested ILI support at least once. Over 99 percent of undergraduate science majors at doctorate-level institutions were in this situation, as were almost as many students at master’s-level institutions (97 percent) and at bachelor’s-level institutions (82 percent). These findings indicate strong awareness of and interest in obtaining program improvement support from the ILI program among institutions that have bachelor’s degree programs in science-related fields. As for support actually delivered by ILI, nearly 9 of every 10 students (89 percent) who received bachelor’s degrees in NSF-supported fields in 1993 studied at institutions that have received one or more ILI awards (Chart 2). Undergraduate students at doctorate-level institutions were especially well-covered (95 percent). The coverage has been lower for students at 2-year institutions: 29 percent of associate’s degree recipients in science-related fields studied at institutions that have received one or more ILI awards. As indicated earlier, many of the larger institutions have won multiple ILI awards over the years. The result is that, overall, 61 percent of 1993 bachelor’s degree recipients in NSF-supported fields received their degrees from institutions that have won four or more ILI awards. ----------------------- ³In the following discussion, ILI coverage of the Nations's population of undergraduate students is measured in terms of coverage of students who receive undergraduate (bachelor's or associate's) degrees with majors in science, mathematics, engineering, or technology fields. This is not to imply that ILI projects are limited to courses for students with science-related majors, however. Many ILI projects involve lower-division courses that are open to nonscience students as well as to science majors.Return ------------------------------------------------------------------------ Chart 2. ILI coverage of undergraduate students in science, mathematics, engineering, and technology programs, 1985-94 [Image] Table of Contents ------------------------------------------------------------------------ Disciplines ------------------------------------------------------------------------ Throughout its history, the ILI program has distributed awards among disciplines in close approximation to the number of proposals received (Table 2). However, some NSF-supported fields have participated more extensively than others in the program and consequently have derived greater benefits. ------------------------------------------------------------------------ Table 2.Distribution of undergraduate degrees and ILI proposals and awards in NSF-supported fields: 1985-1994 Field Under-graduate ILI ILI Award degrees, 1993* proposals awards amount Total number 485,416 17,793 4,704 $158.6 M (Percent) Physical sciences** 4 35 38 34 Engineering/technology 23 25 19 24 Biological sciences 13 17 17 15 Computer science 7 7 6 9 Mathematics 3 5 7 8 Psychology 14 4 4 3 Social sciences 29 3 3 3 Other/interdisciplinary 7 4 5 6 *Includes bachelor’s and associate’s degrees in all science, mathematics, engineering, and technology disciplines. **Includes chemistry, physics, astronomy, and geosciences. NOTE: Percents may not add to 100 due to rounding. ------------------------------------------------------------------------ The physical sciences—chemistry, physics, astronomy, and the geosciences—collectively account for over one-third of all ILI proposals and awards, well above their 4-percent share of undergraduate degrees across all NSF-supported fields. This prominence is not surprising, however, given the highly instrumentation-intensive nature of these laboratory-based disciplines, and also considering that these disciplines provide large numbers of lower division service courses for nonscience majors, in addition to advanced courses for science majors. ILI supports both categories of courses, lower division as well as upper division. Within the physical sciences, chemistry projects have been most dominant in the program over the past decade (960 awards totaling more than $31 million), followed by physics/astronomy (585 awards totaling more than $15 million) and the geosciences (252 awards totaling nearly $8 million). At the other extreme, the social sciences and psychology collectively account for only 6-7 percent of ILI proposals and awards, far lower than their 43 percent share of science-related undergraduate degrees. Engineering/technology disciplines are prominent in ILI: they account for about one-quarter of ILI proposals and awards, which is consistent with their share of undergraduate degree productivity (23 percent). The biological and computer sciences and mathematics also account for significant shares of ILI proposals and awards, at levels commensurate with their undergraduate degree productivity (Table 2). Table of Contents ------------------------------------------------------------------------ Faculty At the individual level, ILI proposals and awards have been widely dispersed. The roughly 18,000 proposals that were submitted to ILI in its first decade of existence were submitted by nearly 12,000 different principal investigators (PIs). Considering that significant numbers of proposals are submitted in revised form after having been declined one or more times, these findings indicate that participation in ILI has been remarkably widespread. Awards have been even more widely distributed, with ILI’s 4,704 awards through 1994 being made to 4,305 different PIs—an average of 1.1 awards per grantee. (Only 9 percent of ILI grantees have won more than one award.) Table of Contents ------------------------------------------------------------------------ Other Coverage Dimensions ------------------------------------------------------------------------ Although the ILI program has achieved broad overall coverage of its target undergraduate institutions and students (especially those with or in bachelor’s-level programs) and has distributed program support widely across disciplines and individual faculty members, it has not been entirely successful in its special initiatives regarding consortia, women, and minorities. Consortia. In each year of the program, since its inception in 1985, the ILI program announcement has made a special point of encouraging proposals for equipment-sharing consortia or centers that would help make modern laboratory equipment more widely accessible to undergraduate students in a particular discipline or geographical area. So far, this has not happened. Of the nearly 18,000 proposals the Foundation has received in the ILI program, only 54 have involved proposed consortia. Of these, only 19 were funded, most at low budget levels: only 2 had awards over $50,000. Women. Sixteen percent of ILI grantees during the program’s first decade have been women. This figure is far below the percentages of women in the population or in the overall workforce. It compares favorably, however, with the percentages of women among the science college faculty who constitute the personnel pool from which ILI PIs are drawn. For example, in 1991-92, only 4 percent of full-time college teaching faculty in engineering were women, as were 7 percent of physics faculty, 9 percent of geosciences faculty, and 30 percent of sociology faculty.4 In the same fields, women have constituted 7 percent of ILI grantees in engineering, 8 percent in physics/ astronomy, 14 percent in geosciences, and 19 percent in the social sciences. In 3 of these 4 fields, women have been more highly represented in ILI than in the workforce from which ILI PIs were drawn. Given NSF's policy objective to support increased participation by women in science-related fields, it is also encouraging to note that the percentage of women grantees in ILI has exceeded the percentage of women proposers in each of the program's first 10 years, and the percentage of women grantees has grown steadily and substantially over this decade, doubling from 12 percent in 1985 to 24 percent in 1994. ----------------------- 4 All reported faculty population statistics are taken from the National Science Foundation report Women, Minorities, and Persons with Disabilities in Science and Engineering: 1994(NSF94-333).Return Table of Contents ----------------------- Underrepresented Minorities. African-Americans, Alaskan Natives, Hispanics, and Native Americans are also historically underrepresented in science-related fields and in the populations of undergraduate science faculty from which ILI PIs are drawn. These underrepresented minorities collectively accounted for about 20 percent of the Nation's civilian labor force in 1990, yet they constitute only 4 percent of the college/ university teaching faculty in three fields: engineering, physics, and geology. In sociology, the fourth field for which population figures are available, the minority percentage is higher: 12 percent. Regarding ILI proposals and participation, available statistics indicate: * 3.7 percent of the proposals received by ILI during its first decade involved PIs from underrepresented minorities; * 3.3 percent of all awards made by ILI during its first decade were to PIs from underrepresented minorities; * In comparison to institutions with lower minority enrollments, institutions with significant (over 15 percent) minority enrollments have been less active in ILI (fewer have submitted any proposals) and have received fewer awards (Table 3); these differences largely reflect the differences noted earlier between 2-year and 4-year institutions in ILI participation and support. 5 * The ILI program’s overall proposal funding rate is 26 percent, and the success rates for proposals from institutions with high minority enrollments range from 25 percent to 29 percent (see Table 3). ---------------------------------------------------------------------- Table 3. ILI participation and success rates for institutions with significant minority enrollment Institution Number of Proposals Proposal characteristic institutions ILI Awards funding rate Total, all institutions 2,665 62% 43% 26% Minority enrollment > 15% 897 52% 32% 26% Enrollment > 15% black 452 50% 28% 25% Enrollment > 15% Hispanic 243 54% 34% 29% Enrollment > 15% Native American 29 38% 21% 28% ---------------------------------------------------------------------- --------------------- 5 Well over half of the institutions with high minority enrollments are 2-year colleges, and their participation and support rates are comparable to those for other 2-year institutions. Return --------------------- * Historically black colleges and universities (HBCUs), which are perhaps most widely recognized as serving distinctly minority students, have fared well in ILI: 66 (74 percent) of the 89 HBCUs that award undergraduate degrees in science-related fields have applied for ILI grants, and 48 percent have received one or more grants; both of these figures are slightly above the program average. * There are nine tribal colleges (located in Native American tribal lands) that award undergraduate degrees in science-related disciplines. All are 2-year institutions. Three have submitted proposals to ILI; one has received an ILI award. * The representation of PIs from minority groups has grown, gradually but perceptibly, over the brief history of the ILI program, from 1985 and 1986 when 1 percent of all ILI grantees were from underrepresented minorities to 1992, 1993, and 1994, when 5 percent were from these groups. Table of Contents ------------------------------------------------------------------------ Summary ------------------------------------------------------------------------ The data analyzed thus far indicate that the ILI program has achieved excellent overall coverage of its target audiences, but that some gaps in the coverage exist for some demographic and institutional groups. Furthermore, there appears to be some unevenness in benefits, with some institutions being heavily impacted with multiple awards and others gaining little or no direct benefit from the program. ------------------------------------------------------------------------ Recent Feedback From Grantees ------------------------------------------------------------------------ In addition to the quantitative data referred to above, the current study also has obtained preliminary qualitative information about the impacts of several individual ILI projects. This information was gathered during the course of 1995 site visits to six ILI recipient institutions that were selected essentially at random and through telephone conversations with other PIs who were identified by NSF program staff as having benefited from ILI in the past. The projects discussed with this small group of PIs proved to be highly diverse; they involved different fields and different institution settings, and they have developed in different directions. Within this small sampling of projects, some impressive examples of several different types of accomplishments were found. As such, these projects provide concrete illustrations of some of the kinds of impacts the ILI program has produced, although the number of cases studied is far too small to draw any conclusions about the relative prevalence or cumulative magnitudes of any single type of impact. More will be known on these subjects after the upcoming components of the evaluation have been completed. Table of Contents ------------------------------------------------------------------------ Resource Investment ------------------------------------------------------------------------ Aside from the matching funds provided by a college or university, several of the ILI project PIs visited by program evaluators mentioned that additional financial investments were generated as a result of their ILI awards. The winning of an ILI grant from NSF is apparently seen as a sign of quality assurance by some private and industrial funding sources that need to verify that programs they have been asked to support are truly meritorious. The peer-review process for ILI awards is presumably regarded by these potential donors as credible and rigorous, with only the highest quality programs receiving ILI support. Thus, with an ILI grant in hand, a PI or institution may be in a position to leverage that funding well beyond the amount awarded by NSF. For example: * A $45,000 1990 ILI grant at Camden County College in New Jersey under the direction of PI James Hudgings, a professor in the department of engineering technology, supported the acquisition of a distributed numerical control system integrating computer-aided design with computer-aided manufacturing (CIM). This ILI project eventually generated $300,000 in state money for laboratory improvement associated with the upgrading of the college's CIM program, $1.5 million for basic CIM equipment, and $4.2 million for upgrading the college's CIM research facility. * At the University of Delaware in Newark, Murray Johnston's $48,000 ILI grant in 1992 allowed the chemistry department to acquire several sophisticated measurement instruments for use in a novel series of independent study courses involving instrumental analysis. An unanticipated benefit for Johnston's plan was that by leveraging the $45,000 his project gained through ILI funding, he was able to get price concessions from the instrument manufacturer totaling about $85,000. * Four different ILI awards to support physics professor Priscilla Laws' innovative courses at Dickinson College have over the years helped generate approximately $10 million in curriculum development and faculty workshop grants from 10 different government agencies, private foundations, and commercial organizations. Table of Contents ------------------------------------------------------------------------ Educational Impacts ------------------------------------------------------------------------ The immediate, direct effect that an ILI award has on an institution is that it enables faculty to acquire instructional equipment they could not have gotten from the university or outside donors alone. This equipment then provides the building blocks with which faculty create new laboratory exercises and experiments, new opportunities for student research, new or expanded courses, and in some cases, new programs of study. Educational impacts of these kinds were seen for all site-visited projects. Some examples: * Over the years, Gary Kader and other faculty in the Department of Mathematical Science at Appalachian State University in Boone, North Carolina, have received several ILI grants, which have enabled them to restructure many of their undergraduate courses. From introductory service courses for liberal arts students to advanced calculus and computer science courses, the bulk of the department's offerings are now being taught in computer labs using equipment and software funded through ILI and assembled as ILI projects. One of these courses, first offered in 2 sections per semester, has now grown to 10-12 sections per semester and has become part of the university's core curriculum. * At the University of Southern Mississippi in Hattiesburg, a new laboratory course in molecular biology was created to accompany an existing lecture course, enabled by a 1990 ILI grant for purchase of requisite equipment. The new course has become so popular, says PI Glen Shearer, a professor in the biological sciences department, that unfortunately some students have been turned away. * At Illinois Benedictine College in Lisle, enhancement of inorganic chemistry and biochemistry courses was fostered by the ILI-supported acquisition of a sophisticated spectrophotometer. According to PIs Edward Ferroni and David Sonneburg of the college's chemistry department, the device has stimulated major revisions in the courses they teach—improvements that would have been impossible without the new machine. Table of Contents Dissemination ------------------------------------------------------------------------ The positive influence of an ILI program often extends beyond the institution receiving an award. NSF encourages the dissemination of instructional materials by PIs who have successfully implemented an ILI-seeded laboratory project, and many awardees are eager to share their experiences and acquired knowledge with others in their field. Presentations, publications, lectures, workshops, and seminars are the most common means for PIs to disseminate information about their projects. Notably, they are taking increasing advantage of electronic media to enhance their dissemination efforts—using the Internet, for example, to distribute lab manuals and other materials. The approaches that several awardees have taken in sharing information about their project-developed products and insights are especially impressive. For example: A project at Lawrence University in Appleton, Wisconsin, was designed to determine the staffing and hardware requirements for meaningful instruction in lasers and modern optics. John Brandenberger, the PI for the project, developed a wealth of dissemination materials including two monographs, a journal article, a series of workshops, and an assortment of hand-out documents ranging from copies of magazines featuring the laser laboratory to reprints of newspaper articles. The major dissemination vehicle was the document Lasers and Modern Optics in Undergraduate Physics, which was distributed free to nearly 700 physics departments throughout the United States. It includes a description of the space, staffing, and equipment requirements for a productive laser laboratory, plus a description of three courses in laser physics—everything one would need to know for setting up a dynamic undergraduate laser physics laboratory and curriculum. * Dickinson College's Priscilla Laws makes an average of 30 trips a year related to the subjects of her ILI grants. In addition to writing articles, giving conference talks, leading workshops, and consulting with institutions and instructors, she networks with Tufts University and the University of Oregon on an ongoing basis concerning further development of Real-Time Physics—a series of PC-based student laboratory guides and other tools for interactive teaching—which she and others developed as an outgrowth of a 1991 ILI project. * Cutting-edge communications are also being used for dissemination purposes. A successful ILI project enabling advanced work in digital imaging by Lawrence Marschall at Pennsylvania's Gettysburg College, for example, has led to the electronic distribution of laboratory modules—free of charge to the educational community nationwide—that can guide other instructors in making introductory astronomy labs more attractive to students. Table of Contents ---------------------------------------------------------------------- Research Gains ---------------------------------------------------------------------- The availability of advanced laboratory instrumentation acquired through ILI has clearly enriched undergraduate research activity at some institutions. And at some schools—such as the University of Southern Mississippi and Illinois Benedictine College—the presence of advanced laboratory equipment has attracted talented new faculty. At Southern Mississippi, PI Glen Shearer allows his ILI-funded equipment to be used by students and faculty from throughout the state for various types of research projects. And at the University of Delaware, as many as 18 faculty members were found to be using the ILI-funded mathematics laboratory on a regular basis. Table of Contents ---------------------------------------------------------------------- Some Grantees Voice Concerns Grantees interviewed by the evaluation team shared in enthusiastic agreement that the ILI program is extremely important and effective. Indeed, the general opinion is that without the NSF imprimatur—expressed in the form of an ILI award—many science, mathematics, engineering, and technology educators today would have serious problems obtaining the laboratory equipment they need in order to improve their courses. Nevertheless, PIs did raise a number of concerns and offered recommendations on how the program could be improved. For example: * The most common recommendation aims at diminishing the heavy time burden placed on PIs during the startup phase of their projects. To alleviate this burden, PIs would like to see their teaching load reduced while they are installing their new equipment and otherwise launching their projects. This could be achieved, many of them feel, if NSF would either—as part of an award—provide funds for temporary teaching substitutes, or—as a condition of an award—require grantee institutions to provide for this need. * PIs would like to see a supply budget included in an ILI grant to help maintain equipment and purchase necessary supplies. (This is an especially serious matter at small schools that have no staff on site to provide computer support or to repair equipment.) * To improve the diffusion of information generated from ILI projects, it is suggested that NSF take greater advantage of electronic communication media, such as the Internet. These and other issues are being explored by NSF on a broad scale as the Foundation moves forward with its comprehensive 1995-96 evaluation of the ILI program. Table of Contents ---------------------------------------------------------------------- LOOKING AHEAD ---------------------------------------------------------------------- The full-scale ILI study now under way—the second multiyear evaluation that the program has undergone since its inception in 1985—is based on the understanding that the program has matured and grown substantially in size and scope over the years and the time has come for NSF to take a fresh look at it. ILI has become a well-established, well-known program that has funded many projects over the past decade. Many of these projects, begun a number of years ago, have now had time to produce real impacts at the recipient institutions and in the larger educational culture. This reevaluation intends to assemble information about ILI's activities and accomplishments during its first decade of operations and about the impacts it has had on undergraduate education. It will also seek to uncover significant problems or limitations that should be addressed as ILI enters its second decade. Additionally, it will seek to identify a small number of useful indicators that can be collected on a regular basis from future ILI grantees, enabling NSF to track significant program outcomes and impacts between large-scale evaluations such as this. The main data collection for the evaluation will occur during the 1995-96 academic year. The evaluation is being funded jointly by NSF's Division of Research, Evaluation and Communication and the Division of Undergraduate Education. Both divisions are branches of the Foundation’s Directorate for Education and Human Resources. The outside contractor for the evaluation is Westat, Inc., a Rockville, Maryland, research and consulting firm that has conducted many other Federal program evaluations and research studies in the postsecondary arena. A six-person panel of academic scientists and administrators who are knowledgeable about the ILI program has been appointed to advise Westat in the design and conduct of the project and in the assessment of the findings. These advisors are as follows: * Biology: Dr. Eric Thomas, Vice President of Academic Affairs, Mississippi Valley State University, Itta Bena, Mississippi. * Chemistry: Dr. Thomas C. Farrar, Professor of Physical Chemistry, University of Wisconsin, Madison, Wisconsin. * Engineering: Dr. Eleanor Baum, Dean of Engineering, Cooper Union, New York, New York. * Engineering Technology: George A. Timblin, Head, Department of Engineering and Advanced Technology, Central Piedmont Community College, Charlotte, North Carolina. * Mathematics: Dr. Anita Solow, Professor of Mathematics, Grinnell College, Grinnell, Iowa. * Physics: Dr. James Parks, Director of Undergraduate Laboratories, University of Tennessee, Knoxville, Tennessee. Table of Contents ---------------------------------------------------------------------- Evaluation Objectives ---------------------------------------------------------------------- The general purpose of the evaluation is to assess how successful the ILI program has been in achieving its objectives over the past decade overall and how well it is meeting the needs of specific categories of undergraduate institutions (that is, 2-year; 4-year associate’s-, bachelor’s-, master’s-, or doctorate-granting). It also is designed to assess how effectively the program is meeting the needs of individual science, mathematics, engineering, and technology disciplines, and whether it is succeeding in reaching its various target audiences. The findings presented in this report are only the first to emerge in this new effort. As it progresses, the evaluation will attempt to document ways in which the program has been successful; at the same time, it will look for ways in which ILI might be strengthened or improved. The evaluation design centers around 10 broad questions, some of which have been addressed in this report and others that have yet to be addressed. 1) What is the scope and coverage of ILI? How has the program grown over its 10 years of operation, in terms of annual and cumulative numbers of proposals received and awards made? To what extent has the program provided support across the full spectrum of ILI-eligible institutions and individuals? 2) How much instructional resource investment has ILI stimulated? To what extent has ILI seed money attracted additional financial support from non-NSF sources for undergraduate instructional laboratory equipment and facilities? 3) What, if any, have been the educational and research impacts of ILI at grantee institutions? How has ILI stimulated the development of new or improved laboratory courses or other kinds of expanded learning opportunities for undergraduate students? To what extent has program-supported equipment been used for undergraduate or other research? 4) What, if any, have been the educational impacts of ILI on the larger culture, beyond the grantee institutions? To what extent has the program spurred the development of new instructional approaches or materials that have become widely adopted outside the grantee institutions? In what fields, and through what dissemination mechanisms, have the largest effects occurred? 5) Has ILI impacted faculty and faculty-reward systems at grantee institutions? To what degree has the program affected faculty members who have been involved in ILI projects, in terms of improved morale, instructional effectiveness, and capacity to update teaching approaches? 6) Has ILI impacted curriculum and career development among unsuccessful applicants? How often has applying for a program grant, even if the proposal was declined, stimulated investigators to develop and find support for worthwhile curriculum improvement projects that eventually became implemented without ILI support? 7) Has ILI impacted underserved groups? How successful has the program been in attracting proposals from PIs who are women, members of underrepresented minorities, or persons with disabilities? How well represented are members of these groups among ILI awardees and students impacted by ILI projects? 8) Has ILI impacted K-12 students and teachers? What effect has the program had on the training of primary and secondary school science and mathematics teachers? 9) What, if any, commercial, community, or other additional impacts has ILI had? Have individual ILI projects provided beneficial services or established mutually beneficial associations with local (or perhaps national or international) governmental, educational, industrial, research, or other organizations. 10) How could the program be improved? Has the program failed to achieve important objectives in some areas? Are there aspects of the program policy, structure, or management that have been ineffective, inadequate, or otherwise problematic? Are there changes that should be considered in the way the program is structured or operated? To address all of these questions, the 1995-96 evaluation comprises five principal components: 1) A thorough examination of existing NSF management information systems data concerning ILI proposals and awards during the first decade of the program (1985-94). Other existing data sources are also being examined to assess ILI's success in reaching its target audiences. 2) A mail survey of ILI grantees. The survey includes all grantees from the 1990 and 1992 cycles of the program. These two program cycles were selected as being generally typical of the numbers and kinds of ILI projects that have been supported throughout the post-CSIP period and as being of an optimal age for impact evaluation. The survey questionnaire collects a variety of basic information about the size, structure, and current status of a grantee's project; about the numbers and characteristics of involved students and faculty; and about the project's impacts and products to date, both within and beyond the grantee institution. 3) A concurrent survey of samples of unsuccessful applicants from the 1990 and 1992 programs (250 from each year) to obtain their views about the program and its impacts on them. 4) Supplemental site visits to approximately 25 institutions, following the mail surveys to validate the questionnaire data and assemble additional documentation and insights about selected instances of especially large ILI-related educational impacts. 5) Tracer studies of 10-20 projects that are identified by NSF program staff or by questionnaire responses as involving extensive external dissemination of procedures and products developed as an outgrowth of ILI projects. The PIs of these exemplary, high-impact projects are being interviewed to learn more about the dissemination activities that have occurred and about the surrounding circumstances and support systems. In addition, selected second-stage recipients of project-disseminated products and information will be contacted to learn how the materials have affected their own undergraduate teaching, thereby documenting in some measure the kinds of national-level impacts the ILI program has had on undergraduate instruction in science, mathematics, engineering, and technology. The first of these five components has been completed, with the results summarized earlier in this report. The survey collection activities (components 2 and 3) were conducted during the fall and winter of the 1995-96 academic year, and the supplemental data (components 4 and 5) are being collected in the spring and summer of 1996. Once these data have been collected and analyzed, a report summarizing the evaluation study findings and recommendations will be prepared. The Foundation provides awards for research in the sciences and engineering. The awardee is wholly responsible for the conduct of such research and preparation of the results for publication. The Foundation, therefore, does not assume responsibility for the research findings or their interpretation. The Foundation welcomes proposals from all qualified scientists and engineers and strongly encourages women, minorities, and persons with disabilities to compete fully in any of the research related programs described here. In accordance with federal statutes, regulations, and NSF policies, no person on grounds of race, color, age, sex, national origin, or disability shall be excluded from participation in, be denied the benefits of, or be subject to discrimination under any program or activity receiving financial assistance from the National Science Foundation. Facilitation Awards for Scientists and Engineers with Disabilities (FASED) provide funding for special assistance or equipment to enable persons with disabilities (investigators and other staff, including student research assistants) to work on NSF projects. See the program announcement or contact the program coordinator at (703) 306-1636. The National Science Foundation has TDD (Telephonic Device for the Deaf) capability, which enables individuals with hearing impairment to communicate with the Foundation about NSF programs, employment, or general information. To access NSF TDD dial (703) 306-0090; for FIRS, 1-800-877-8339. ---------------------------------------------------------------------- Table of Contents