A Short-Term Impact Study
of the
National Science Foundation’s



Instrumentation and

Laboratory Improvement

Program


Table of Contents

Prepared under Contract
RED 94-52966

by

Kenneth Burgdorf
Tom Ewing
Michael Feinstein
Paul Tuss
Stephen Roey
Diane Ward
Westat, Inc.

with the assistance of

Phillip Tavel
NSF

NSF Program Officers

Mary Sladek, Division of Research, Evaluation & Communication
Duncan McBride, Division of Undergraduate Education


Directorate for Education
and Human Resources
Division of Research
Evaluation & Communication

NOTE: The views expressed in this document are those of Westat, Inc. and do not necessarily reflect the views of the
National Science Foundation.

October 1996


Table of Contents

Executive Summary

  1. The ILI Program
  2. Signs Of Progress
  3. Looking Ahead


EXECUTIVE SUMMARY


The Directorate for Education and Human Resources (EHR) of the National Science Foundation (NSF), which periodically assesses all of its programs, is currently in the process of reviewing its Instrumentation and Laboratory Improvement (ILI) program. Under the direction of EHR’s Division of Research, Evaluation and Communication (REC) and Division of Undergraduate Education (DUE), the current study is scheduled for completion by the end of 1996. This three-section report is an initial product of the evaluation now in progress.

The first section(The ILI Program) briefly describes the history, objectives, and structure of this annual, peer-reviewed competition, which provides funds for the purchase of laboratory equipment that will be used in projects aimed at improving undergraduate science, mathematics, engineering, and technology curricula.

The second section(Signs of Progress) summarizes what the Foundation has learned thus far about the program’s success and shortcomings since its establishment in 1985. Information in this section is based on previous evaluations, the analysis of existing NSF record systems, and direct contacts with selected ILI grantee institutions as well as individual Principal Investigators (PIs) whose projects have incorporated laboratory equipment acquired under an ILI grant.

Among the previous evaluations referred to in this section is a 1988-89 ILI study that was based on mail surveys of grantees and site visits to recipient institutions. This earlier evaluation revealed, for example, that in the first 3 years of ILI’s existence, projects funded by the program had provided students with previously unattainable opportunities for beneficial “hands-on” learning experiences; that ILI-supported projects were spurring progress among faculty members in improving their undergraduate science and mathematics curricula; and that many recipient institutions were successfully parlaying their financial and material support from NSF into broader recognition and support from other outside sources.

Preliminary findings presented in the current evaluation--which focuses to a great extent on grantees from 1990 and 1992--affirm many of the positive conclusions reached in the earlier study, but go beyond those findings to answer a number of other very important evaluation questions. For example, the more recent findings presented here indicate that ILI has been generally successful in reaching its target audience; that many students have pursued advanced science/technology studies or careers following experiences in ILI-enabled laboratories; and that the presence of ILI-funded laboratory equipment has attracted new faculty at some institutions.

At the same time, preliminary findings of the current study suggest some areas where further refinements of the program may be warranted--for example, to address gaps or unevenness of coverage, to reduce time burdens or delays sometimes associated with project start-up, or to facilitate more widespread dissemination of ILI-initiated innovations.

The third and concluding section of this report (Looking Ahead) discusses the approach that REC is taking to further evaluate ILI’s impacts and to identify specific steps that might be taken to enhance the program.

Table of Contents

THE ILI PROGRAM


Thanks largely to funding support from the National Science Foundation (NSF), Priscilla Laws, a physics faculty member at Dickinson College in Carlisle, Pennsylvania, was able to acquire a sophisticated microcomputer system for use in her innovative work to improve the teaching of undergraduate physics.

Another NSF grantee—chemist Benjamin DeGraff, a professor at James Madison University in Harrisonburg, Virginia—teaches challenging courses on the use of lasers and advanced measuring devices to perform thermal analysis. The courses depend heavily on the presence in his laboratory of costly state-of-the-art laser units and measuring devices. The support he has received from the Foundation has made acquisition of these instruments possible.

Likewise, NSF funding has allowed mathematician Eugene Herman of Grinnell College in Iowa to purchase the powerful software he needs to nurture his students' insights into linear algebra, differential equations, and calculus by representing mathematical ideas and objects in onscreen forms that permit manipulation and experimentation.

Despite the diversity of their disciplines and pedagogical pursuits, Drs. Laws, DeGraff, and Herman have in common that they and their institutions are all beneficiaries of NSF's Instrumentation and Laboratory Improvement (ILI) program. ILI is an annual, peer-reviewed grant competition that provides funds for curriculum improvement through the purchase of laboratory equipment for use in projects aimed at improving undergraduate education in science, mathematics, engineering, and technology. ILI grants are in the $5,000-$100,000 range and must be matched with nonfederal funds; the grant and matching funds must be used solely for purchase of instructional laboratory equipment.

ILI grants are awarded for projects that show promise of advancing the state of the technology in a given field: ILI projects are expected to produce models for the improvement of undergraduate laboratory instruction nationally, as well as at the project site.

ILI has become one of the Foundation's most popular programs, attracting more than 2,000 proposals annually and awarding some $23 million a year to successful applicants.

In its first decade, from the program’s inception in 1985 through 1994, ILI has considered nearly 18,000 proposals from nearly 1,800 different institutions (Table 1). It has made a total of more than 4,700 grants to applicants from approximately 1,200 different institutions and has awarded more than $158 million. Since grantees are required to match ILI funding dollar for dollar, the program has stimulated investment of at least $316 million in the acquisition of undergraduate laboratory equipment.


Table 1. Cummulative ILI proposal and award data,
1985-94
Proposals Awards Award amount
($ million)
Total number/amount 17,793 4,704 $158.6
Number of institutions 1,777 1,185 --

Table of Contents

History of ILI in the NSF Portfolio


Established in 1985 as the College Science Instrumentation Program (CSIP), the ILI program bolsters the NSF mission of providing leadership for the Nation's efforts toward stimulating student interest in science, mathematics, engineering, and technology. Operated by the Division of Undergraduate Education (DUE)—a unit of NSF’s Directorate for Education and Human Resources (EHR)—ILI complements other programs aimed at enhancing the quality of instruction in institutions of higher education, including 2-year as well as 4-year colleges and universities.

In addition to ILI, DUE's program portfolio includes Course and Curriculum Development, a program aimed at improving the learning environment, content, and experience of instruction; Undergraduate Faculty Enhancement, which supports projects enabling faculty members to gain experience with technological advances and new experimental techniques in order to incorporate them effectively into undergraduate instruction; and Collaboratives for Excellence in Teacher Preparation, which supports large-scale efforts to achieve comprehensive change in the undergraduate education of future teachers and to increase substantially the quality and number of teachers well prepared in science and mathematics, especially members of underrepresented population groups; and Advanced Technological Education, a program (administered jointly with EHR’s Division of Elementary, Secondary and Informal Education) that promotes curriculum improvement and teacher development in undergraduate and secondary technological education through national and regional alliances between associate’s-degree-granting institutions and other secondary and postsecondary institutions, business, industry, and government.

During its first 3 years of operation (1985-87), ILI had annual budgets in the $5-$8 million range, with grantees—then restricted to non-doctorate-granting 4-year colleges and universities—receiving awards ranging from $5,000 to $50,000. Beginning in 1988, the program was given its current name and was expanded in several respects. Most notably, the range of eligible institutions was expanded to include 2-year institutions and 4-year doctorate-granting colleges and universities; and the upper limit of individual awards was increased to $100,000. Between 1988 and 1994, the program was flat-funded at $23 million annually.

Table of Contents

Program Objectives


The principal objective of the ILI program is

In addition to its explicit, primary focus on laboratory curriculum innovation, ILI implicitly has several other program objectives. Many of these relate to the programs and personnel of the recipient institutions:

In addition, the ILI program has from its inception sought to do more than support improvements in the quality of instruction at individual institutions. It has gone beyond this with an interest in innovative curriculum development projects that show promise for improving the quality of undergraduate science, mathematics, engineering, and technology education nationwide.

According to a recent DUE program announcement (NSF 96-10), “The ILI program seeks projects that will improve laboratory instruction nationally as well as at the project site and that will produce and adapt national models for the improvement of undergraduate laboratory instruction.” Given this emphasis, the program actively discourages proposals that request equipment solely on the basis of an institution's peculiar financial need or increased enrollments.

Furthermore, throughout its existence ILI has particularly encouraged proposals that involve provisions for equipment sharing through consortia or centers, or that would strengthen undergraduate education by increasing the participation of women, underrepresented minorities, and persons with disabilities, especially if the projects present models for increasing the numbers who successfully pursue careers in science, mathematics, engineering, and technology.

Table of Contents

SIGNS OF PROGRESS


What has been discovered so far about the effectiveness of its ILI program? What patterns have emerged over the years that shed light on the institutions, individuals, and scientific disciplines benefiting from it? And what issues must it try to resolve as the program enters its second decade? Presented below is an initial overview of findings. The main sources of information are an extensive evaluation of the program that was conducted in 1988-89 (3 years after ILI’s inception); an analysis of ILI records covering the program’s first 10 years (1985-94); and preliminary findings from a series of site visits to grantee institutions and interviews with individual awardees.

Table of Contents

The Early Years


The 1988-89 evaluation of ILI’s first 3 years—relying on data from mail surveys of grantees and from site visits—revealed the following observations:

A particularly telling indication of the program's value was yielded by a hypothetical question put by evaluators to the For 85 percent of awardees, without NSF support projects would have been significantly scaled back or not implemented at all. grantees being surveyed. What, they were asked, would have happened to the projects they envisioned if NSF funding had been denied?For 85 percent of the awardees, the grants were of critical importance; without the Foundation's support, they said, their projects would either have been significantly scaled back or not implemented at all.

Commenting at the time of the 1988-89 evaluation, Thomas Tucker, a Colgate University mathematics professor, reflected the enthusiasm often expressed by grantees. He cited a Colgate project supported by an ILI grant as an important factor in the Hamilton, New York, institution’s receiving a $100,000 state grant to interest inner-city youth in science. Tucker said: “The program is now a flagship. It would be difficult to imagine a project more successful than this one.”

Table of Contents

Success in Reaching Target Audiences


In the initial phases of the 1994-97 evaluation, ILI proposal and award data residing in NSF’s mainframe database were compiled for the program's first 10 years (1985-94). These data reveal an abundance of information about the program's effectiveness thus far in reaching its target audiences of institutions, disciplines, students, faculty members, and others.
Table of Contents

Institution Level¹


Approximately 1,500 colleges and universities in the United States award bachelor’s degrees in NSF-supported fields (science, mathematics, engineering, and technology). The great majority of them (79 percent) have sought support from ILI by submitting one or more proposals during the 1985-94 period (Chart 1). Nearly all doctorate-level colleges and universities (95 percent) have requested ILI support as have 88 percent of master’s-level institutions and 68 percent of bachelor’s-level colleges. The lowest rate of participation has been from 2-year institutions that award associate’s degrees in science-related fields: 40 percent of the 1,160 such institutions have submitted proposals to ILI during the 7-year period that they have been eligible to participate in the program.


¹In this discussion, institutions that award undergraduate degrees in NSF-supported fields are classified according to the highest degree they award in these fields: doctorate, master's, bachelor's, or associate's. All institutions in the first three categories are 4-year;all in the last category are 2- year.Return
²Most of the exceptions are seminaries or other institutions whose only science-related doctoral programs are in psychology or the social sciences.Return


Chart 1. ILI coverage of institutions, by highest degree awarded in science, mathematics, engineering, and technology, 1985-94

Over half (61 percent) of all 4-year colleges and universities with undergraduate degree programs in NSF-supported fields have received one or more ILI awards, including 87 percent of doctorate-level institutions, 74 percent of master’s-level institutions, and 45 percent of bachelor’s-level institutions. Nineteen percent of relevant 2-year institutions have received ILI awards. This last figure, while low, indicates that nearly half (47 percent) of the 2-year institutions that have submitted one or more ILI proposals have been successful in winning one or more awards.

Among institutions that participated in ILI during its first decade, most submitted multiple proposals. The most proposals submitted by any single institution was 126. The overall average was 10 proposals per institution, but this varied considerably by institution size and level, ranging from an average of 24 proposals per institution for doctorate-level institutions to 2.4 proposals per institution for 2-year, associate’s-level institutions. Most institutions that have won any ILI awards during the 1985-94 period have won multiple awards. The highest number of awards for any single institution was 36. The overall average was 4 awards per institution, generally spread across several disciplines. Doctorate-level and master’s-level institutions have been most successful (both averaging about 5.5 awards per institution), followed by bachelor’s-level institutions (3.4 awards per institution) and 2-year institutions (1.4 awards per institution).

Table of Contents

Students³


In general, institutions that serve relatively large numbers of undergraduate science, mathematics, engineering, and technology students have been most active and most successful in the ILI program. Consequently, program coverage statistics are even higher when expressed in terms of coverage of students than when calculated with institutions as the unit. The extent of knowledge about and interest in ILI in the academic community is indicated by the finding that 96 percent of all students who received bachelor’s degrees in 1993 in NSF-supported fields got their degrees from institutions that have requested ILI support at least once. Over 99 percent of undergraduate science majors at doctorate-level institutions were in this situation, as were almost as many students at master’s-level institutions (97 percent) and at bachelor’s-level institutions (82 percent). These findings indicate strong awareness of and interest in obtaining program improvement support from the ILI program among institutions that have bachelor’s degree programs in science-related fields.

As for support actually delivered by ILI, nearly 9 of every 10 students (89 percent) who received bachelor’s degrees in NSF-supported fields in 1993 studied at institutions that have received one or more ILI awards (Chart 2). Undergraduate students at doctorate-level institutions were especially well-covered (95 percent). The coverage has been lower for students at 2-year institutions: 29 percent of associate’s degree recipients in science-related fields studied at institutions that have received one or more ILI awards.

As indicated earlier, many of the larger institutions have won multiple ILI awards over the years. The result is that, overall, 61 percent of 1993 bachelor’s degree recipients in NSF-supported fields received their degrees from institutions that have won four or more ILI awards.


³In the following discussion, ILI coverage of the Nations's population of undergraduate students is measured in terms of coverage of students who receive undergraduate (bachelor's or associate's) degrees with majors in science, mathematics, engineering, or technology fields. This is not to imply that ILI projects are limited to courses for students with science-related majors, however. Many ILI projects involve lower-division courses that are open to nonscience students as well as to science majors.Return
Chart 2. ILI coverage of undergraduate students in science, mathematics, engineering, and technology programs, 1985-94

Table of Contents

Disciplines


Throughout its history, the ILI program has distributed awards among disciplines in close approximation to the number of proposals received (Table 2). However, some NSF-supported fields have participated more extensively than others in the program and consequently have derived greater benefits.


Table 2.Distribution of undergraduate degrees and ILI proposals and awards in NSF-supported fields: 1985-1994
Field Under-graduate degrees, 1993* ILI proposals ILI awards Award amount
Total number 485,416 17,793 4,704 $158.6 M
(Percent)
Physical sciences** 4 35 38 34
Engineering/technology 23 25 19 24
Biological sciences 13 17 17 15
Computer science 7 7 6 9
Mathematics 3 5 7 8
Psychology 14 4 4 3
Social sciences 29 3 3 3
Other/interdisciplinary 7 4 5 6
*Includes bachelor’s and associate’s degrees in all science, mathematics, engineering, and technology disciplines.
**Includes chemistry, physics, astronomy, and geosciences.
NOTE: Percents may not add to 100 due to rounding.

The physical sciences—chemistry, physics, astronomy, and the geosciences—collectively account for over one-third of all ILI proposals and awards, well above their 4-percent share of undergraduate degrees across all NSF-supported fields. This prominence is not surprising, however, given the highly instrumentation-intensive nature of these laboratory-based disciplines, and also considering that these disciplines provide large numbers of lower division service courses for nonscience majors, in addition to advanced courses for science majors. ILI supports both categories of courses, lower division as well as upper division.

Within the physical sciences, chemistry projects have been most dominant in the program over the past decade (960 awards totaling more than $31 million), followed by physics/astronomy (585 awards totaling more than $15 million) and the geosciences (252 awards totaling nearly $8 million).

At the other extreme, the social sciences and psychology collectively account for only 6-7 percent of ILI proposals and awards, far lower than their 43 percent share of science-related undergraduate degrees.

Engineering/technology disciplines are prominent in ILI: they account for about one-quarter of ILI proposals and awards, which is consistent with their share of undergraduate degree productivity (23 percent). The biological and computer sciences and mathematics also account for significant shares of ILI proposals and awards, at levels commensurate with their undergraduate degree productivity (Table 2).

Table of Contents

Faculty

At the individual level, ILI proposals and awards have been widely dispersed. The roughly 18,000 proposals that were submitted to ILI in its first decade of existence were submitted by nearly 12,000 different principal investigators (PIs). Considering that significant numbers of proposals are submitted in revised form after having been declined one or more times, these findings indicate that participation in ILI has been remarkably widespread.

Awards have been even more widely distributed, with ILI’s 4,704 awards through 1994 being made to 4,305 different PIs—an average of 1.1 awards per grantee. (Only 9 percent of ILI grantees have won more than one award.)

Table of Contents

Other Coverage Dimensions


Although the ILI program has achieved broad overall coverage of its target undergraduate institutions and students (especially those with or in bachelor’s-level programs) and has distributed program support widely across disciplines and individual faculty members, it has not been entirely successful in its special initiatives regarding consortia, women, and minorities.

Consortia. In each year of the program, since its inception in 1985, the ILI program announcement has made a special point of encouraging proposals for equipment-sharing consortia or centers that would help make modern laboratory equipment more widely accessible to undergraduate students in a particular discipline or geographical area. So far, this has not happened. Of the nearly 18,000 proposals the Foundation has received in the ILI program, only 54 have involved proposed consortia. Of these, only 19 were funded, most at low budget levels: only 2 had awards over $50,000.

Women. Sixteen percent of ILI grantees during the program’s first decade have been women. This figure is far below the percentages of women in the population or in the overall workforce. It compares favorably, however, with the percentages of women among the science college faculty who constitute the personnel pool from which ILI PIs are drawn. For example, in 1991-92, only 4 percent of full-time college teaching faculty in engineering were women, as were 7 percent of physics faculty, 9 percent of geosciences faculty, and 30 percent of sociology faculty.4 In the same fields, women have constituted 7 percent of ILI grantees in engineering, 8 percent in physics/ astronomy, 14 percent in geosciences, and 19 percent in the social sciences. In 3 of these 4 fields, women have been more highly represented in ILI than in the workforce from which ILI PIs were drawn.

Given NSF's policy objective to support increased participation by women in science-related fields, it is also encouraging to note that the percentage of women grantees in ILI has exceeded the percentage of women proposers in each of the program's first 10 years, and the percentage of women grantees has grown steadily and substantially over this decade, doubling from 12 percent in 1985 to 24 percent in 1994.


4 All reported faculty population statistics are taken from the National Science Foundation report Women, Minorities, and Persons with Disabilities in Science and Engineering: 1994(NSF94-333).Return
Table of Contents

Underrepresented Minorities. African-Americans, Alaskan Natives, Hispanics, and Native Americans are also historically underrepresented in science-related fields and in the populations of undergraduate science faculty from which ILI PIs are drawn. These underrepresented minorities collectively accounted for about 20 percent of the Nation's civilian labor force in 1990, yet they constitute only 4 percent of the college/ university teaching faculty in three fields: engineering, physics, and geology. In sociology, the fourth field for which population figures are available, the minority percentage is higher: 12 percent.

Regarding ILI proposals and participation, available statistics indicate:

Table of Contents

Summary


The data analyzed thus far indicate that the ILI program has achieved excellent overall coverage of its target audiences, but that some gaps in the coverage exist for some demographic and institutional groups. Furthermore, there appears to be some unevenness in benefits, with some institutions being heavily impacted with multiple awards and others gaining little or no direct benefit from the program.


Recent Feedback From Grantees


In addition to the quantitative data referred to above, the current study also has obtained preliminary qualitative information about the impacts of several individual ILI projects. This information was gathered during the course of 1995 site visits to six ILI recipient institutions that were selected essentially at random and through telephone conversations with other PIs who were identified by NSF program staff as having benefited from ILI in the past.

The projects discussed with this small group of PIs proved to be highly diverse; they involved different fields and different institution settings, and they have developed in different directions. Within this small sampling of projects, some impressive examples of several different types of accomplishments were found. As such, these projects provide concrete illustrations of some of the kinds of impacts the ILI program has produced, although the number of cases studied is far too small to draw any conclusions about the relative prevalence or cumulative magnitudes of any single type of impact. More will be known on these subjects after the upcoming components of the evaluation have been completed.

Table of Contents

Resource Investment


Aside from the matching funds provided by a college or university, several of the ILI project PIs visited by program evaluators mentioned that additional financial investments were generated as a result of their ILI awards. The winning of an ILI grant from NSF is apparently seen as a sign of quality assurance by some private and industrial funding sources that need to verify that programs they have been asked to support are truly meritorious. The peer-review process for ILI awards is presumably regarded by these potential donors as credible and rigorous, with only the highest quality programs receiving ILI support. Thus, with an ILI grant in hand, a PI or institution may be in a position to leverage that funding well beyond the amount awarded by NSF. For example:

Table of Contents

Educational Impacts


The immediate, direct effect that an ILI award has on an institution is that it enables faculty to acquire instructional equipment they could not have gotten from the university or outside donors alone. This equipment then provides the building blocks with which faculty create new laboratory exercises and experiments, new opportunities for student research, new or expanded courses, and in some cases, new programs of study. Educational impacts of these kinds were seen for all site-visited projects. Some examples:

Table of Contents

Dissemination


The positive influence of an ILI program often extends beyond the institution receiving an award. NSF encourages the dissemination of instructional materials by PIs who have successfully implemented an ILI-seeded laboratory project, and many awardees are eager to share their experiences and acquired knowledge with others in their field.

Presentations, publications, lectures, workshops, and seminars are the most common means for PIs to disseminate information about their projects. Notably, they are taking increasing advantage of electronic media to enhance their dissemination efforts—using the Internet, for example, to distribute lab manuals and other materials. The approaches that several awardees have taken in sharing information about their project-developed products and insights are especially impressive. For example:

A project at Lawrence University in Appleton, Wisconsin, was designed to determine the staffing and hardware requirements for meaningful instruction in lasers and modern optics. John Brandenberger, the PI for the project, developed a wealth of dissemination materials including two monographs, a journal article, a series of workshops, and an assortment of hand-out documents ranging from copies of magazines featuring the laser laboratory to reprints of newspaper articles. The major dissemination vehicle was the document Lasers and Modern Optics in Undergraduate Physics, which was distributed free to nearly 700 physics departments throughout the United States. It includes a description of the space, staffing, and equipment requirements for a productive laser laboratory, plus a description of three courses in laser physics—everything one would need to know for setting up a dynamic undergraduate laser physics laboratory and curriculum.