Executive Summary
The Directorate for Education and Human Resources (EHR) of the National Science Foundation (NSF), which periodically assesses all of its programs, is currently in the process of reviewing its Instrumentation and Laboratory Improvement (ILI) program. Under the direction of EHR’s Division of Research, Evaluation and Communication (REC) and Division of Undergraduate Education (DUE), the current study is scheduled for completion by the end of 1996. This three-section report is an initial product of the evaluation now in progress.
The first section(The ILI Program) briefly describes the history, objectives, and structure of this annual, peer-reviewed competition, which provides funds for the purchase of laboratory equipment that will be used in projects aimed at improving undergraduate science, mathematics, engineering, and technology curricula.
The second section(Signs of Progress) summarizes what the Foundation has learned thus far about the program’s success and shortcomings since its establishment in 1985. Information in this section is based on previous evaluations, the analysis of existing NSF record systems, and direct contacts with selected ILI grantee institutions as well as individual Principal Investigators (PIs) whose projects have incorporated laboratory equipment acquired under an ILI grant.
Among the previous evaluations referred to in this section is a 1988-89 ILI study that was based on mail surveys of grantees and site visits to recipient institutions. This earlier evaluation revealed, for example, that in the first 3 years of ILI’s existence, projects funded by the program had provided students with previously unattainable opportunities for beneficial “hands-on” learning experiences; that ILI-supported projects were spurring progress among faculty members in improving their undergraduate science and mathematics curricula; and that many recipient institutions were successfully parlaying their financial and material support from NSF into broader recognition and support from other outside sources.
Preliminary findings presented in the current evaluation--which focuses to a great extent on grantees from 1990 and 1992--affirm many of the positive conclusions reached in the earlier study, but go beyond those findings to answer a number of other very important evaluation questions. For example, the more recent findings presented here indicate that ILI has been generally successful in reaching its target audience; that many students have pursued advanced science/technology studies or careers following experiences in ILI-enabled laboratories; and that the presence of ILI-funded laboratory equipment has attracted new faculty at some institutions.
At the same time, preliminary findings of the current study suggest some areas where further refinements of the program may be warranted--for example, to address gaps or unevenness of coverage, to reduce time burdens or delays sometimes associated with project start-up, or to facilitate more widespread dissemination of ILI-initiated innovations.
The third and concluding section of this report (Looking Ahead) discusses the approach that REC is taking to further evaluate ILI’s impacts and to identify specific steps that might be taken to enhance the program.
Table of ContentsThanks largely to funding support from the National Science Foundation (NSF), Priscilla Laws, a physics faculty member at Dickinson College in Carlisle, Pennsylvania, was able to acquire a sophisticated microcomputer system for use in her innovative work to improve the teaching of undergraduate physics.
Another NSF grantee—chemist Benjamin DeGraff, a professor at James Madison University in Harrisonburg, Virginia—teaches challenging courses on the use of lasers and advanced measuring devices to perform thermal analysis. The courses depend heavily on the presence in his laboratory of costly state-of-the-art laser units and measuring devices. The support he has received from the Foundation has made acquisition of these instruments possible.
Likewise, NSF funding has allowed mathematician Eugene Herman of Grinnell College in Iowa to purchase the powerful software he needs to nurture his students' insights into linear algebra, differential equations, and calculus by representing mathematical ideas and objects in onscreen forms that permit manipulation and experimentation.
Despite the diversity of their disciplines and pedagogical pursuits, Drs. Laws, DeGraff, and Herman have in common that they and their institutions are all beneficiaries of NSF's Instrumentation and Laboratory Improvement (ILI) program. ILI is an annual, peer-reviewed grant competition that provides funds for curriculum improvement through the purchase of laboratory equipment for use in projects aimed at improving undergraduate education in science, mathematics, engineering, and technology. ILI grants are in the $5,000-$100,000 range and must be matched with nonfederal funds; the grant and matching funds must be used solely for purchase of instructional laboratory equipment.
ILI grants are awarded for projects that show promise of advancing the state of the technology in a given field: ILI projects are expected to produce models for the improvement of undergraduate laboratory instruction nationally, as well as at the project site.
ILI has become one of the Foundation's most popular programs, attracting more than 2,000 proposals annually and awarding some $23 million a year to successful applicants.
In its first decade, from the program’s inception in 1985 through 1994, ILI has considered nearly 18,000 proposals from nearly 1,800 different institutions (Table 1). It has made a total of more than 4,700 grants to applicants from approximately 1,200 different institutions and has awarded more than $158 million. Since grantees are required to match ILI funding dollar for dollar, the program has stimulated investment of at least $316 million in the acquisition of undergraduate laboratory equipment.
| Proposals | Awards | Award amount ($ million) |
|
|---|---|---|---|
| Total number/amount | 17,793 | 4,704 | $158.6 |
| Number of institutions | 1,777 | 1,185 | -- |
Established in 1985 as the College Science Instrumentation Program (CSIP), the ILI program bolsters the NSF mission of providing leadership for the Nation's efforts toward stimulating student interest in science, mathematics, engineering, and technology. Operated by the Division of Undergraduate Education (DUE)—a unit of NSF’s Directorate for Education and Human Resources (EHR)—ILI complements other programs aimed at enhancing the quality of instruction in institutions of higher education, including 2-year as well as 4-year colleges and universities.
In addition to ILI, DUE's program portfolio includes Course and Curriculum Development, a program aimed at improving the learning environment, content, and experience of instruction; Undergraduate Faculty Enhancement, which supports projects enabling faculty members to gain experience with technological advances and new experimental techniques in order to incorporate them effectively into undergraduate instruction; and Collaboratives for Excellence in Teacher Preparation, which supports large-scale efforts to achieve comprehensive change in the undergraduate education of future teachers and to increase substantially the quality and number of teachers well prepared in science and mathematics, especially members of underrepresented population groups; and Advanced Technological Education, a program (administered jointly with EHR’s Division of Elementary, Secondary and Informal Education) that promotes curriculum improvement and teacher development in undergraduate and secondary technological education through national and regional alliances between associate’s-degree-granting institutions and other secondary and postsecondary institutions, business, industry, and government.
During its first 3 years of operation (1985-87), ILI had annual budgets in the $5-$8 million range, with grantees—then restricted to non-doctorate-granting 4-year colleges and universities—receiving awards ranging from $5,000 to $50,000. Beginning in 1988, the program was given its current name and was expanded in several respects. Most notably, the range of eligible institutions was expanded to include 2-year institutions and 4-year doctorate-granting colleges and universities; and the upper limit of individual awards was increased to $100,000. Between 1988 and 1994, the program was flat-funded at $23 million annually.
Table of Contents“...to support the development of experiments and laboratory curricula which improve the science, mathematics, engineering, and technology education of undergraduate students, both science majors and non science majors, including pre-service teachers.
Proposals are sought for the development of innovative methods for using laboratory exercises to improve student understanding of basic principles and for use of modern instrumentation, new technologies, or new applications for instruments that extend the instructional capability of the equipment” (1996 ILI Program Announcement; NSF 96-10).
In addition to its explicit, primary focus on laboratory curriculum innovation, ILI implicitly has several other program objectives. Many of these relate to the programs and personnel of the recipient institutions:
In addition, the ILI program has from its inception sought to do more than support improvements in the quality of instruction at individual institutions. It has gone beyond this with an interest in innovative curriculum development projects that show promise for improving the quality of undergraduate science, mathematics, engineering, and technology education nationwide.
According to a recent DUE program announcement (NSF 96-10), “The ILI program seeks projects that will improve laboratory instruction nationally as well as at the project site and that will produce and adapt national models for the improvement of undergraduate laboratory instruction.” Given this emphasis, the program actively discourages proposals that request equipment solely on the basis of an institution's peculiar financial need or increased enrollments.
Furthermore, throughout its existence ILI has particularly encouraged proposals that involve provisions for equipment sharing through consortia or centers, or that would strengthen undergraduate education by increasing the participation of women, underrepresented minorities, and persons with disabilities, especially if the projects present models for increasing the numbers who successfully pursue careers in science, mathematics, engineering, and technology.
Table of ContentsWhat has been discovered so far about the effectiveness of its ILI program? What patterns have emerged over the years that shed light on the institutions, individuals, and scientific disciplines benefiting from it? And what issues must it try to resolve as the program enters its second decade? Presented below is an initial overview of findings. The main sources of information are an extensive evaluation of the program that was conducted in 1988-89 (3 years after ILI’s inception); an analysis of ILI records covering the program’s first 10 years (1985-94); and preliminary findings from a series of site visits to grantee institutions and interviews with individual awardees.
Table of ContentsA particularly telling indication of the program's value was yielded by a hypothetical question put by evaluators to the
For 85 percent of awardees, without NSF support projects would have been significantly scaled back or not implemented at all.
grantees being surveyed. What, they were asked, would have happened to the projects they envisioned if NSF funding had been denied?For 85 percent of the awardees, the grants were of critical importance; without the Foundation's support, they said, their projects would either have been significantly scaled back or not implemented at all.
Commenting at the time of the 1988-89 evaluation, Thomas Tucker, a Colgate University mathematics professor, reflected the enthusiasm often expressed by grantees. He cited a Colgate project supported by an ILI grant as an important factor in the Hamilton, New York, institution’s receiving a $100,000 state grant to interest inner-city youth in science. Tucker said: “The program is now a flagship. It would be difficult to imagine a project more successful than this one.”
Table of ContentsApproximately 1,500 colleges and universities in the United States award bachelor’s degrees in NSF-supported fields (science, mathematics, engineering, and technology). The great majority of them (79 percent) have sought support from ILI by submitting one or more proposals during the 1985-94 period (Chart 1). Nearly all doctorate-level colleges and universities (95 percent) have requested ILI support,² as have 88 percent of master’s-level institutions and 68 percent of bachelor’s-level colleges. The lowest rate of participation has been from 2-year institutions that award associate’s degrees in science-related fields: 40 percent of the 1,160 such institutions have submitted proposals to ILI during the 7-year period that they have been eligible to participate in the program.
*Base = 4-year colleges and institutions that awarded bachelor’s degrees in science, mathematics, engineering, or technology fields in 1993, and 2-year colleges that awarded associate’s degrees in these fields in 1993.
Over half (61 percent) of all 4-year colleges and universities with undergraduate degree programs in NSF-supported fields have received one or more ILI awards, including 87 percent of doctorate-level institutions, 74 percent of master’s-level institutions, and 45 percent of bachelor’s-level institutions. Nineteen percent of relevant 2-year institutions have received ILI awards. This last figure, while low, indicates that nearly half (47 percent) of the 2-year institutions that have submitted one or more ILI proposals have been successful in winning one or more awards.
Among institutions that participated in ILI during its first decade, most submitted multiple proposals. The most proposals submitted by any single institution was 126. The overall average was 10 proposals per institution, but this varied considerably by institution size and level, ranging from an average of 24 proposals per institution for doctorate-level institutions to 2.4 proposals per institution for 2-year, associate’s-level institutions. Most institutions that have won any ILI awards during the 1985-94 period have won multiple awards. The highest number of awards for any single institution was 36. The overall average was 4 awards per institution, generally spread across several disciplines. Doctorate-level and master’s-level institutions have been most successful (both averaging about 5.5 awards per institution), followed by bachelor’s-level institutions (3.4 awards per institution) and 2-year institutions (1.4 awards per institution).
Table of ContentsIn general, institutions that serve relatively large numbers of undergraduate science, mathematics, engineering, and technology students have been most active and most successful in the ILI program. Consequently, program coverage statistics are even higher when expressed in terms of coverage of students than when calculated with institutions as the unit. The extent of 
knowledge about and interest in ILI in the academic community is indicated by the finding that 96 percent of all students who received bachelor’s degrees in 1993 in NSF-supported fields got their degrees from institutions that have requested ILI support at least once. Over 99 percent of undergraduate science majors at doctorate-level institutions were in this situation, as were almost as many students at master’s-level institutions (97 percent) and at bachelor’s-level institutions (82 percent). These findings indicate strong awareness of and interest in obtaining program improvement support from the ILI program among institutions that have bachelor’s degree programs in science-related fields.
As for support actually delivered by ILI, nearly 9 of every 10 students (89 percent) who received bachelor’s degrees in NSF-supported fields in 1993 studied at institutions that have received one or more ILI awards (Chart 2). Undergraduate students at doctorate-level institutions were especially well-covered (95 percent). The coverage has been lower for students at 2-year institutions: 29 percent of associate’s degree recipients in science-related fields studied at institutions that have received one or more ILI awards.
As indicated earlier, many of the larger institutions have won multiple ILI awards over the years. The result is that, overall, 61 percent of 1993 bachelor’s degree recipients in NSF-supported fields received their degrees from institutions that have won four or more ILI awards.
Throughout its history, the ILI program has distributed awards among disciplines in close approximation to the number of proposals received (Table 2). However, some NSF-supported fields have participated more extensively than others in the program and consequently have derived greater benefits.
| Field | Under-graduate degrees, 1993* | ILI proposals | ILI awards | Award amount |
|---|---|---|---|---|
| Total number | 485,416 | 17,793 | 4,704 | $158.6 M |
| (Percent) | ||||
| Physical sciences** | 4 | 35 | 38 | 34 |
| Engineering/technology | 23 | 25 | 19 | 24 |
| Biological sciences | 13 | 17 | 17 | 15 |
| Computer science | 7 | 7 | 6 | 9 |
| Mathematics | 3 | 5 | 7 | 8 |
| Psychology | 14 | 4 | 4 | 3 |
| Social sciences | 29 | 3 | 3 | 3 |
| Other/interdisciplinary | 7 | 4 | 5 | 6 |
The physical sciences—chemistry, physics, astronomy, and the geosciences—collectively account for over one-third of all ILI proposals and awards, well above their 4-percent share of undergraduate degrees across all NSF-supported fields. This prominence is not surprising, however, given the highly instrumentation-intensive nature of these laboratory-based disciplines, and also considering that these disciplines provide large numbers of lower division service courses for nonscience majors, in addition to advanced courses for science majors. ILI supports both categories of courses, lower division as well as upper division.
Within the physical sciences, chemistry projects have been most dominant in the program over the past decade (960 awards totaling more than $31 million), followed by physics/astronomy (585 awards totaling more than $15 million) and the geosciences (252 awards totaling nearly $8 million).
At the other extreme, the social sciences and psychology collectively account for only 6-7 percent of ILI proposals and awards, far lower than their 43 percent share of science-related undergraduate degrees.
Engineering/technology disciplines are prominent in ILI: they account for about one-quarter of ILI proposals and awards, which is consistent with their share of undergraduate degree productivity (23 percent). The biological and computer sciences and mathematics also account for significant shares of ILI proposals and awards, at levels commensurate with their undergraduate degree productivity (Table 2).
Table of ContentsAt the individual level, ILI proposals and awards have been widely dispersed. The roughly 18,000 proposals that were submitted to ILI in its first decade of existence were submitted by nearly 12,000 different principal investigators (PIs). Considering that significant numbers of proposals are submitted in revised form after having been declined one or more times, these findings indicate that participation in ILI has been remarkably widespread.
Awards have been even more widely distributed, with ILI’s 4,704 awards through 1994 being made to 4,305 different PIs—an average of 1.1 awards per grantee. (Only 9 percent of ILI grantees have won more than one award.)
Table of ContentsAlthough the ILI program has achieved broad overall coverage of its target undergraduate institutions and students (especially those with or in bachelor’s-level programs) and has distributed program support widely across disciplines and individual faculty members, it has not been entirely successful in its special initiatives regarding consortia, women, and minorities.
Consortia. In each year of the program, since its inception in 1985, the ILI program announcement has made a special point of encouraging proposals for equipment-sharing consortia or centers that would help make modern laboratory equipment more widely accessible to undergraduate students in a particular discipline or geographical area. So far, this has not happened. Of the nearly 18,000 proposals the Foundation has received in the ILI program, only 54 have involved proposed consortia. Of these, only 19 were funded, most at low budget levels: only 2 had awards over $50,000.Women. Sixteen percent of ILI grantees during the program’s first decade have been women. This figure is far below the percentages of women in the population or in the overall workforce. It compares favorably, however, with the percentages of women among the science college faculty who constitute the personnel pool from which ILI PIs are drawn. For example, in 1991-92, only 4 percent of full-time college teaching faculty in engineering were women, as were 7 percent of physics faculty, 9 percent of geosciences faculty, and 30 percent of sociology faculty.4 In the same fields, women have constituted 7 percent of ILI grantees in engineering, 8 percent in physics/ astronomy, 14 percent in geosciences, and 19 percent in the social sciences. In 3 of these 4 fields, women have been more highly represented in ILI than in the workforce from which ILI PIs were drawn.
Given NSF's policy objective to support increased participation by women in science-related fields, it is also encouraging to note that the percentage of women grantees in ILI has exceeded the percentage of women proposers in each of the program's first 10 years, and the percentage of women grantees has grown steadily and substantially over this decade, doubling from 12 percent in 1985 to 24 percent in 1994.
Regarding ILI proposals and participation, available statistics indicate:
| Institution characteristic | Number of institutions | Proposals ILI | Awards | Proposal funding rate |
|---|---|---|---|---|
| Total, all institutions | 2,665 | 62% | 43% | 26% |
| Minority enrollment > 15% |
897 | 52% | 32% | 26% |
| Enrollment > 15% black | 452 | 50% | 28% | 25% |
| Enrollment > 15% Hispanic | 243 | 54% | 34% | 29% |
| Enrollment > 15% Native American | 29 | 38% | 21% | 28% |
The data analyzed thus far indicate that the ILI program has achieved excellent overall coverage of its target audiences, but that some gaps in the coverage exist for some demographic and institutional groups. Furthermore, there appears to be some unevenness in benefits, with some institutions being heavily impacted with multiple awards and others gaining little or no direct benefit from the program.
In addition to the quantitative data referred to above, the current study also has obtained preliminary qualitative information about the impacts of several individual ILI projects. This information was gathered during the course of 1995 site visits to six ILI recipient institutions that were selected essentially at random and through telephone conversations with other PIs who were identified by NSF program staff as having benefited from ILI in the past.
The projects discussed with this small group of PIs proved to be highly diverse; they involved different fields and different institution settings, and they have developed in different directions. Within this small sampling of projects, some impressive examples of several different types of accomplishments were found. As such, these projects provide concrete illustrations of some of the kinds of impacts the ILI program has produced, although the number of cases studied is far too small to draw any conclusions about the relative prevalence or cumulative magnitudes of any single type of impact. More will be known on these subjects after the upcoming components of the evaluation have been completed.
Table of ContentsAside from the matching funds provided by a college or university, several of the ILI project PIs visited by program evaluators mentioned that additional financial investments were generated as a result of their ILI awards. The winning of an ILI grant from NSF is apparently seen as a sign of quality assurance by some private and industrial funding sources that need to verify that programs they have been asked to support are truly meritorious. The peer-review process for ILI awards is presumably regarded by these potential donors as credible and rigorous, with only the highest quality programs receiving ILI support. Thus, with an ILI grant in hand, a PI or institution may be in a position to leverage that funding well beyond the amount awarded by NSF. For example:
The immediate, direct effect that an ILI award has on an institution is that it enables faculty to acquire instructional equipment they could not have gotten from the university or outside donors alone. This equipment then provides the building blocks with which faculty create new laboratory exercises and experiments, new opportunities for student research, new or expanded courses, and in some cases, new programs of study. Educational impacts of these kinds were seen for all site-visited projects. Some examples:
The positive influence of an ILI program often extends beyond the institution receiving an award. NSF encourages the dissemination of instructional materials by PIs who have successfully implemented an ILI-seeded laboratory project, and many awardees are eager to share their experiences and acquired knowledge with others in their field.
Presentations, publications, lectures, workshops, and seminars are the most common means for PIs to disseminate information about their projects. Notably, they are taking increasing advantage of electronic media to enhance their dissemination efforts—using the Internet, for example, to distribute lab manuals and other materials. The approaches that several awardees have taken in sharing information about their project-developed products and insights are especially impressive. For example:
A project at Lawrence University in Appleton, Wisconsin, was designed to determine the staffing and hardware requirements for meaningful instruction in lasers and modern optics. John Brandenberger, the PI for the project, developed a wealth of 
dissemination materials including two monographs, a journal article, a series of workshops, and an assortment of hand-out documents ranging from copies of magazines featuring the laser laboratory to reprints of newspaper articles. The major dissemination vehicle was the document Lasers and Modern Optics in Undergraduate Physics, which was distributed free to nearly 700 physics departments throughout the United States. It includes a description of the space, staffing, and equipment requirements for a productive laser laboratory, plus a description of three courses in laser physics—everything one would need to know for setting up a dynamic undergraduate laser physics laboratory and curriculum.
The availability of advanced laboratory instrumentation acquired through ILI has clearly enriched undergraduate research activity at some institutions. And at some schools—such as the University of Southern Mississippi and Illinois Benedictine College—the presence of advanced laboratory equipment has attracted talented new faculty. At Southern Mississippi, PI Glen Shearer allows his ILI-funded equipment to be used by students and faculty from throughout the state for various types of research projects. And at the University of Delaware, as many as 18 faculty members were found to be using the ILI-funded mathematics laboratory on a regular basis.
Table of ContentsGrantees interviewed by the evaluation team shared in enthusiastic agreement that the ILI program is extremely important and effective. Indeed, the general opinion is that without the NSF imprimatur—expressed in the form of an ILI award—many science, mathematics, engineering, and technology educators today would have serious problems obtaining the laboratory equipment they need in order to improve their courses. Nevertheless, PIs did raise a number of concerns and offered recommendations on how the program could be improved. For example:
These and other issues are being explored by NSF on a broad scale as the Foundation moves forward with its comprehensive 1995-96 evaluation of the ILI program.
Table of ContentsThe full-scale ILI study now under way—the second multiyear evaluation that the program has undergone since its inception in 1985—is based on the understanding that the program has matured and grown substantially in size and scope over the years and the time has come for NSF to take a fresh look at it. ILI has become a well-established, well-known program that has funded many projects over the past decade. Many of these projects, begun a number of years ago, have now had time to produce real impacts at the recipient institutions and in the larger educational culture.
This reevaluation intends to assemble information about ILI's activities and accomplishments during its first decade of operations and about the impacts it has had on undergraduate education. It will also seek to uncover significant problems or limitations that should be addressed as ILI enters its second decade. Additionally, it will seek to identify a small number of useful indicators that can be collected on a regular basis from future ILI grantees, enabling NSF to track significant program outcomes and impacts between large-scale evaluations such as this.
The main data collection for the evaluation will occur during the 1995-96 academic year. The evaluation is being funded jointly by NSF's Division of Research, Evaluation and Communication and the Division of Undergraduate Education. Both divisions are branches of the Foundation’s Directorate for Education and Human Resources.
The outside contractor for the evaluation is Westat, Inc., a Rockville, Maryland, research and consulting firm that has conducted many other Federal program evaluations and research studies in the postsecondary arena. A six-person panel of academic scientists and administrators who are knowledgeable about the ILI program has been appointed to advise Westat in the design and conduct of the project and in the assessment of the findings. These advisors are as follows:
The general purpose of the evaluation is to assess how successful the ILI program has been in achieving its objectives over the past decade overall and how well it is meeting the needs of specific categories of undergraduate institutions (that is, 2-year; 4-year associate’s-, bachelor’s-, master’s-, or doctorate-granting). It also is designed to assess how effectively the program is meeting the needs of individual science, mathematics, engineering, and technology disciplines, and whether it is succeeding in reaching its various target audiences. The findings presented in this report are only the first to emerge in this new effort. As it progresses, the evaluation will attempt to document ways in which the program has been successful; at the same time, it will look for ways in which ILI might be strengthened or improved.
The evaluation design centers around 10 broad questions, some of which have been addressed in this report and others that have yet to be addressed.
1) What is the scope and coverage of ILI? How has the program grown over its 10 years of operation, in terms of annual and cumulative numbers of proposals received and awards made? To what extent has the program provided support across the full spectrum of ILI-eligible institutions and individuals?
2) How much instructional resource investment has ILI stimulated? To what extent has ILI seed money attracted additional financial support from non-NSF sources for undergraduate instructional laboratory equipment and facilities?
3) What, if any, have been the educational and research impacts of ILI at grantee institutions? How has ILI stimulated the development of new or improved laboratory courses or other kinds of expanded learning opportunities for undergraduate students? To what extent has program-supported equipment been used for undergraduate or other research?
4) What, if any, have been the educational impacts of ILI on the larger culture, beyond the grantee institutions? To what extent has the program spurred the development of new instructional approaches or materials that have become widely adopted outside the grantee institutions? In what fields, and through what dissemination mechanisms, have the largest effects occurred?
5) Has ILI impacted faculty and faculty-reward systems at grantee institutions? To what degree has the program affected faculty members who have been involved in ILI projects, in terms of improved morale, instructional effectiveness, and capacity to update teaching approaches?
6) Has ILI impacted curriculum and career development among unsuccessful applicants? How often has applying for a program grant, even if the proposal was declined, stimulated investigators to develop and find support for worthwhile curriculum improvement projects that eventually became implemented without ILI support?
7) Has ILI impacted underserved groups? How successful has the program been in attracting proposals from PIs who are women, members of underrepresented minorities, or persons with disabilities? How well represented are members of these groups among ILI awardees and students impacted by ILI projects?
8) Has ILI impacted K-12 students and teachers? What effect has the program had on the training of primary and secondary school science and mathematics teachers?
9) What, if any, commercial, community, or other additional impacts has ILI had? Have individual ILI projects provided beneficial services or established mutually beneficial associations with local (or perhaps national or international) governmental, educational, industrial, research, or other organizations.
10) How could the program be improved? Has the program failed to achieve important objectives in some areas? Are there aspects of the program policy, structure, or management that have been ineffective, inadequate, or otherwise problematic? Are there changes that should be considered in the way the program is structured or operated?
To address all of these questions, the 1995-96 evaluation comprises five principal components:
1) A thorough examination of existing NSF management information systems data concerning ILI proposals and awards during the first decade of the program (1985-94). Other existing data sources are also being examined to assess ILI's success in reaching its target audiences.
2) A mail survey of ILI grantees. The survey includes all grantees from the 1990 and 1992 cycles of the program. These two program cycles were selected as being generally typical of the numbers and kinds of ILI projects that have been supported throughout the post-CSIP period and as being of an optimal age for impact evaluation. The survey questionnaire collects a variety of basic information about the size, structure, and current status of a grantee's project; about the numbers and characteristics of involved students and faculty; and about the project's impacts and products to date, both within and beyond the grantee institution.
3) A concurrent survey of samples of unsuccessful applicants from the 1990 and 1992 programs (250 from each year) to obtain their views about the program and its impacts on them.
4) Supplemental site visits to approximately 25 institutions, following the mail surveys to validate the questionnaire data and assemble additional documentation and insights about selected instances of especially large ILI-related educational impacts.
5) Tracer studies of 10-20 projects that are identified by NSF program staff or by questionnaire responses as involving extensive external dissemination of procedures and products developed as an outgrowth of ILI projects. The PIs of these exemplary, high-impact projects are being interviewed to learn more about the dissemination activities that have occurred and about the surrounding circumstances and support systems. In addition, selected second-stage recipients of project-disseminated products and information will be contacted to learn how the materials have affected their own undergraduate teaching, thereby documenting in some measure the kinds of national-level impacts the ILI program has had on undergraduate instruction in science, mathematics, engineering, and technology.
The first of these five components has been completed, with the results summarized earlier in this report. The survey collection activities (components 2 and 3) were conducted during the fall and winter of the 1995-96 academic year, and the supplemental data (components 4 and 5) are being collected in the spring and summer of 1996. Once these data have been collected and analyzed, a report summarizing the evaluation study findings and recommendations will be prepared. The Foundation provides awards for research in the sciences and engineering. The awardee is wholly responsible for the conduct of such research and preparation of the results for publication. The Foundation, therefore, does not assume responsibility for the research findings or their interpretation.
The Foundation welcomes proposals from all qualified scientists and engineers and strongly encourages women, minorities, and persons with disabilities to compete fully in any of the research related programs described here. In accordance with federal statutes, regulations, and NSF policies, no person on grounds of race, color, age, sex, national origin, or disability shall be excluded from participation in, be denied the benefits of, or be subject to discrimination under any program or activity receiving financial assistance from the National Science Foundation.
Facilitation Awards for Scientists and Engineers with Disabilities (FASED) provide funding for special assistance or equipment to enable persons with disabilities (investigators and other staff, including student research assistants) to work on NSF projects. See the program announcement or contact the program coordinator at (703) 306-1636. The National Science Foundation has TDD (Telephonic Device for the Deaf) capability, which enables individuals with hearing impairment to communicate with the Foundation about NSF programs, employment, or general information. To access NSF TDD dial (703) 306-0090; for FIRS, 1-800-877-8339.