A Short-Term Impact Study
of the
National Science Foundation’s
Instrumentation and
Laboratory Improvement
Program
Table of Contents
Prepared under Contract
RED 94-52966
by
Kenneth Burgdorf
Tom Ewing
Michael Feinstein
Paul Tuss
Stephen Roey
Diane Ward
Westat, Inc.
with the assistance of
Phillip Tavel
NSF
NSF Program Officers
Mary Sladek, Division of Research, Evaluation & Communication
Duncan McBride, Division of Undergraduate Education
Directorate for Education
and Human Resources
[Image]
Division of Research
Evaluation & Communication
NOTE: The views expressed in this document are those of Westat, Inc. and do
not necessarily reflect the views of the
National Science Foundation.
October 1996
------------------------------------------------------------------------
Table of Contents
Executive Summary
I. The ILI Program
* History of ILI in the NSF Portfolio
* Program Objective
II. Signs Of Progress
* The Early Years
* Success in reaching target audiences
o Institution Level
o Students
o Disciplines
o Faculty
o Other Coverage Dimensions
+ Consortia
+ Women
+ Underrepresented Minorities
o Summary
* Recent Feedback from Grantees
o Resource Investment
o Educational Impacts
o Dissemination
o Research Gains
* Some Grantees Voice Concerns
III. Looking Ahead
* Evaluation Objectives
------------------------------------------------------------------------
EXECUTIVE SUMMARY
------------------------------------------------------------------------
The Directorate for Education and Human Resources (EHR) of the National
Science Foundation (NSF), which periodically assesses all of its programs,
is currently in the process of reviewing its Instrumentation and Laboratory
Improvement (ILI) program. Under the direction of EHR’s Division of
Research, Evaluation and Communication (REC) and Division of Undergraduate
Education (DUE), the current study is scheduled for completion by the end
of 1996. This three-section report is an initial product of the evaluation
now in progress.
The first section(The ILI Program) briefly describes the history,
objectives, and structure of this annual, peer-reviewed competition, which
provides funds for the purchase of laboratory equipment that will be used
in projects aimed at improving undergraduate science, mathematics,
engineering, and technology curricula.
The second section(Signs of Progress) summarizes what the Foundation has
learned thus far about the program’s success and shortcomings since its
establishment in 1985. Information in this section is based on previous
evaluations, the analysis of existing NSF record systems, and direct
contacts with selected ILI grantee institutions as well as individual
Principal Investigators (PIs) whose projects have incorporated laboratory
equipment acquired under an ILI grant.
Among the previous evaluations referred to in this section is a 1988-89 ILI
study that was based on mail surveys of grantees and site visits to
recipient institutions. This earlier evaluation revealed, for example, that
in the first 3 years of ILI’s existence, projects funded by the program had
provided students with previously unattainable opportunities for beneficial
“hands-on” learning experiences; that ILI-supported projects were spurring
progress among faculty members in improving their undergraduate science and
mathematics curricula; and that many recipient institutions were
successfully parlaying their financial and material support from NSF into
broader recognition and support from other outside sources.
Preliminary findings presented in the current evaluation--which focuses to
a great extent on grantees from 1990 and 1992--affirm many of the positive
conclusions reached in the earlier study, but go beyond those findings to
answer a number of other very important evaluation questions. For example,
the more recent findings presented here indicate that ILI has been
generally successful in reaching its target audience; that many students
have pursued advanced science/technology studies or careers following
experiences in ILI-enabled laboratories; and that the presence of
ILI-funded laboratory equipment has attracted new faculty at some
institutions.
At the same time, preliminary findings of the current study suggest some
areas where further refinements of the program may be warranted--for
example, to address gaps or unevenness of coverage, to reduce time burdens
or delays sometimes associated with project start-up, or to facilitate more
widespread dissemination of ILI-initiated innovations.
The third and concluding section of this report (Looking Ahead) discusses
the approach that REC is taking to further evaluate ILI’s impacts and to
identify specific steps that might be taken to enhance the program.
Table of Contents
------------------------------------------------------------------------
THE ILI PROGRAM
------------------------------------------------------------------------
Thanks largely to funding support from the National Science Foundation
(NSF), Priscilla Laws, a physics faculty member at Dickinson College in
Carlisle, Pennsylvania, was able to acquire a sophisticated microcomputer
system for use in her innovative work to improve the teaching of
undergraduate physics.
Another NSF grantee—chemist Benjamin DeGraff, a professor at James Madison
University in Harrisonburg, Virginia—teaches challenging courses on the use
of lasers and advanced measuring devices to perform thermal analysis. The
courses depend heavily on the presence in his laboratory of costly
state-of-the-art laser units and measuring devices. The support he has
received from the Foundation has made acquisition of these instruments
possible.
Likewise, NSF funding has allowed mathematician Eugene Herman of Grinnell
College in Iowa to purchase the powerful software he needs to nurture his
students' insights into linear algebra, differential equations, and
calculus by representing mathematical ideas and objects in onscreen forms
that permit manipulation and experimentation.
Despite the diversity of their disciplines and pedagogical pursuits, Drs.
Laws, DeGraff, and Herman have in common that they and their institutions
are all beneficiaries of NSF's Instrumentation and Laboratory Improvement
(ILI) program. ILI is an annual, peer-reviewed grant competition that
provides funds for curriculum improvement through the purchase of
laboratory equipment for use in projects aimed at improving undergraduate
education in science, mathematics, engineering, and technology. ILI grants
are in the $5,000-$100,000 range and must be matched with nonfederal funds;
the grant and matching funds must be used solely for purchase of
instructional laboratory equipment.
ILI grants are awarded for projects that show promise of advancing the
state of the technology in a given field: ILI projects are expected to
produce models for the improvement of undergraduate laboratory instruction
nationally, as well as at the project site.
ILI has become one of the Foundation's most popular programs, attracting
more than 2,000 proposals annually and awarding some $23 million a year to
successful applicants.
In its first decade, from the program’s inception in 1985 through 1994, ILI
has considered nearly 18,000 proposals from nearly 1,800 different
institutions (Table 1). It has made a total of more than 4,700 grants to
applicants from approximately 1,200 different institutions and has awarded
more than $158 million. Since grantees are required to match ILI funding
dollar for dollar, the program has stimulated investment of at least $316
million in the acquisition of undergraduate laboratory equipment.
------------------------------------------------------------------------
Table 1. Cummulative ILI proposal and award data,
1985-94
ProposalsAwards Award amount
($ million)
Total number/amount 17,793 4,704 $158.6
Number of institutions 1,777 1,185 --
Table of Contents
------------------------------------------------------------------------
History of ILI in the NSF Portfolio
------------------------------------------------------------------------
Established in 1985 as the College Science Instrumentation Program (CSIP),
the ILI program bolsters the NSF mission of providing leadership for the
Nation's efforts toward stimulating student interest in science,
mathematics, engineering, and technology. Operated by the Division of
Undergraduate Education (DUE)—a unit of NSF’s Directorate for Education and
Human Resources (EHR)—ILI complements other programs aimed at enhancing the
quality of instruction in institutions of higher education, including
2-year as well as 4-year colleges and universities.
In addition to ILI, DUE's program portfolio includes Course and Curriculum
Development, a program aimed at improving the learning environment,
content, and experience of instruction; Undergraduate Faculty Enhancement,
which supports projects enabling faculty members to gain experience with
technological advances and new experimental techniques in order to
incorporate them effectively into undergraduate instruction; and
Collaboratives for Excellence in Teacher Preparation, which supports
large-scale efforts to achieve comprehensive change in the undergraduate
education of future teachers and to increase substantially the quality and
number of teachers well prepared in science and mathematics, especially
members of underrepresented population groups; and Advanced Technological
Education, a program (administered jointly with EHR’s Division of
Elementary, Secondary and Informal Education) that promotes curriculum
improvement and teacher development in undergraduate and secondary
technological education through national and regional alliances between
associate’s-degree-granting institutions and other secondary and
postsecondary institutions, business, industry, and government.
During its first 3 years of operation (1985-87), ILI had annual budgets in
the $5-$8 million range, with grantees—then restricted to
non-doctorate-granting 4-year colleges and universities—receiving awards
ranging from $5,000 to $50,000. Beginning in 1988, the program was given
its current name and was expanded in several respects. Most notably, the
range of eligible institutions was expanded to include 2-year institutions
and 4-year doctorate-granting colleges and universities; and the upper
limit of individual awards was increased to $100,000. Between 1988 and
1994, the program was flat-funded at $23 million annually.
Table of Contents
------------------------------------------------------------------------
Program Objectives
------------------------------------------------------------------------
The principal objective of the ILI program is
“...to support the development of experiments and laboratory
curricula which improve the science, mathematics, engineering,
and technology education of undergraduate students, both science
majors and non science majors, including pre-service teachers.
Proposals are sought for the development of innovative methods
for using laboratory exercises to improve student understanding
of basic principles and for use of modern instrumentation, new
technologies, or new applications for instruments that extend the
instructional capability of the equipment” (1996 ILI Program
Announcement; NSF 96-10).
In addition to its explicit, primary focus on laboratory curriculum
innovation, ILI implicitly has several other program objectives. Many of
these relate to the programs and personnel of the recipient institutions:
* Encouraging and assisting individual institutions and departments to
upgrade and expand their instructional laboratory equipment,
facilities, and curricula by enabling them to acquire modern equipment
they would not otherwise be able to afford;
* Spotlighting, recognizing, and rewarding individual faculty,
departments, and institutions that develop high-quality educational
programs in science, engineering, and technology, thereby encouraging
and attracting similar recognition and support from others;
* Stimulating, encouraging, and enabling faculty members to pursue their
interests in improving undergraduate laboratory curricula, and
retaining and reinvigorating where necessary their enthusiasm for
undergraduate teaching;
* Encouraging institutions and departments to provide increased
recognition and reward for faculty accomplishments in improving
undergraduate science education; and
* Helping undergraduate science, mathematics, engineering, and
technology courses to become more up to date, attractive,
understandable, stimulating, and relevant to modern life, thereby
generating increased student interest and enrollment in science- and
technology-related courses and degree programs.
In addition, the ILI program has from its inception sought to do more than
support improvements in the quality of instruction at individual
institutions. It has gone beyond this with an interest in innovative
curriculum development projects that show promise for improving the quality
of undergraduate science, mathematics, engineering, and technology
education nationwide.
According to a recent DUE program announcement (NSF 96-10), “The ILI
program seeks projects that will improve laboratory instruction nationally
as well as at the project site and that will produce and adapt national
models for the improvement of undergraduate laboratory instruction.” Given
this emphasis, the program actively discourages proposals that request
equipment solely on the basis of an institution's peculiar financial need
or increased enrollments.
Furthermore, throughout its existence ILI has particularly encouraged
proposals that involve provisions for equipment sharing through consortia
or centers, or that would strengthen undergraduate education by increasing
the participation of women, underrepresented minorities, and persons with
disabilities, especially if the projects present models for increasing the
numbers who successfully pursue careers in science, mathematics,
engineering, and technology.
Table of Contents
------------------------------------------------------------------------
SIGNS OF PROGRESS
------------------------------------------------------------------------
What has been discovered so far about the effectiveness of its ILI program?
What patterns have emerged over the years that shed light on the
institutions, individuals, and scientific disciplines benefiting from it?
And what issues must it try to resolve as the program enters its second
decade? Presented below is an initial overview of findings. The main
sources of information are an extensive evaluation of the program that was
conducted in 1988-89 (3 years after ILI’s inception); an analysis of ILI
records covering the program’s first 10 years (1985-94); and preliminary
findings from a series of site visits to grantee institutions and
interviews with individual awardees.
Table of Contents
------------------------------------------------------------------------
The Early Years
------------------------------------------------------------------------
The 1988-89 evaluation of ILI’s first 3 years—relying on data from mail
surveys of grantees and from site visits—revealed the following
observations:
* Projects afforded students previously unavailable opportunities for
direct, hands-on learning that significantly stimulated, enriched, and
enlivened their undergraduate experiences. In many cases,
ILI-supported projects were viewed as having produced increased
department enrollments, increased student interest in science and
technology careers, and increased postcollege employment opportunities
for students in science-related fields.
* ILI-supported projects consistently had substantial positive effects
on faculty and departments, both by providing a prestigious form of
recognition and by encouraging and enabling faculty to pursue their
ideas for updating and improving undergraduate curricula. As well as
having beneficial effects on individual faculty members' careers, many
of these projects were said to be responsible for reinvigorating
faculty morale and enthusiasm for teaching.
* From the initial impetus provided by ILI grants, many projects had
attracted additional financial and other support enabling them to
obtain additional instructional equipment and to grow in scope well
beyond what had initially been contemplated. (On average, 1985-87
projects attracted total financial support at least 4.5 times the size
of the NSF grant amount.)
A particularly telling indication of the program's value was yielded by a
hypothetical question put by evaluators to the For 85 percent of awardees,
without NSF support projects would have been significantly scaled back or
not implemented at all. grantees being surveyed. What, they were asked,
would have happened to the projects they envisioned if NSF funding
had been denied?For 85 percent of the awardees, the grants were of
critical importance; without the Foundation's support, they said, their
projects would either have been significantly scaled back or not
implemented at all.
Commenting at the time of the 1988-89 evaluation, Thomas Tucker, a Colgate
University mathematics professor, reflected the enthusiasm often expressed
by grantees. He cited a Colgate project supported by an ILI grant as an
important factor in the Hamilton, New York, institution’s receiving a
$100,000 state grant to interest inner-city youth in science. Tucker said:
“The program is now a flagship. It would be difficult to imagine a project
more successful than this one.”
Table of Contents
------------------------------------------------------------------------
Success in Reaching Target Audiences
------------------------------------------------------------------------
In the initial phases of the 1994-97 evaluation, ILI proposal and award
data residing in NSF’s mainframe database were compiled for the program's
first 10 years (1985-94). These data reveal an abundance of information
about the program's effectiveness thus far in reaching its target audiences
of institutions, disciplines, students, faculty members, and others.
Table of Contents
------------------------------------------------------------------------
Institution Level¹
------------------------------------------------------------------------
Approximately 1,500 colleges and universities in the United States award
bachelor’s degrees in NSF-supported fields (science, mathematics,
engineering, and technology). The great majority of them (79 percent) have
sought support from ILI by submitting one or more proposals during the
1985-94 period (Chart 1). Nearly all doctorate-level colleges and
universities (95 percent) have requested ILI support,² as have 88 percent
of master’s-level institutions and 68 percent of bachelor’s-level colleges.
The lowest rate of participation has been from 2-year institutions that
award associate’s degrees in science-related fields: 40 percent of the
1,160 such institutions have submitted proposals to ILI during the 7-year
period that they have been eligible to participate in the program.
-----------------------
¹In this discussion, institutions that award undergraduate degrees in
NSF-supported fields are classified according to the highest degree they
award in these fields: doctorate, master's, bachelor's, or associate's. All
institutions in the first three categories are 4-year;all in the last
category are 2- year.Return
²Most of the exceptions are seminaries or other institutions whose only
science-related doctoral programs are in psychology or the social
sciences.Return
------------------------------------------------------------------------
Chart 1. ILI coverage of institutions, by highest degree awarded in
science, mathematics, engineering, and technology, 1985-94
[Image]
*Base = 4-year colleges and institutions that awarded
bachelor’s degrees in science, mathematics,
engineering, or technology fields in 1993, and 2-year
colleges that awarded associate’s degrees in these
fields in 1993.
------------------------------------------------------------------------
Over half (61 percent) of all 4-year colleges and universities with
undergraduate degree programs in NSF-supported fields have received one or
more ILI awards, including 87 percent of doctorate-level institutions, 74
percent of master’s-level institutions, and 45 percent of bachelor’s-level
institutions. Nineteen percent of relevant 2-year institutions have
received ILI awards. This last figure, while low, indicates that nearly
half (47 percent) of the 2-year institutions that have submitted one or
more ILI proposals have been successful in winning one or more awards.
Among institutions that participated in ILI during its first decade, most
submitted multiple proposals. The most proposals submitted by any single
institution was 126. The overall average was 10 proposals per institution,
but this varied considerably by institution size and level, ranging from an
average of 24 proposals per institution for doctorate-level institutions to
2.4 proposals per institution for 2-year, associate’s-level institutions.
Most institutions that have won any ILI awards during the 1985-94 period
have won multiple awards. The highest number of awards for any single
institution was 36. The overall average was 4 awards per institution,
generally spread across several disciplines. Doctorate-level and
master’s-level institutions have been most successful (both averaging about
5.5 awards per institution), followed by bachelor’s-level institutions (3.4
awards per institution) and 2-year institutions (1.4 awards per
institution).
Table of Contents
------------------------------------------------------------------------
Students³
------------------------------------------------------------------------
In general, institutions that serve relatively large numbers of
undergraduate science, mathematics, engineering, and technology students
have been most active and most successful in the ILI program. Consequently,
program coverage statistics are even higher when expressed in terms of
coverage of students than when calculated with institutions as the unit.
The extent of knowledge about and interest in ILI in the academic community
is indicated by the finding that 96 percent of all students who
received bachelor’s degrees in 1993 in NSF-supported fields got
their degrees from institutions that have requested ILI support at least
once. Over 99 percent of undergraduate science majors at doctorate-level
institutions were in this situation, as were almost as many students at
master’s-level institutions (97 percent) and at bachelor’s-level
institutions (82 percent). These findings indicate strong awareness of and
interest in obtaining program improvement support from the ILI program
among institutions that have bachelor’s degree programs in science-related
fields.
As for support actually delivered by ILI, nearly 9 of every 10 students (89
percent) who received bachelor’s degrees in NSF-supported fields in 1993
studied at institutions that have received one or more ILI awards (Chart
2). Undergraduate students at doctorate-level institutions were especially
well-covered (95 percent). The coverage has been lower for students at
2-year institutions: 29 percent of associate’s degree recipients in
science-related fields studied at institutions that have received one or
more ILI awards.
As indicated earlier, many of the larger institutions have won multiple ILI
awards over the years. The result is that, overall, 61 percent of 1993
bachelor’s degree recipients in NSF-supported fields received their degrees
from institutions that have won four or more ILI awards.
-----------------------
³In the following discussion, ILI coverage of the Nations's population of
undergraduate students is measured in terms of coverage of students who
receive undergraduate (bachelor's or associate's) degrees with majors in
science, mathematics, engineering, or technology fields. This is not to
imply that ILI projects are limited to courses for students with
science-related majors, however. Many ILI projects involve lower-division
courses that are open to nonscience students as well as to science
majors.Return
------------------------------------------------------------------------
Chart 2. ILI coverage of undergraduate students in science, mathematics,
engineering, and technology programs, 1985-94
[Image]
Table of Contents
------------------------------------------------------------------------
Disciplines
------------------------------------------------------------------------
Throughout its history, the ILI program has distributed awards among
disciplines in close approximation to the number of proposals received
(Table 2). However, some NSF-supported fields have participated more
extensively than others in the program and consequently have derived
greater benefits.
------------------------------------------------------------------------
Table 2.Distribution of undergraduate degrees and ILI proposals and awards
in NSF-supported fields: 1985-1994
Field Under-graduate ILI ILI Award
degrees, 1993* proposals awards amount
Total number 485,416 17,793 4,704 $158.6 M
(Percent)
Physical sciences** 4 35 38 34
Engineering/technology 23 25 19 24
Biological sciences 13 17 17 15
Computer science 7 7 6 9
Mathematics 3 5 7 8
Psychology 14 4 4 3
Social sciences 29 3 3 3
Other/interdisciplinary 7 4 5 6
*Includes bachelor’s and associate’s degrees in all science, mathematics,
engineering, and technology disciplines.
**Includes chemistry, physics, astronomy, and geosciences.
NOTE: Percents may not add to 100 due to rounding.
------------------------------------------------------------------------
The physical sciences—chemistry, physics, astronomy, and the
geosciences—collectively account for over one-third of all ILI proposals
and awards, well above their 4-percent share of undergraduate degrees
across all NSF-supported fields. This prominence is not surprising,
however, given the highly instrumentation-intensive nature of these
laboratory-based disciplines, and also considering that these disciplines
provide large numbers of lower division service courses for nonscience
majors, in addition to advanced courses for science majors. ILI supports
both categories of courses, lower division as well as upper division.
Within the physical sciences, chemistry projects have been most dominant in
the program over the past decade (960 awards totaling more than $31
million), followed by physics/astronomy (585 awards totaling more than $15
million) and the geosciences (252 awards totaling nearly $8 million).
At the other extreme, the social sciences and psychology collectively
account for only 6-7 percent of ILI proposals and awards, far lower than
their 43 percent share of science-related undergraduate degrees.
Engineering/technology disciplines are prominent in ILI: they account for
about one-quarter of ILI proposals and awards, which is consistent with
their share of undergraduate degree productivity (23 percent). The
biological and computer sciences and mathematics also account for
significant shares of ILI proposals and awards, at levels commensurate with
their undergraduate degree productivity (Table 2).
Table of Contents
------------------------------------------------------------------------
Faculty
At the individual level, ILI proposals and awards have been widely
dispersed. The roughly 18,000 proposals that were submitted to ILI in its
first decade of existence were submitted by nearly 12,000 different
principal investigators (PIs). Considering that significant numbers of
proposals are submitted in revised form after having been declined one or
more times, these findings indicate that participation in ILI has been
remarkably widespread.
Awards have been even more widely distributed, with ILI’s 4,704 awards
through 1994 being made to 4,305 different PIs—an average of 1.1 awards per
grantee. (Only 9 percent of ILI grantees have won more than one award.)
Table of Contents
------------------------------------------------------------------------
Other Coverage Dimensions
------------------------------------------------------------------------
Although the ILI program has achieved broad overall coverage of its target
undergraduate institutions and students (especially those with or in
bachelor’s-level programs) and has distributed program support widely
across disciplines and individual faculty members, it has not been entirely
successful in its special initiatives regarding consortia, women, and
minorities.
Consortia. In each year of the program, since its inception in 1985, the
ILI program announcement has made a special point of encouraging proposals
for equipment-sharing consortia or centers that would help make modern
laboratory equipment more widely accessible to undergraduate students in a
particular discipline or geographical area. So far, this has not happened.
Of the nearly 18,000 proposals the Foundation has received in the ILI
program, only 54 have involved proposed consortia. Of these, only 19 were
funded, most at low budget levels: only 2 had awards over $50,000.
Women. Sixteen percent of ILI grantees during the program’s first decade
have been women. This figure is far below the percentages of women in the
population or in the overall workforce. It compares favorably, however,
with the percentages of women among the science college faculty who
constitute the personnel pool from which ILI PIs are drawn. For example, in
1991-92, only 4 percent of full-time college teaching faculty in
engineering were women, as were 7 percent of physics faculty, 9 percent of
geosciences faculty, and 30 percent of sociology faculty.4 In the same
fields, women have constituted 7 percent of ILI grantees in engineering, 8
percent in physics/ astronomy, 14 percent in geosciences, and 19 percent in
the social sciences. In 3 of these 4 fields, women have been more highly
represented in ILI than in the workforce from which ILI PIs were drawn.
Given NSF's policy objective to support increased participation by women in
science-related fields, it is also encouraging to note that the percentage
of women grantees in ILI has exceeded the percentage of women proposers in
each of the program's first 10 years, and the percentage of women grantees
has grown steadily and substantially over this decade, doubling from 12
percent in 1985 to 24 percent in 1994.
-----------------------
4 All reported faculty population statistics are taken from the National
Science Foundation report Women, Minorities, and Persons with Disabilities
in Science and Engineering: 1994(NSF94-333).Return
Table of Contents
-----------------------
Underrepresented Minorities. African-Americans, Alaskan Natives, Hispanics,
and Native Americans are also historically underrepresented in
science-related fields and in the populations of undergraduate science
faculty from which ILI PIs are drawn. These underrepresented minorities
collectively accounted for about 20 percent of the Nation's civilian labor
force in 1990, yet they constitute only 4 percent of the college/
university teaching faculty in three fields: engineering, physics, and
geology. In sociology, the fourth field for which population figures are
available, the minority percentage is higher: 12 percent.
Regarding ILI proposals and participation, available statistics indicate:
* 3.7 percent of the proposals received by ILI during its first decade
involved PIs from underrepresented minorities;
* 3.3 percent of all awards made by ILI during its first decade were to
PIs from underrepresented minorities;
* In comparison to institutions with lower minority enrollments,
institutions with significant (over 15 percent) minority enrollments
have been less active in ILI (fewer have submitted any proposals) and
have received fewer awards (Table 3); these differences largely
reflect the differences noted earlier between 2-year and 4-year
institutions in ILI participation and support. 5
* The ILI program’s overall proposal funding rate is 26 percent, and the
success rates for proposals from institutions with high minority
enrollments range from 25 percent to 29 percent (see Table 3).
----------------------------------------------------------------------
Table 3. ILI participation and success rates for institutions
with significant minority enrollment
Institution Number of Proposals Proposal
characteristic institutions ILI Awards funding
rate
Total, all
institutions 2,665 62% 43% 26%
Minority enrollment
> 15% 897 52% 32% 26%
Enrollment > 15%
black 452 50% 28% 25%
Enrollment > 15%
Hispanic 243 54% 34% 29%
Enrollment > 15%
Native American 29 38% 21% 28%
----------------------------------------------------------------------
---------------------
5 Well over half of the institutions with high minority enrollments
are 2-year colleges, and their participation and support rates are
comparable to those for other 2-year institutions. Return
---------------------
* Historically black colleges and universities (HBCUs), which are
perhaps most widely recognized as serving distinctly minority
students, have fared well in ILI: 66 (74 percent) of the 89 HBCUs that
award undergraduate degrees in science-related fields have applied for
ILI grants, and 48 percent have received one or more grants; both of
these figures are slightly above the program average.
* There are nine tribal colleges (located in Native American tribal
lands) that award undergraduate degrees in science-related
disciplines. All are 2-year institutions. Three have submitted
proposals to ILI; one has received an ILI award.
* The representation of PIs from minority groups has grown, gradually
but perceptibly, over the brief history of the ILI program, from 1985
and 1986 when 1 percent of all ILI grantees were from underrepresented
minorities to 1992, 1993, and 1994, when 5 percent were from these
groups.
Table of Contents
------------------------------------------------------------------------
Summary
------------------------------------------------------------------------
The data analyzed thus far indicate that the ILI program has achieved
excellent overall coverage of its target audiences, but that some gaps in
the coverage exist for some demographic and institutional groups.
Furthermore, there appears to be some unevenness in benefits, with some
institutions being heavily impacted with multiple awards and others gaining
little or no direct benefit from the program.
------------------------------------------------------------------------
Recent Feedback From Grantees
------------------------------------------------------------------------
In addition to the quantitative data referred to above, the current study
also has obtained preliminary qualitative information about the impacts of
several individual ILI projects. This information was gathered during the
course of 1995 site visits to six ILI recipient institutions that were
selected essentially at random and through telephone conversations with
other PIs who were identified by NSF program staff as having benefited from
ILI in the past.
The projects discussed with this small group of PIs proved to be highly
diverse; they involved different fields and different institution settings,
and they have developed in different directions. Within this small sampling
of projects, some impressive examples of several different types of
accomplishments were found. As such, these projects provide concrete
illustrations of some of the kinds of impacts the ILI program has produced,
although the number of cases studied is far too small to draw any
conclusions about the relative prevalence or cumulative magnitudes of any
single type of impact. More will be known on these subjects after the
upcoming components of the evaluation have been completed.
Table of Contents
------------------------------------------------------------------------
Resource Investment
------------------------------------------------------------------------
Aside from the matching funds provided by a college or university, several
of the ILI project PIs visited by program evaluators mentioned that
additional financial investments were generated as a result of their ILI
awards. The winning of an ILI grant from NSF is apparently seen as a sign
of quality assurance by some private and industrial funding sources that
need to verify that programs they have been asked to support are truly
meritorious. The peer-review process for ILI awards is presumably regarded
by these potential donors as credible and rigorous, with only the highest
quality programs receiving ILI support. Thus, with an ILI grant in hand, a
PI or institution may be in a position to leverage that funding well beyond
the amount awarded by NSF. For example:
* A $45,000 1990 ILI grant at Camden County College in New Jersey under
the direction of PI James Hudgings, a professor in the department of
engineering technology, supported the acquisition of a distributed
numerical control system integrating computer-aided design with
computer-aided manufacturing (CIM). This ILI project eventually
generated $300,000 in state money for laboratory improvement
associated with the upgrading of the college's CIM program, $1.5
million for basic CIM equipment, and $4.2 million for upgrading the
college's CIM research facility.
* At the University of Delaware in Newark, Murray Johnston's $48,000 ILI
grant in 1992 allowed the chemistry department to acquire several
sophisticated measurement instruments for use in a novel series of
independent study courses involving instrumental analysis. An
unanticipated benefit for Johnston's plan was that by leveraging the
$45,000 his project gained through ILI funding, he was able to get
price concessions from the instrument manufacturer totaling about
$85,000.
* Four different ILI awards to support physics professor Priscilla Laws'
innovative courses at Dickinson College have over the years helped
generate approximately $10 million in curriculum development and
faculty workshop grants from 10 different government agencies, private
foundations, and commercial organizations.
Table of Contents
------------------------------------------------------------------------
Educational Impacts
------------------------------------------------------------------------
The immediate, direct effect that an ILI award has on an institution is
that it enables faculty to acquire instructional equipment they could not
have gotten from the university or outside donors alone. This equipment
then provides the building blocks with which faculty create new laboratory
exercises and experiments, new opportunities for student research, new or
expanded courses, and in some cases, new programs of study. Educational
impacts of these kinds were seen for all site-visited projects. Some
examples:
* Over the years, Gary Kader and other faculty in the Department of
Mathematical Science at Appalachian State University in Boone, North
Carolina, have received several ILI grants, which have enabled them to
restructure many of their undergraduate courses. From introductory
service courses for liberal arts students to advanced calculus and
computer science courses, the bulk of the department's offerings are
now being taught in computer labs using equipment and software funded
through ILI and assembled as ILI projects. One of these courses, first
offered in 2 sections per semester, has now grown to 10-12 sections
per semester and has become part of the university's core curriculum.
* At the University of Southern Mississippi in Hattiesburg, a new
laboratory course in molecular biology was created to accompany an
existing lecture course, enabled by a 1990 ILI grant for purchase of
requisite equipment. The new course has become so popular, says PI
Glen Shearer, a professor in the biological sciences department, that
unfortunately some students have been turned away.
* At Illinois Benedictine College in Lisle, enhancement of inorganic
chemistry and biochemistry courses was fostered by the ILI-supported
acquisition of a sophisticated spectrophotometer. According to PIs
Edward Ferroni and David Sonneburg of the college's chemistry
department, the device has stimulated major revisions in the courses
they teach—improvements that would have been impossible without the
new machine.
Table of Contents
Dissemination
------------------------------------------------------------------------
The positive influence of an ILI program often extends beyond the
institution receiving an award. NSF encourages the dissemination of
instructional materials by PIs who have successfully implemented an
ILI-seeded laboratory project, and many awardees are eager to share their
experiences and acquired knowledge with others in their field.
Presentations, publications, lectures, workshops, and seminars are the most
common means for PIs to disseminate information about their projects.
Notably, they are taking increasing advantage of electronic media to
enhance their dissemination efforts—using the Internet, for example, to
distribute lab manuals and other materials. The approaches that several
awardees have taken in sharing information about their project-developed
products and insights are especially impressive. For example:
A project at Lawrence University in Appleton, Wisconsin, was designed to
determine the staffing and hardware requirements for meaningful instruction
in lasers and modern optics. John Brandenberger, the PI for the project,
developed a wealth of dissemination materials including two monographs, a
journal article, a series of workshops, and an assortment of
hand-out documents ranging from copies of magazines featuring the
laser laboratory to reprints of newspaper articles. The major dissemination
vehicle was the document Lasers and Modern Optics in Undergraduate Physics,
which was distributed free to nearly 700 physics departments throughout the
United States. It includes a description of the space, staffing, and
equipment requirements for a productive laser laboratory, plus a
description of three courses in laser physics—everything one would need to
know for setting up a dynamic undergraduate laser physics laboratory and
curriculum.
* Dickinson College's Priscilla Laws makes an average of 30 trips a year
related to the subjects of her ILI grants. In addition to writing
articles, giving conference talks, leading workshops, and consulting
with institutions and instructors, she networks with Tufts University
and the University of Oregon on an ongoing basis concerning further
development of Real-Time Physics—a series of PC-based student
laboratory guides and other tools for interactive teaching—which she
and others developed as an outgrowth of a 1991 ILI project.
* Cutting-edge communications are also being used for dissemination
purposes. A successful ILI project enabling advanced work in digital
imaging by Lawrence Marschall at Pennsylvania's Gettysburg College,
for example, has led to the electronic distribution of laboratory
modules—free of charge to the educational community nationwide—that
can guide other instructors in making introductory astronomy labs more
attractive to students.
Table of Contents
----------------------------------------------------------------------
Research Gains
----------------------------------------------------------------------
The availability of advanced laboratory instrumentation acquired
through ILI has clearly enriched undergraduate research activity at
some institutions. And at some schools—such as the University of
Southern Mississippi and Illinois Benedictine College—the presence of
advanced laboratory equipment has attracted talented new faculty. At
Southern Mississippi, PI Glen Shearer allows his ILI-funded equipment
to be used by students and faculty from throughout the state for
various types of research projects. And at the University of Delaware,
as many as 18 faculty members were found to be using the ILI-funded
mathematics laboratory on a regular basis.
Table of Contents
----------------------------------------------------------------------
Some Grantees Voice Concerns
Grantees interviewed by the evaluation team shared in enthusiastic
agreement that the ILI program is extremely important and effective.
Indeed, the general opinion is that without the NSF
imprimatur—expressed in the form of an ILI award—many science,
mathematics, engineering, and technology educators today would have
serious problems obtaining the laboratory equipment they need in order
to improve their courses. Nevertheless, PIs did raise a number of
concerns and offered recommendations on how the program could be
improved. For example:
* The most common recommendation aims at diminishing the heavy time
burden placed on PIs during the startup phase of their projects.
To alleviate this burden, PIs would like to see their teaching
load reduced while they are installing their new equipment and
otherwise launching their projects. This could be achieved, many
of them feel, if NSF would either—as part of an award—provide
funds for temporary teaching substitutes, or—as a condition of an
award—require grantee institutions to provide for this need.
* PIs would like to see a supply budget included in an ILI grant to
help maintain equipment and purchase necessary supplies. (This is
an especially serious matter at small schools that have no staff
on site to provide computer support or to repair equipment.)
* To improve the diffusion of information generated from ILI
projects, it is suggested that NSF take greater advantage of
electronic communication media, such as the Internet.
These and other issues are being explored by NSF on a broad scale as
the Foundation moves forward with its comprehensive 1995-96 evaluation
of the ILI program.
Table of Contents
----------------------------------------------------------------------
LOOKING AHEAD
----------------------------------------------------------------------
The full-scale ILI study now under way—the second multiyear evaluation
that the program has undergone since its inception in 1985—is based on
the understanding that the program has matured and grown substantially
in size and scope over the years and the time has come for NSF to take
a fresh look at it. ILI has become a well-established, well-known
program that has funded many projects over the past decade. Many of
these projects, begun a number of years ago, have now had time to
produce real impacts at the recipient institutions and in the larger
educational culture.
This reevaluation intends to assemble information about ILI's
activities and accomplishments during its first decade of operations
and about the impacts it has had on undergraduate education. It will
also seek to uncover significant problems or limitations that should
be addressed as ILI enters its second decade. Additionally, it will
seek to identify a small number of useful indicators that can be
collected on a regular basis from future ILI grantees, enabling NSF to
track significant program outcomes and impacts between large-scale
evaluations such as this.
The main data collection for the evaluation will occur during the
1995-96 academic year. The evaluation is being funded jointly by NSF's
Division of Research, Evaluation and Communication and the Division of
Undergraduate Education. Both divisions are branches of the
Foundation’s Directorate for Education and Human Resources.
The outside contractor for the evaluation is Westat, Inc., a
Rockville, Maryland, research and consulting firm that has conducted
many other Federal program evaluations and research studies in the
postsecondary arena. A six-person panel of academic scientists and
administrators who are knowledgeable about the ILI program has been
appointed to advise Westat in the design and conduct of the project
and in the assessment of the findings. These advisors are as follows:
* Biology: Dr. Eric Thomas, Vice President of Academic Affairs,
Mississippi Valley State University, Itta Bena, Mississippi.
* Chemistry: Dr. Thomas C. Farrar, Professor of Physical Chemistry,
University of Wisconsin, Madison, Wisconsin.
* Engineering: Dr. Eleanor Baum, Dean of Engineering, Cooper Union,
New York, New York.
* Engineering Technology: George A. Timblin, Head, Department of
Engineering and Advanced Technology, Central Piedmont Community
College, Charlotte, North Carolina.
* Mathematics: Dr. Anita Solow, Professor of Mathematics, Grinnell
College, Grinnell, Iowa.
* Physics: Dr. James Parks, Director of Undergraduate Laboratories,
University of Tennessee, Knoxville, Tennessee.
Table of Contents
----------------------------------------------------------------------
Evaluation Objectives
----------------------------------------------------------------------
The general purpose of the evaluation is to assess how successful the
ILI program has been in achieving its objectives over the past decade
overall and how well it is meeting the needs of specific categories of
undergraduate institutions (that is, 2-year; 4-year associate’s-,
bachelor’s-, master’s-, or doctorate-granting). It also is designed to
assess how effectively the program is meeting the needs of individual
science, mathematics, engineering, and technology disciplines, and
whether it is succeeding in reaching its various target audiences. The
findings presented in this report are only the first to emerge in this
new effort. As it progresses, the evaluation will attempt to document
ways in which the program has been successful; at the same time, it
will look for ways in which ILI might be strengthened or improved.
The evaluation design centers around 10 broad questions, some of which
have been addressed in this report and others that have yet to be
addressed.
1) What is the scope and coverage of ILI? How has the program grown
over its 10 years of operation, in terms of annual and cumulative
numbers of proposals received and awards made? To what extent has the
program provided support across the full spectrum of ILI-eligible
institutions and individuals?
2) How much instructional resource investment has ILI stimulated? To
what extent has ILI seed money attracted additional financial support
from non-NSF sources for undergraduate instructional laboratory
equipment and facilities?
3) What, if any, have been the educational and research impacts of ILI
at grantee institutions? How has ILI stimulated the development of new
or improved laboratory courses or other kinds of expanded learning
opportunities for undergraduate students? To what extent has
program-supported equipment been used for undergraduate or other
research?
4) What, if any, have been the educational impacts of ILI on the
larger culture, beyond the grantee institutions? To what extent has
the program spurred the development of new instructional approaches or
materials that have become widely adopted outside the grantee
institutions? In what fields, and through what dissemination
mechanisms, have the largest effects occurred?
5) Has ILI impacted faculty and faculty-reward systems at grantee
institutions? To what degree has the program affected faculty members
who have been involved in ILI projects, in terms of improved morale,
instructional effectiveness, and capacity to update teaching
approaches?
6) Has ILI impacted curriculum and career development among
unsuccessful applicants? How often has applying for a program grant,
even if the proposal was declined, stimulated investigators to develop
and find support for worthwhile curriculum improvement projects that
eventually became implemented without ILI support?
7) Has ILI impacted underserved groups? How successful has the program
been in attracting proposals from PIs who are women, members of
underrepresented minorities, or persons with disabilities? How well
represented are members of these groups among ILI awardees and
students impacted by ILI projects?
8) Has ILI impacted K-12 students and teachers? What effect has the
program had on the training of primary and secondary school science
and mathematics teachers?
9) What, if any, commercial, community, or other additional impacts
has ILI had? Have individual ILI projects provided beneficial services
or established mutually beneficial associations with local (or perhaps
national or international) governmental, educational, industrial,
research, or other organizations.
10) How could the program be improved? Has the program failed to
achieve important objectives in some areas? Are there aspects of the
program policy, structure, or management that have been ineffective,
inadequate, or otherwise problematic? Are there changes that should be
considered in the way the program is structured or operated?
To address all of these questions, the 1995-96 evaluation comprises
five principal components:
1) A thorough examination of existing NSF management information
systems data concerning ILI proposals and awards during the first
decade of the program (1985-94). Other existing data sources are also
being examined to assess ILI's success in reaching its target
audiences.
2) A mail survey of ILI grantees. The survey includes all grantees
from the 1990 and 1992 cycles of the program. These two program cycles
were selected as being generally typical of the numbers and kinds of
ILI projects that have been supported throughout the post-CSIP period
and as being of an optimal age for impact evaluation. The survey
questionnaire collects a variety of basic information about the size,
structure, and current status of a grantee's project; about the
numbers and characteristics of involved students and faculty; and
about the project's impacts and products to date, both within and
beyond the grantee institution.
3) A concurrent survey of samples of unsuccessful applicants from the
1990 and 1992 programs (250 from each year) to obtain their views
about the program and its impacts on them.
4) Supplemental site visits to approximately 25 institutions,
following the mail surveys to validate the questionnaire data and
assemble additional documentation and insights about selected
instances of especially large ILI-related educational impacts.
5) Tracer studies of 10-20 projects that are identified by NSF program
staff or by questionnaire responses as involving extensive external
dissemination of procedures and products developed as an outgrowth of
ILI projects. The PIs of these exemplary, high-impact projects are
being interviewed to learn more about the dissemination activities
that have occurred and about the surrounding circumstances and support
systems. In addition, selected second-stage recipients of
project-disseminated products and information will be contacted to
learn how the materials have affected their own undergraduate
teaching, thereby documenting in some measure the kinds of
national-level impacts the ILI program has had on undergraduate
instruction in science, mathematics, engineering, and technology.
The first of these five components has been completed, with the
results summarized earlier in this report. The survey collection
activities (components 2 and 3) were conducted during the fall and
winter of the 1995-96 academic year, and the supplemental data
(components 4 and 5) are being collected in the spring and summer of
1996. Once these data have been collected and analyzed, a report
summarizing the evaluation study findings and recommendations will be
prepared. The Foundation provides awards for research in the sciences
and engineering. The awardee is wholly responsible for the conduct of
such research and preparation of the results for publication. The
Foundation, therefore, does not assume responsibility for the research
findings or their interpretation.
The Foundation welcomes proposals from all qualified scientists and
engineers and strongly encourages women, minorities, and persons with
disabilities to compete fully in any of the research related programs
described here. In accordance with federal statutes, regulations, and
NSF policies, no person on grounds of race, color, age, sex, national
origin, or disability shall be excluded from participation in, be
denied the benefits of, or be subject to discrimination under any
program or activity receiving financial assistance from the National
Science Foundation.
Facilitation Awards for Scientists and Engineers with Disabilities
(FASED) provide funding for special assistance or equipment to enable
persons with disabilities (investigators and other staff, including
student research assistants) to work on NSF projects. See the program
announcement or contact the program coordinator at (703) 306-1636. The
National Science Foundation has TDD (Telephonic Device for the Deaf)
capability, which enables individuals with hearing impairment to
communicate with the Foundation about NSF programs, employment, or
general information. To access NSF TDD dial (703) 306-0090; for FIRS,
1-800-877-8339.
----------------------------------------------------------------------
Table of Contents