This document has been archived.
The evaluation results reported in the previous section indicate that ILI awards directly stimulate innovative curricular changes that produce many kinds of beneficial impacts on under-graduate students at the grantee institutions. The results also show that ILI is effective in leveraging resources at grantee institution and stimulates broader investments in SMET education. However, an important additional measure of the achievements and impacts of ILI projects is their influence on individuals and organizations that are not direct beneficiaries of ILI awards. In fact, ILI proposals are evaluated by review panels based on the promise that they will benefit undergraduate SMET education nationally, not just at the grantee institution. To obtain a greater understanding of the far-reaching effects of ILI activities, the evaluation team examined the indirect impacts of the program on the larger culture beyond the grantee institutions.
Summary of Findings About
Far-Reaching Impacts
ILI-generated instructional innovations are widely disseminated using a variety of mechanisms. PIs report an average of 5 PI-initiated dissemination products or activities. This includes presentations of papers at regional or national conferences, publications in journals and other professional media, presentations or colloquia at other institutions, and workshops or short courses to acquaint others with project developed approaches and materials.
Disseminated products stimulate
innovation at other institutions and organizations. Although the cumulative
national and worldwide impacts on undergraduate SMET education of the PI-initiated
innovation dissemination activities are difficult to quantify precisely,
the results of the tracer studies
conducted by the evaluation team show that significant instructional innovations
have been adopted by workshop attendees and other identifiable second-stage
users of project-generated products throughout the nation.
ILI grants stimulate research productivity. As a combined consequence of student and faculty research usage, ILI projects have generated significant numbers of research products, resulting in overall averages of 2 research publications and 6 presentations of research findings for 1990 and 1992 projects.
Unsuccessful ILI applicants persevere to achieve their goals. Altogether, 78 percent of the curricular improvement projects that were submitted as proposals to ILI in 1990 or 1992 ultimately became implemented to some degree, either with ILI support won in subsequent competitions or with support from other sources.
Fewer impacts are reported on K-12 students and teachers and on other community organizations. One-third of ILI PIs report that their project has impacted K-12 students in one way or another, and a similar percentage report project impacts on K-12 teachers. Seventeen percent of PIs report involvement in other forms of community outreach, local industry, public health agencies, or other community organizations.
Dissemination of
ILI-Generated Products
If ILI projects are to exert a far-reaching influence beyond the grantee institutions, their activities and accomplishments first must become widely known among other members of the SMET education fields at the undergraduate level and beyond. In fact, this is a fundamental objective of the ILI program. Grantees are expected to make conscientious efforts to share the innovative instructional approaches and materials they develop in their projects.
Types and Extent of Dissemination
Activities. The instructional innovations generated by the ILI program
are widely shared with others outside the grantee institution using a variety
of dissemination mechanisms (Table 9). Two-thirds of the grantees in 1990
and 1992 initiated some form of dissemination, and a similar percentage
engaged in less formal sharing of project-related information in response
to individual inquiries. Presentations at national and regional conferences
are the most frequently reported means of PI-initiated dissemination, although
publications in journals and other professional media, presentations or
colloquia at other institutions, and workshops or short courses are also
employed to acquaint others with project-developed approaches and materials.
However, only 32 percent of PIs can confirm that undergraduate students
from outside their own institutions have been impacted by the instructional
innovations developed in their ILI projects.
Table 9. Percentage of PIs employing various dissemination mechanisms, and average number of dissemination activities per award: 1990 and 1992 grants
|
|
|
|
1. Papers or presentations at regional or national association/ meetings | 54 | 1.7 | 1,807 |
2. Workshops or short courses | 26 | 1.1 | 1,158 |
3. Speeches or colloquia at college campuses | 31 | 1.1 | 1,132 |
4. Publications in professional media (journals, newsletters, etc.) | 31 | 0.8 | 909 |
5. Total (1 to 4) | 67 | 4.6 | 4,940 |
6. Informal responses to individual inquiries about project | 65 | 6.6 | 7,088 |
SOURCE: 1995 mail survey of ILI grantees.
Many PIs commented during the site visits that they use the Internet as a vehicle for sharing project-developed laboratory experiments and other curricular materials. For example, the Computer-Integrated Process Engineering Laboratory at University of Florida is linked to the SUCCEED23 (Southeastern University and College Coalition for Engineering Education) consortium through the Internet. It is difficult, however, to quantify precisely the cumulative national and worldwide impacts of this rapidly growing means of dissemination. Two of the 15 tracer studies that are discussed in the next section used Internet-based dissemination efforts. One of these projects features an Internet web site in which users can access software for 16 computer-based exercises that provide realistic observing experiences for introductory astronomy labs based around simulations, digital images, and observations. The site is accessed approximately 30 times per day.
Because ILI equipment also
stimulates student and faculty research, significant numbers of research
products have been developed and disseminated that are directly attributable
to ILI awards. In fact, a greater number of research publications and presentations
result from ILI awards than publications and presentations about instructional
innovations. Respondents to the 1990 grantee survey reported about 2 research
publications and 6 presentations of research findings per project.
Cumulative
Dissemination Impacts
An estimate24 of the cumulative impacts resulting from all grants funded during the first decade of the program shows the program generating a wealth of activities and products: · 8,000 papers presented at national and regional conferences; · 5,600 workshops and short courses; · 4,700 presentations or colloquia at other colleges and universities; and · 4,200 publications, mostly in the form of journal articles but occasionally also including major products such as commercially published textbooks, laboratory manuals, etc. |
Other community and industry-based groups benefit to an even smaller extent from the ILI program; 17 percent of PIs report involvement in some form of community outreach. Outreach activities such as training workshops for local industry or law enforcement personnel and research partnerships with local health departments, industries, or other community organizations (e.g., environmental assessment, archeological excavations, forensic testing) are included in this class of outreach.
Comparisons of dissemination rates among various subgroups of ILI projects revealed notable differences along three dimensions:
___________________________________________________________________
23SUCCEED is sponsored
by NSF's Directorate for Engineering.
24The extrapolations presented
are based on the average numbers of such products generated by 1990 grantees
during the five-year period following their grant awards. Because dissemination
activities are likely to continue beyond the fifth year for at least some
projects, these estimations are considered to be conservative.
25Instances of dissemination
refers to an aggregation of all reported instances of (1) publications
in professional media(journals, newsletter, etc.), (2) Workshops or short
courses, (3) presentations at other campuses, (4) papers or presentations
at regional or national conferences and meetings, and (5) responses to
information requests.
A Closer Look at Selected
Dissemination Efforts
To obtain a deeper understanding of the impacts that can result from ILI-related dissemination activities, tracer studies26 were conducted with 15 selected projects that were thought to have had significant and traceable27 dissemination impacts beyond the initiating/host institution. The results, though not generalizable, showed that the ILI products generated from these awards were disseminated widely and that a large majority of recipients incorporated the information and material into their undergraduate curriculum.
According to the tracer study
assessment, the form of dissemination activity with the greatest impact
was short courses and workshops for undergraduate faculty. Eleven of the
15 tracer study PIs provided the evaluators with participant lists indicating
that over 900 undergraduate faculty members had been directly impacted
by such workshops. Other forms of dissemination assessed in the tracer
studies and the corresponding numbers of people who were directly impacted
are shown in Table 10.
Astronomy Gettysburg College, Gettysburg, PA, baccalaureate, 1991 Biology San Francisco State University, San Francisco, CA, master's, 1992 Chemistry Hope College, Holland, MI, baccalaureate, 1990 James Madison University, Harrisonburg, VA, doctorate, 1986 Williams College, Williamstown, MA, master's, 1990 Computer Science Rochester Institute of Technology, Rochester, NY, doctorate, 1990 Engineering-Civil Kansas State University, Salina, KS, doctorate, 1990 Engineering-Robotics University of Illinois, Champaign, IL, doctorate, 1992 Geology Harvey Mudd College, Claremont, CA, master's, 1992 Mathematics City University of New York, Borough of Manhattan Community College, New York, NY, associate, 1992 Grinnell College, Grinnell, IA, baccalaureate, 1987 Temple University, Philadelphia, PA, doctorate, 1990 Physics Dickinson College, Carlisle, PA, baccalaureate, 1992. Spinoff project at Joliet Junior College, Joliet, IL, associate, 1991 Lawrence University, Appleton, WI, baccalaureate, 1987 Rose-Hulman Institute of Technology, Terre Haute, IN, master's, 1992 |
Table of Contents
Table 10. Forms of dissemination
assessed
in the tracer studies
and
the number of people impacted
(total 15 projects)
|
|
|
Workshops or short courses for undergraduate faculty | 11 | 928 |
Publications in professional journals | 1 | 206 |
Training courses for private industry/ community groups | 1 | 195 |
Workshops for high school faculty | 3 | 181 |
Papers or presentations at regional or national association meetings | 2 | 147 |
Participation in shared-equipment consortia or centers | 2 | 142 |
Internet | 2 | 100 |
Newsletter | 1 | 82 |
Distribution of plans and/or hardware for laboratory instrumentation | 1 | 9 |
Dissemination efforts were generally quite extensive in the tracer study projects. In most cases, dissemina-tion activities had reached at least 100 undergraduate professors per project, although the total number of dissemination recipients ranges widely from a minimum of 4 to a maximum of nearly 200, depen-ding on the length of time the project has been in operation and the appeal of the project materials and information (i.e., targeting introductory courses or specialized upper division courses). Other audiences besides undergraduate faculty were exposed to ILI-generated products as well. Three projects used their ILI equipment in summer enhancement programs that impacted 150 high school teachers altogether. Nearly 200 law enforcement officials were exposed to forensics equipment and instructional materials related to one PI's ILI project while attending a workshop at the 20th Annual Symposium on Crime Laboratory Development, held at the FBI Training Center in Quantico, Virginia. And one PI who developed sophisticated robotics software with his ILI grant has sold the software to 68 individuals working with robotics in private industry.
For two-thirds of the tracer study projects, the workshop participants and other recipients of ILI-related dissemination report that they readily incorporated the information and materials that they received into their undergraduate courses. Eighty percent or more of the respondents for these projects reported using ILI-related materials in their undergraduate instruction. Adoption rates in the remaining six projects were more moderate, in the 40- to 55-percent range. Depending on the number of courses in which the material was used and the enrollment for these courses, the cumulative numbers of undergraduate students exposed to the ILI projects examined in the tracer studies outside the grantee institution can become quite substantial. In one case, over 150,000 undergraduates from around the country were estimated to have been introduced to the instructional innovations developed from a series of four interrelated ILI grants that involve a workshop-based physics curriculum. Estimates of cumulative number of students impacted for most of the other tracer study projects are, however, more moderate, in the range of 2,000 to 50,000, and the estimated number of students impacted for three projects is less than 500. An overview of the documented recipients of ILI-related dissemination from 15 tracer study projects is shown in Figure 6.
Figure 6. Overview of
dissemination recipients from 15 tracer study projects
SOURCE: 1996 tracer studies
interviews.
Undergraduate faculty who are recipients of dissemination activities often discover how to improve their SMET courses but lack the requisite equipment to do so. Data concerning the number of proposals that were stimulated by exposure to ILI-related dissemination activities suggest that ILI can be a catalyst to proactive laboratory improvement efforts. Among the 300 undergraduate faculty respondents examined in the tracer studies, a total of 100 proposals for laboratory instrumentation were submitted. Fifty-four of these proposals were successfully funded, including 34 ILI grants.
It is interesting to note that most dissemination activities examined in the tracer studies were supplemented or enhanced by other, usually NSF, grants. Dissemination activities in six projects were supported by one or more Undergraduate Faculty Enhancement grants. Five project PIs reported that dissemination activities were supplemented by one or more Course and Curriculum Development grants, and dissemination activities in three projects were based on two or more interrelated ILI grants.
Three interesting examples of successful dissemination efforts found in the tracer study sites are described below.
Tracer Study 1: Impacting Large Numbers of Undergraduate Students. The dissemination activities associated with a 1992 project housed at San Francisco State University (SFSU) entitled "Molecular Biology Laboratory Instrumentation for Undergraduate Instruction" exemplify how a large number of students at the secondary and postsecondary levels from across the country have been impacted by a single ILI grant. For this project, the PIs purchased equipment for use in teaching laboratory exercises illustrating basic molecular biological concepts in both the lower and upper division components of the undergraduate curriculum. The project created a modern molecular biology core facility that will house the major equipment, namely, an ultracentrifuge and rotors for fractionating and purifying macromolecules, a DNA sequencing and computer analysis system, a high technology TV camera with analysis software, a scintillation counter to monitor radioisotopes, and a set of general-use equipment, i.e., microfuges and gel boxes. Most of the dissemination from this project resulted from two types of workshops, one for high school biology teachers and the other for college professors.28 Most attendees gained a great deal of knowledge that they used to their benefit, as illustrated by the following quote:
The experience was so positive to me that I decided to go back to SFSU during sabbatical in 1993 and spent eight months there working with [a co-PI in the chemistry department]. One of the outcomes has been a collaboration with a colleague in our biology department who is a molecular biologist (I am a biochemist in the chemistry department). We designed a new major called Biochemistry and Molecular Biology, which has been well received and has attracted some of our best students. I would not have had the confidence or vision to do this without my work in [the PI's] course.
Of the 192 undergraduate faculty members who participated in the NSF Chautauqua short courses based on this ILI project from 1992-95, 26 were contacted through electronic mail and invited to serve as respondents in the tracer study. A total of 19 responded, providing a 73 percent response rate. Respondents reported that the workshop provided direct experience in the field of recombinant DNA technology which helped "fill in a lot of gaps" that existed in their personal theoretical understanding of the field. In most cases, the workshop provided an impetus for respondents to incorporate recombinant technology into their own microbiology curriculums.
Fifteen of the 19 respondents incorporated information and materials from the workshop into their undergraduate instruction, usually in two or three different courses per respondent. All four respondents who did not make use of the workshop material explained that the content was not relevant to the courses that they teach. Among those professors who were able to apply the workshop material in their courses, the numbers of students per respondent who were exposed to the material ranged from 100 to 700 in most cases, although three respondents reported substantially higher student impacts in the range of 2,000 to 3,000 students per respondent over the course of three years. Extrapolations from the sample to the entire population of 192 faculty members exposed to this ILI project through the NSF Chautauqua short courses suggest that approximately 123,000 students may have been impacted at undergraduate institutions other than the grantee institution during the four years that this project has been operational.
________________________________________________________________
28The workshops for college faculty were supported
by NSF's Undergraduate Faculty Enhancement program
In addition, the PIs facilitated a teacher education in biology program with funding from NSF's Teacher Preparation program. This program provides summer training to high school biology teachers; 68 teachers participated between 1993 and 1995. The PIs also make project equipment available to freshmen and sophomores from the local community college under the National Institutes of Health Bridge for Underrepresented Minorities program. Approximately 60 students from the community college had used the equipment for summer research projects from 1993 to 1995.
Tracer Study Example 2: Exposing High School Mathematics Students to Computer Technology. A 1992 project that represents dissemination efforts that more strongly impact high school teachers than undergraduate faculty was called "PC2: Personal Computers/ Pre-Calculus." The project was housed at the CUNY Borough of Manhattan Community College (BMCC), a large, urban, two-year college whose student body is two-thirds female and over 80 percent minority. In order to attract and retain more women in the calculus sequence, the school secured an NSF calculus reform grant in 1989-91 that enabled them to add computer-based collaborative workshops under the Course and Curriculum Development program to calculus I, II, and III. It became clear to the college that to recruit and retain women and minorities, the program must begin at an earlier level, so they pushed their successful efforts back to the precalculus level. The computers acquired as a result of the ILI grant allowed BMCC to attract and prepare more women and minorities for careers in science, mathematics, engineering and technology. The PI for this project held two types of workshops in which project materials and instructional approaches were introduced: one for mathematics faculty at two-year institutions, and a second for high school mathematics teachers.
A total of 120 undergraduate faculty, nearly all from other two-year institutions, have been the participants in numerous NSF-sponsored workshops conducted by the PI. Of these participants, 29 were contacted through electronic mail or by telephone and invited to serve as respondents in the tracer study. Among the 14 professors who responded to our query, 6 reported using the information related to the project as an integral part of the undergraduate courses they teach. Most of those respondents who did not make use of the material from the workshop reported that it was not appropriate for the courses they teach-remedial math in most cases. One nonuser was interested in implementing the information but was hampered by a lack of equipment; another felt that the material was "too cumbersome and complicated"; and one other "may use it sometime in the future." Among the six respondents who did use the workshop material, the number of undergraduate students exposed to the ideas and materials presented in the workshop ranged from 100 to 200 per respondent.
Of the 33 high school mathematics teachers who attended the PI's Advanced Placement Mathematics Graphic Calculator Training, 15 were contacted by telephone and invited to serve as respondents in the tracer study. All 11 teachers who responded to our inquiries reported using the information related to the ILI project as an integral part of the high school courses they teach. The courses affected include advanced placement calculus, calculus, precalculus, advanced placement math, algebra, and trigonometry. The respondents reported that an average of 117 students were impacted by the ILI project. They also indicated that the calculator training has profoundly influenced their instructional practices as evidenced by the following representative quote:
We used everything we learned at the workshop. She exposed us to many of the different technologies that were out there at the time. Before we attended the workshop, none of our courses had technology incorporated into them. Now technology is incorporated into our statistics, precalculus, and calculus classes.
Many respondents also expressed appreciation for the opportunity to share and interact with their colleagues and indicated that the information was very much in line with the reform movement that is taking place in high school mathematics.
Tracer Study Example 3: Using a Broad Range of Dissemination Activities to Stimulate Further ILI Grant Activity. The PI for a 1986 project entitled "Applications of Lasers in Chemistry" has initiated a broad range of dissemination activities including the development and distribution of 20 laboratory experiments suitable for undergraduate chemistry courses, distribution of low-cost laser apparatus kit plans, publication of a newsletter, and facilitation of an annual short course on laser experimentation. Most of the dissemination activities mentioned above were funded through four Undergraduate Faculty Enhancement grants that were awarded in 1988, 1990, 1992, and 1993. The project was designed to introduce upper level undergraduate chemistry students to the uses of lasers in various aspects of chemistry. Courses in instrumental analysis, physical chemistry, and biochemistry have been enhanced by experiments involving Raman spectroscopy; regular, polarized, and phase-sensitive emission spectroscopy; multiple photon effects; laser vaporization; and thermal lensing that were developed as part of the project. The impacts of this project are far reaching, as demonstrated by the following quote from a participant in the PI's short course:
After attending the laser course, I developed my own lab experiments for the physical chem. lab using the He-Ne laser to measure refractive index, and to measure the rates of reactions. I am currently working on experiments for the pulsed nitrogen laser system. I recently went on a lecture tour in Thailand and gave talks on these experiments at Mahidol University in Bankok, and Chieng Mai University in Chieng Mai.
The PI for this project provided the active mailing list for his newsletter, which contained the names of 82 people (primarily undergraduate faculty) who had contacted him to request copies of his experiments, advice, or a tour of his laser facility. Among the 82 are 14 individuals, 10 of whom attended the PI's short course on laser experimentation and 4 of the PI's former students, who were known to have submitted successful ILI proposals for laser-related materials. Thirty-one individuals were contacted as part of the tracer study, and 24 ultimately responded.
Every respondent who had participated
in the laser short course, and most that were subscribers to the newsletter,
have used the information in an undergraduate teaching and research setting
quite successfully. In particular, those that teach junior/senior-level
physical chemistry courses have incorporated a great deal of the laser
material in their own courses. They report using many of the experiments
developed by the PI as examples of how lasers can be used to study chemical
processes on the molecular level. A total 1,081 of students (excluding
the PI's own students) were reported by the respondents to be exposed to
information and experiments relating to this ILI project. The number of
students impacted through individual respondents ranged from none in two
cases (because the topic was not directly relevant to the courses taught
by one respondent and the other does not have access to the appropriate
laser apparatus) to approximately 75 students each year for a professor
at a state college who incorporated the material into his general chemistry
courses and his two upper division analytical courses after taking the
short course in 1990. An overview of other impacts associated with the
various dissemination activities is presented in
Figure 7.
One of the more productive avenues for sharing information with colleagues that was uncovered in this tracer study was related to activities carried out at the regional meetings of the Association of Chemistry Teachers at Liberal Arts Colleges as the following quotes illustrate:
At last fall's [1995] meeting of the Midwest Association of Chemistry Teachers at Liberal Arts Colleges, we held a group discussion between about 20 members who are physical chemistry teachers. The use of lasers was the main focus of the discussion. Indeed, more than a few of the participants were graduates of [the PI's] workshop so we talked extensively about his experiments. The discussion went over so well that we have now set up an e-mail network of p-chem. teachers to discuss each other's projects with lasers and other equipment.
Two years ago [my school] hosted the annual meeting of the Mid-Atlantic Association of Liberal Arts Chemistry Teachers. I invited [the PI] to present a session on lasers at that meeting. I believe approximately 10 faculty from various colleges in the mid-Atlantic region attended. I do not know how many were able to incorporate [the PI's] suggestions in their teaching, but I do know the session was very well received by those who attended. They especially appreciated [the PI's] sensitivity to their resource limitations, and his thorough presentation.
Figure 7. Impacts associated
with dissemination activities from Tracer Study 3: "Applications of Lasers
in Chemistry"
SOURCE: 1996 tracer study
interviews.
Even those respondents that did not teach a relevant upper division laboratory course nevertheless felt that the knowledge gained from the PI's dissemination activities have greatly enhanced their understanding of the applications of lasers in chemistry, their ability to read and understand the research journal articles pertaining to laser experimentation, and their ability to present current information during their introductory lecture courses. In short, the PI has done an excellent job sharing his knowledge and experiences from the ILI grant.
Yet, the most impressive evidence of successful dissemination relating to this ILI project is the direct assistance the PI provided to 14 other successful ILI grantees. When preparing their ILI proposals, these successful grantees felt that the material in the PI's laser short course was very useful and that the PI himself was an invaluable consultant.
Impacts of ILI on
Unsuccessful Applicants
In order to more tangibly determine how ILI impacts undergraduate education and career development among unsuccessful applicants, we investigated the broader outcomes of the application experience for unsuccessful PIs who submitted proposals to the 1990 and 1992 programs.
Outcomes of Unsuccessful ILI Applications. Nearly two-thirds of the unsuccessful applicants (62 percent) credited the ILI proposal development process with having "stimulated the formulation and refinement of a worthwhile project," the great majority of which ultimately were implemented. Of the approximately 3,000 unsuccessful proposals submitted in the 1990 and 1992 program years, 70 percent of the proposed ideas for improving undergraduate curricula had been implemented to some degree, by late 1995. (See Figure 8 for a breakdown of proposal outcomes.) Nearly one-fifth of the unsuccessful projects received funding support in subsequent ILI grant competitions.29 A far greater proportion of the unsuccessful projects received subsequent funding support from other sources-usually the college or university itself, but occasionally from business and industry, or other federal sources.
Figure 8. Outcomes of
unsuccessful proposals/ ideas submitted to ILI in 1990 and 1992 (N=3,064)
SOURCE: 1995 mail survey
of unsuccessful applicants.
The outcomes reported above suggest that the stimulative effects of ILI extend well beyond the direct and indirect impacts of the projects that are actually funded in any given program cycle. The outcome data also provide an indication of the extent to which the ILI program achieves its objective of stimulating the generation and development of serious, feasible ideas for improving undergraduate education in the sciences.
In spite of these encouraging
findings, nearly half (47 percent) of the unsuccessful applicants did not
feel that the ILI reviewers' comments on their proposals were useful. Although
many unsuccessful applicants reported that the reviewers' comments "constructively
identified weak areas and suggested improvements," a nearly equal proportion
reported that the comments were "confusing," "condescending," "discouraging,"
"excessively harsh," "vague," and "inconsistent" from year to year.
__________________________________________________________
29Forty-six percent of unsuccessful applicants
have reapplied for ILI funding for the same or a closely related project.Forty-two
percent of the ILI reapplications were successful.
Comparisons of Successful and Unsuccessful Applicants. Comparisons were made between successful ILI applicants and unsuccessful applicants for the program years 1990 and 1992 to investigate how responsive the program is to the needs of various types of applicants. Although the successful and unsuccessful applicants were found to be very similar in terms of demographic characteristics such as academic rank, tenure status, and teaching load, notable differences were found among the two groups on the following characteristics:
Several differences also emerged between the two classes of applicants in terms of their prior funding experience and other variables associated with the proposal process. Successful applicants, as compared with unsuccessful applicants, were more likely to have
Table of Contents | Start of Chapter 1 |
Start of Chapter 2 | Start of Chapter 3 |