This document has been archived. Title : ASSESSMENT OF THE NSF 1988-90 UNDERGRADUATE FACULTY ENHANCEMENT PROGRAM (INTERPRETIVE OVERVIEW) Type : Report NSF Org: EHR Date : September 1, 1993 File : nsf93123 ASSESSMENT OF THE NATIONAL SCIENCE FOUNDATION'S 1988-90 UNDERGRADUATE FACULTY ENHANCEMENT PROGRAM INTERPRETIVE OVERVIEW A Statement from the Assessment Advisory Committee Prepared for the National Science Foundation by Westat, Inc., Rockville, Maryland September 1992 INTERPRETIVE OVERVIEW A Statement from the Assessment Advisory Committee Nancy S. Mills (Principal Author), Professor and Chair, Department of Chemistry, Trinity University Neal B. Abraham, Professor and Chair, Department of Physics, Bryn Mawr College Tim Anderson, Professor and Chair, Department of Chemical Engineering, University of Florida Dale Ewen, Vice President for Academic Services, Parkland College William Higgins, Associate Dean, Colleges of Agriculture and Life Sciences, University of Maryland-College Park In 1986, a task force headed by Homer Neal reported to the National Science Board about the state of college-level education in the United States in mathematics, engineering, and the sciences.11 Recognizing that the essential bridge between the schools and the national apparatus for research and development is undergraduate education in mathematics, engineering, and the sciences, the Neal Report made several recommendations. Among these was the recommendation that the National Science Foundation establish "a comprehensive set of programs to catalyze and stimulate national efforts to assure a vital faculty, maintain engaging and high quality curricula, develop effective laboratories, and attract an increasing fraction of the Nation's most talented students to careers in engineering, mathematics, and the sciences." This report led to the establishment of a number of new NSF programs, including the Undergraduate Faculty Enhancement Program. The evaluation of UFEP by the Advisory Committee considered effectiveness in several areas: did the projects funded meet the needs of the participants, did the program as implemented meet the needs of the profession, and were the program and criteria as defined by NSF appropriate to meet the program goals? Impact of the UFEP Projects on the Participants The National Science Foundation believes that faculty members who are current in their field and excited about their disciplines are more effective teachers, and that this is especially crucial in undergraduate education. By this measure, UFEP is a very effective program. Both participants and project directors expressed uniform enthusiasm for the projects in which they were involved. Participants stated that the opportunity to work with the experts in the field was invaluable in their professional development, and that the contacts made in the various workshops with other faculty members, as well as with the project director, proved extremely important in their ability to incorporate the information of the workshops into the curriculum at their home institution. A primary goal of UFEP is to assist undergraduate faculty members in learning new ideas and techniques in their fields, and in using the knowledge and experience to improve their undergraduate teaching abilities. Based on the assessment by the participants and the project directors, this goal was met very satisfactorily. Faculty members found the projects to be highly valuable or worthwhile. The participants felt that the format of the project they attended was appropriate for its topic, but also expressed a concern that workshops longer than 2 weeks were not as accessible to many faculty members involved in summer school or with other constraints. Incorporation of this knowledge into the classroom is an important goal of the projects. Almost all of the participants who were introduced to new technologies at the UFEP project they attended found that the new technologies were useful or applicable to their undergraduate teaching responsibilities. A more important concern is whether faculty members were able to transfer this information to the classroom or laboratory. More than three-quarters of the participants reported modifying teaching methods, introducing new content in courses and laboratories, acquiring new equipment, and incorporating that equipment into undergraduate courses and laboratories. The majority of the participants who had made these changes felt that their experience in the UFEP project influenced these instructional improvements. One barrier to remaining up to date in their disciplines for many faculty members involved in undergraduate instruction is the relative scientific isolation of the faculty member. He or she is frequently the only one in a department in a particular subdiscipline. This problem is particularly acute for faculty members in small departments, although intellectual isolation is also possible in larger departments. Participant data show that for chemistry, biology, and computer science, in particular, the majority of participants came from departments that are relatively small. Thus, an unanticipated benefit of the program has been to reduce the isolation of many faculty members in small departments. Impact of UFEP on the Profession The Advisory Committee was particularly interested in evaluating the effect of the program in the disciplines involved, and in examining that effect in a larger context than the responses of the participants and project directors would allow. As described in the report, information was solicited from professional organizations or groups of faculty members from the disciplines of biology, chemistry, engineering, mathematics, and physics. A major concern of the Advisory Committee was whether the program was funding an appropriate range of topics in each field; the responses of the members of the professional societies varied according to the discipline, with the responses from the physics and chemistry communities representing the extremes of the spectrum. Some members of the American Physical Society Committee on Education expressed some concern about missing topics, suggesting that it may be necessary to target certain topics or to solicit proposals in certain areas. Members of the Committee on Professional Training of the American Chemical Society felt that, in general, the topical coverage was appropriate. They were in favor of allowing the peer review process to select the best projects, rather than attempting to target certain topics for funding. UFEP is currently restricted to programs that are not purely pedagogical in nature, and this was seen by some members of the professional societies as a problem for projects in mathematics and engineering, and to a lesser extent in physics. In these disciplines, members of the professional societies saw a need to improve the skills of faculty members teaching introductory courses. This was seen as especially important in making the workshops valuable to faculty members at community colleges. In addition to favoring a relaxing of the ban on pedagogical projects, some members of the Faculty Development Committee of the Mathematical Association of America expressed interest in projects dealing with general curriculum issues. UFEP might be seen as an excellent avenue to disseminate the results of NSF-supported research to the university/college community, but the members of the professional societies differed substantially on whether they perceive that as a major need of faculty members in their areas. Some representatives stated that since research topics often do not have counterparts in the undergraduate curriculum, faculty members may not need to be exposed to these topics. On the other hand, research topics often provide the justification for the basic preparation provided in introductory courses. Both teachers and students can be excited about covering the basics when they understand what cutting-edge results can be achieved by careful preparation. Faculty members may benefit from the intellectual stimulation of learning about cutting-edge research even when it cannot be directly translated into their teaching syllabi. Some representatives expressed concern, however, that relatively few faculty members from major research institutions were represented among the principal investigators of the projects funded, suggesting that information about cutting-edge research was not being transmitted by the faculty members actually involved in the research. Effectiveness of UFEP in Meeting its Goals The program is designed to assist faculty members teaching undergraduate students by exposing them to new experimental techniques and the way those techniques can be implemented in the curriculum, to recent theoretical developments, to knowledge that cuts across their discipline and others, and to experts in the fields. From the vantage point of both the participants and the project directors, these goals were very effectively met. The Advisory Committee, however, has concerns about the audience reached by this program, with particular concern about the level of professional activity and the racial/ethnic balance of participants. The participant interviews showed that faculty members who were involved in UFEP projects were particularly active, with nearly all having attended professional meetings, seminars, and workshops in the 3 years prior to their participation in the UFEP project. Even more striking, nearly half of the participants had submitted an article to a professional journal in that same period of time. For comparison purposes, one can consider the activity in this area by faculty at a selected set of liberal arts colleges that "have historically expected their faculty to perform research".22 A study of 50 research-active liberal arts colleges from 1979-84 showed that 58 percent of the faculty had published at least one article.3 The UFEP data show, in comparison, that 54 percent of the faculty participants at 4-year schools had submitted an article to a professional journal in the 3 years prior to their attendance at a UFEP workshop, a level of activity very similar to that of faculty members at those 50 researchactive liberal arts colleges. This strongly suggests that a significant number of the faculty members who attended the workshops were already participating in activities that lead to the renewal and maintenance of intellectual vigor, which is a goal of UFEP. We must recognize that even these relatively more active faculty members are in need of the recognition and stimulation of the workshops. However, greater effort must be made to attract participants from the group of faculty members with a lower level of professional activity who can benefit much more from the stimulation of a workshop. On balance, we believe that a mix of experiences, activity levels, and institutional types can lead to the best workshops. Although the participant demographics reveal that minority and women participants were more heavily represented in the workshops than in the population of academic scientists, engineers, and mathematicians, there is still substantial room for improvement in this regard. An examination of the demographics of the United States shows a decreasing percentage of the white males traditionally attracted to careers in the sciences, mathematics, and engineering. The scientific professions must, therefore, be able to draw an increasing percentage of women and minorities in order to "attract an increasing fraction of the Nation's most talented students to careers in engineering, mathematics, and the sciences."4 Minority and female scientists, mathematicians, and engineers play a crucial role in this endeavor and must be given all possible support. There are a number of possible explanations for the lack of participation of minorities and women. They tend to be employed at resource-poor institutions, such as some community colleges.55 Access to the workshops may be limited by the cost of traveling to the workshop, since travel costs are not allowed under current program guidelines; institutional reward systems may not place high value on these activities; or information about the projects may not be reaching these participants. Because of the importance of their inclusion in UFEP, these faculty members may need additional incentives to participate, such as increased stipends or travel funds. Concerns about the effectiveness of recruitment to individual projects by project directors, and the dissemination of information about the program as a whole by NSF, deserve special comment. Project directors used a variety of methods to inform potential participants about their projects, including mailings to deans and chairs and direct mailings to participants themselves, and participants reported receiving information by these methods. There was a strong sense, supported by some anecdotal information from NSF, that direct mailings were the most effective of these methods. It is important that project directors have guidance about the most effective recruiting tools if minorities and women are to be reached. By the same token, it is crucial that information about UFEP and its projects reach faculty members teaching undergraduate students in a timely fashion. Although NSF has consolidated information about programs in a brochure distributed widely, primarily to department chairs and deans, very few of the participants interviewed at professional society meetings had seen the brochure. Many of the participants expressed concern about the timing of the information they received about the program, and the probability that this poor timing had made the workshops inaccessible to certain faculty members. Professional society members from the various disciplines were not familiar with UFEP. Although nearly all UFEP workshops have been fully subscribed, low visibility of the program may contribute to a low rate of participation by faculty overall. If faculty members are aware of a program, they can begin to make plans to attend, even though details of specific projects may not be available at that time. It also became apparent during the course of the evaluation that the data collected by the NSF UFE Program Office in project reports in the initial years of the program were of limited usefulness. The UFE Program Office has begun a more systematic collection of data relating to characteristics of program participants, but a broader, more continuous assessment of the effectiveness of the projects than can be performed from these demographic data is necessary to ensure that UFEP meets the needs of its targeted population. The mechanics of the program operation were evaluated independently by a Committee of Visitors in June 1991, and that evaluation determined that the review process was effective and free of bias, and that the awards were appropriately balanced in regard to number versus size of awards, geographical distribution, and types of institutions.66 According to NSF program officers, UFEP funds are allocated initially according to the percentage of undergraduate faculty members teaching in the various disciplines. Adjustments are then made based on the number and quality of proposals submitted, geographical and institutional distribution, and specific goals of NSF such as the responsibility to meet the needs of underrepresented minorities. The Committee of Visitors felt that the "funded projects reviewed reflected a probability of a large return in terms of undergraduate vitality for the money expended." Recommendations UFEP plays a crucial role in maintaining intellectual vigor in faculty members who teach undergraduate science, mathematics, and engineering courses, and in most disciplines is seen as one of the primary vehicles for faculty development and renewal. During the first 5 years of its existence, the program has reached between 5 and 10 percent of the approximately 162,000 faculty members in these disciplines,77 including those reached through the ASEE and Chautauqua workshops, which are partially funded by UFEP. We recommend that NSF set a goal of reaching one-third of all faculty members teaching undergraduate courses in science, mathematics, and engineering within the next 3 years. This would require that substantial additional resources be made available for UFEP projects. A crucial element in achieving this goal is to access a larger number of potential participants. A larger participant pool will be necessary, of course, if the number of projects funded increases to meet this target, but a larger participant pool is inherently desirable even now because of the potential for increasing the diversity of its members. To enable project directors to more effectively target potential participants, we recommend that NSF develop a project director's handbook that would include guidelines and suggestions for participant recruitment and selection. This might include suggestions on how to recruit and select faculty members who are less professionally active, as well as women and minority faculty members. The handbook should also contain suggestions from previous project directors that will maximize the effectiveness of the workshop, such as techniques for ensuring that participant expertise matches the level of the course, followup activities that create a network of expertise among the participants, and assessment procedures that allow the project director and NSF to assess accurately the effectiveness of the workshop. We recommend that participation by targeted groups be enhanced by encouraging more regional workshops, so that travel costs become less of a burden. Workshops that are regional in nature can also be targeted to areas of the country in which there are large concentrations of faculty members of underrepresented groups, since their geographic distribution is not uniform. Certain types of projects may be hosted at a variety of locations, thus maximizing access to the workshops. Currently, program directors are unable to use UFEP funds to support travel. We recommend that this restriction be lifted in certain cases. Although a number of the participants felt that travel funds from NSF would have greatly enhanced access, we feel that decisions about awarding funds, such as those for travel, are best left to the discretion of the program director. We recommend that NSF take a more active role in enhancing the visibility of UFEP, that they market the program as a whole and make sure that the projects are seen as a part of the whole. One method of accomplishing this would be to require project directors to mention UFEP by name in their recruiting materials. As part of this effort, we recommend that the UFEP staff aggressively seek involvement with other NSF programs, such as the Instructional and Laboratory Instrumentation (ILI) and Undergraduate Course and Curriculum Development (UCCD) programs, as well as the research directorates, by using UFEP as a means for disseminating the results of these programs wherever possible. Coordination between programs is a highly desirable method of maximizing the effectiveness of all programs. We encourage the development of closer ties between UFEP and the professional societies, in part as a mechanism for enhancing UFEP's visibility. In addition, NSF should ask the appropriate professional societies, through their committees on education/training, to take an active role in the solicitation of proposals from their members in order to help ensure that the needs of each discipline are met in terms of topic areas funded. We recommend that the program description be modified to include explicitly the improvement of introductory courses among the areas that may be funded, but we do not recommend that the prohibition of projects of a purely pedagogical nature be dropped. We believe that the primary focus of UFEP should be on the intellectual activity in the discipline rather than predominantly on the methodology of teaching. In order to maximize the benefit of the effort spent in developing workshops, we recommend that multiyear projects be encouraged. Multiyear funding will normally come at the expense of new projects with new content. It is important that a balance be struck in this area, perhaps by a cap on the number of multiyear projects. In addition, we reiterate our encouragement to NSF that the Foundation develop additional means for assessing the effectiveness of the workshops as they expand the number of multiyear projects. It might be appropriate to evaluate the projects nominated for multiyear funding before a second year's funding is awarded. The mode of assessment should be at the discretion of the program directors, but might include site visits as a method for obtaining evaluation by participants. Although we appreciate the desire by many of the representatives of the disciplines to encourage greater participation by faculty members at the premier research institutions as UFEP project directors and instructors, we are not convinced that this would make the workshops more effective. Faculty members involved in the instruction of undergraduate students may require an understanding of a research area at a very different level than that of the preeminent researchers in the field. It is also not always true that the most effective researcher is the most effective teacher. Summary UFEP plays a unique role in the effort of NSF to improve undergraduate education in science, mathematics, and engineering. More so than any other program, it is targeted at the vast majority of faculty members involved in undergraduate education, those who are primarily involved in teaching and for whom professional development opportunities are limited. It is obviously a highly successful program, and we join with the Committee of Visitors in supporting its growth as an important part of the Foundation's overall effort to improve the quality of undergraduate teaching in science, mathematics, and engineering. _______________________________ 1National Science Board. 1986. Undergraduate Science, Mathematics and Engineering Education. NSB 86-100. Washington, D.C.: NSB Task Committe on Upgraduate Science and Engineering Education. _______________________________ 2Ruscio, K. P. The Distinctive Scholarship of the Selective Liberal Arts of Higher Education, 58 (1987):205-22. 3Carrier, S., Davis-Van Atta, D., and Frankfort, F. Educating America's Scientists: The Role of the Research Colleges. Paper presented at the conference "The Future of Sciences at Liberal Arts Colleges", Oberlin College, June 9-10, 1985. _______________________________ 4National Science Board. 1986. Undergraduate Science, Mathematics and Engineering Education. NSB 86-100. Washington, D.C.: NSB Task Committee on Undergraduate Science and Engineering Education. 5U.S. Department of Education. 1991. Digest of Education Statistics. NCES 91-697, Washington, D.C.: National Center for Education Statistics Table 213. _______________________________ 6National Science Foundation. 1991. Committee of Visitors Report, Undergraduate Faculty Enhancement Program. 7U.S. Department of Education. 1991. Digest of Education Statistics. NCES 91-697, Washington D.C.: National Center for Education Statistics Table 214.