Title: Progress of the Engineering Education Coalitions (NSF 00-116) Date: July 5, 2000 SRI International May, 2000 Final Report Progress of the Engineering Education Coalitions By: H. Roberts Coward, SRI International Catherine P. Ailes, SRI International Roland Bardon, SRI International Prepared for: Engineering Education and Centers Division National Science Foundation 1611 N Kent Street Arlington, VA 22209-2192 (703) 524-2053 FAX: (703) 247-8569 Acknowledgments and Disclaimer SRI International wishes to acknowledge the extensive and detailed assistance, as well as the hospitality, of the many Engineering Education Coalition faculty and other participants - especially students, as well as university administrators and staff, who aided in providing information to the study team that performed this effort. Of special note are the efforts of the team from SRI's subcontractor COSMOS Corporation, who performed much of the field research, which involved eleven campus visits by COSMOS and SRI personnel. The encouragement and helpful review of NSF staff members, including Ernest Smerdon, Susan Kemnitzer, and Linda Parker are also greatly appreciated. The views expressed in this report are those of SRI International alone, and do not necessarily represent the views of the National Science Foundation or the U.S. government. TABLE OF CONTENTS Table of Contents i EXECUTIVE SUMMARY iii I. INTRODUCTION AND METHODS 1 A. Background: the Engineering Education Coalitions Program 1 B. Study Methods 4 II. FINDINGS FROM FIELD RESEARCH 6 A. Curriculum and Instructional Delivery 6 B. Coalition-building efforts 7 1. Communication Across Coalitions 7 2. Concrete Operational Plans 7 3. Institutional Commitment 7 4. Reassessment of Goals in Light of Experiences 8 5. Dimensions and Complexity of Tasks 8 C. Assessment Practices 8 D. Impact on Students 10 E. Faculty and Academic Culture Outcomes 12 III. FINDINGS FROM CAMPUS-LEVEL PRINCIPAL INVESTIGATOR INTERVIEWS 15 A. Interactions 15 B. Dissemination 16 C. Challenges 17 D. Broad Issues and Special Concerns 18 IV. FINDINGS FROM STUDENT FOCUS GROUPS 20 A. Coalition Courses 20 B. Computer-based Instruction 21 C. Student Recruitment and Retention in Coalition Programs 21 D. Design Projects 22 E. Teaming Efforts 23 F. Industry Response 24 G. Students' Overall Coalitions Experience 24 V. FACULTY SURVEY 25 A. Coalition Inputs 25 B. Curriculum and Instructional Delivery 27 C. Student Outcomes 29 D. Outcomes: Faculty and Academic Culture 31 VI. SURVEY OF DEANS OF SCHOOLS OF ENGINEERING 32 VII. CONCLUSIONS AND RECOMMENDATIONS 37 A. Overall Findings 37 1. Progress Towards Reform Thus Far 37 2. How Long Should Reform Take? 38 3. Conditions Impeding Reform 38 4. Conditions Accelerating Reform 39 B. Building on the Progress of the Coalitions 39 ATTACHMENTS A. Survey of Faculty Participants A-1 B. Survey of Deans/Academic Affairs B-1 EXECUTIVE SUMMARY The need for significant changes in the character of engineering education emerged in a series of studies beginning in the mid-1980s, culminating with the 1989 Belmont Conference on Imperatives in Undergraduate Engineering Education. The National Science Foundation (NSF) responded to this challenge in 1989 with the announcement of the Engineering Education Coalitions, an initiative to support a small number of major Coalitions of U.S. institutions in a multi-year effort at a level of $2-3 million for up to five years to pursue the vision and goals outlined in their proposals within the NSF Program's objectives. Over the course of three annual competitions, NSF funded six Coalitions. Program goals included: - a dramatic increase in both the quality of engineering education and the number of degrees awarded in engineering, including those to women and underrepresented minorities; - the design, implementation, evaluation and dissemination of new structures and approaches affecting all aspects of undergraduate engineering education; - the establishment of new linkages among all types of U.S. engineering institutions, large and small. At the mid-term of the Program's prospective life span, NSF desired a review that would synthesize the broad variety of information available on this complex endeavor in order to provide a description of the progress of the Coalitions Program during its first five years of operation. The intent was not to evaluate either the Program overall, or any individual Coalition or its members, but to capture an overview of the progress of the coalitions collectively toward achieving the Program goals. Consonant with the complex character of the Program, SRI International, working with COSMOS Corporation, conducted a multidimensional study that used a variety of information gathering tools and sought to derive from the results a comprehensive picture that would meet NSF's requirements. The elements of the study consisted of: 1)Archival review; 2)Field visits to the lead and one other member institution in each Coalition; 3)Focus group interviews with students using Coalition- developed products; 4)Telephone interviews with the Coalition leader at institutions not visited; 5)A survey of faculty participants in Coalition projects at the various member institutions; 6)A survey of deans of engineering schools and colleges not involved in a Coalition. The time-frame of the study was essentially the prospective midpoint of the Coalitions' life span - 1997-1998 - the point at which they were being reviewed for their second phase of funding, depending on which competition had funded the individual Coalitions. The Coalitions Program has had many important impacts during the first five years, but these cannot be said to be "the comprehensive and systemic new models for engineering education reform" anticipated. Most impacts had been intra-institutional, indeed, intra-disciplinary. Participating institutions cover less than one-third of engineering faculty and one quarter of engineering students in the United States, and not all of their engineering faculty and students participate in Coalition- influenced courses. Dissemination is nominal at this stage: there have been many conference presentations and much exchange of information, but the Dean survey showed very limited evidence of actual adoption outside participating institutions, and there remains resistance within even those, ranging from the resistance of conservative faculty to the "not invented here" syndrome. Participating faculty are enthusiastic and generate a wide variety of innovations, but are not temperamentally suited, nor professionally motivated, to generate the follow-on documentation that aids in adaptation or adoption at other institutions. Coalition products generally get good reviews, but primary interaction between the innovators and the rest of the community is fleeting, documentation limited, and adaptability widely questioned. Costs in terms of faculty effort, equipment, and customized classroom configurations are high. Coalition courses add to student workloads at a time when there is pressure to reduce the number of credit hours needed for an engineering degree. Information on student outcomes was derived primarily from a survey of faculty participants and a series of focus group interviews with students, which were carried out during the field research visits. Faculty regarded students taking Coalition courses as rating highly on Coalition-related criteria such as working in groups and tackling interdisciplinary projects. Both surveyed faculty and student groups agreed that the workload associated with Coalition-based courses was heavier than traditional ones, and was perceived as a negative by both groups. The students, however, generally recognized that the effort had given them valuable experience, often to their advantage in the job market and other long-term professional prospects. Many problems were identified in the course of the study, but the Coalitions have made progress. Many materials have been developed, pedagogical research legitimized as part of the academic reward structure, and, perhaps most important, significant contributions made to the efforts of U.S. engineering education to meet ABET 2000 criteria. The complexity of the task of "comprehensive curriculum restructuring" is perceived by Program participants as being greater and more difficult than envisaged in the time-frame imposed on the original Program. However, given the complex problems facing undergraduate engineering education when the Program began and the changes that have occurred since then, the Coalitions Program was widely viewed as the right thing to do at the time, and as having provided the critical funding increment to get things moving. There is, in fact, widespread concern about how the momentum initiated by the Program will be sustained when Coalition funding is phased out. With several years of funding remaining in the Coalitions Program, there are important opportunities to build on the progress that has been made. A significant base of enthusiastic human resources has been developed, as well as a wide range of products that meet Program criteria. The Program can seek to enhance the adoption of the best work produced by assisting in selecting the "best of the best" and encouraging the types of documentation and dissemination efforts that will facilitate the spread of these products through the engineering education system. I. BACKGROUND AND METHODS A. Background of the Engineering Education Coalitions Program The need for significant changes in the character of engineering education emerged in a series of studies beginning in the mid-1980s, culminating with the 1989 Belmont Conference on Imperatives in Undergraduate Engineering Education. Among the needs identified during this period were: a shift from disciplinary approaches toward broader synthesis of knowledge and interdisciplinary modes of operation, including the ability to work in teams; a systems approach to problem-solving; greater knowledge and consideration of social, environmental, and other implications of problem-solving efforts, along with management skills and a capacity for life-long learning; increased ability to attract and retain students in fields of engineering, as well as increased participation of under-represented groups; more emphasis on engineering practice, including early opportunities to encounter the design process; and interaction with industry. These needs were generally viewed within the context of concerns about the ability of U.S. industry to be competitive in the emerging global economy through rapid and successful innovation. The National Science Foundation (NSF) responded to this challenge in 1989 with the announcement of the Engineering Education Coalitions, an initiative to establish a small number of major Coalitions of U.S. institutions in a multi-year effort at a level of $2-3 million for up to five years, to pursue vision and goals outlined in their proposals within the NSF Program's objectives.1 These included: - a dramatic increase in both the quality of engineering education and the number of degrees awarded in engineering, including those to women and underrepresented minorities; - the design, implementation, evaluation and dissemination of new structures and approaches affecting all aspects of undergraduate engineering education; - the establishment of new linkages among all types of U.S. engineering institutions, large and small. Over the course of three competitions, NSF funded six Coalitions. The Coalitions were implemented through co-operative agreements, a funding arrangement between NSF and the lead institution that included monitoring of the Coalitions by NSF throughout the five-year term of the award. While awards were originally intended to be for a single term, NSF decided to allow each Coalition to submit a proposal for an additional five-year award as they neared the end of their original funding. NSF monitoring included annual site visits to Coalition institutions. An in-depth review occurred during the third year that determined whether to continue full funding or phase out funding during the remaining two years. In the fifth year of each Coalition award, each Coalition had the opportunity to submit a proposal for up to five more years of funding, which was required to focus on plans for evaluation, institutionalization and dissemination. Review of these proposals determined the level of support that NSF would provide during a second five-year award. Each Coalition received on average $2 to $3 million per year. While collectively this represents a large block of money, field interviews pointed out that the structure of Coalitions, with a lead institution's overhead and multiple institutional members conducting multiple projects, meant that only small amounts - on the order of several thousand dollars - flowed out of the end of the pipelines to individuals and/or teams for specific projects. The first competition funded two Coalitions, ECSEL and Synthesis. ECSEL, which stands for Engineering Coalition of Schools for Excellence in Education and Leadership, was a geographically and structurally diverse set of schools intent on the desire to incorporate design throughout undergraduate engineering education, as well as the recruitment and retention of women and underrepresented minorities in engineering. Its membership included: ECSEL (Funded 1990-94) Howard University Massachusetts Institute of Technology Morgan State University CCNY of The City University of New York The Pennsylvania State University University of Maryland (current lead) University of Washington Synthesis was the second Coalition established in 1990. Like ECSEL, geographically and structurally diverse, Synthesis has one of the most abstract and broad visions of all the Coalitions. Founded on the "synthesizing" of the teaching and learning process and its facilitation through advanced information technology. More concretely, it focused on courses in "mechatronics," the merging of mechanical and electronic devices in contemporary engineering. Synthesis membership included: Synthesis (Funded 1990-94) California Polytechnic State University-San Luis Obispo Cornell University Hampton University Iowa State University Southern University Stanford University Tuskegee University University of California The second competition resulted in the 1992 funding of the Gateway and SUCCEED Coalitions. Gateway sought to develop a combination of curriculum integration and development (CID), faculty and student development (human potential development - HPD), and "embedded educational technology" (ETM). Its starting point was the "E4" integrated curriculum developed at Drexel University. Its members were: Gateway (Funded 1992-96) Columbia University Drexel University New Jersey Institute of Technology Polytechnic University- Brooklyn, NY The Cooper Union The Ohio State University University of Pennsylvania University of South Carolina SUCCEED is a geographically coherent Coalition, the "Southeastern University and College Coalition for Engineering Education." Its strategy was the development of a model "curriculum 21" aimed at the 21st century that combined an integrated engineering core with the sciences and humanities to be spread and institutionalized through its membership. The latter included: SUCCEED (Funded 1992-96) Clemson University Florida A&M/Florida State University Georgia Institute of Technology North Carolina A&T State University North Carolina State University University of Florida University of North Carolina-Charlotte Virginia Polytechnic Institute and State University The third competition funded Foundation and Greenfield Coalitions beginning in 1993. Foundation Coalition was thematically based on an integrated freshman curriculum originally developed at Rose-Hulman Institute of Technology, combined with similar development efforts at Texas A&M, which was to be adapted to its diverse membership and its concepts extended into the upper division and the full engineering curriculum. Membership was unusual in including a community college and the need for articulation of courses. Members of Foundation included: Foundation (Funded 1993-97) Arizona State University Maricopa Community College District (MCCD) Rose-Hulman Institute of Technology Texas A&M University-College Station Texas A&M University-Kingsville Texas Woman's University University of Alabama The Greenfield Coalition for New Manufacturing Education is unique. A principal feature of the Coalition is its relationship to Focus:HOPE, an organization focused on civil rights and human resource development, located in Detroit, Michigan, that serves the non-traditional students who are often ignored by academia. The mission of the Coalition includes developing three new programs that focus on manufacturing engineering and engineering technology. All but one of Greenfield's active members (Lehigh University) are located in the Detroit area, and its manufacturing focus is primarily on the needs of the automobile and related industries. In addition to Focus:Hope, the members are: Greenfield (Funded 1993-97) Lawrence Technological University Lehigh University University of Detroit Mercy University of Michigan Wayne State University It is worth noting that three of the Coalitions were curriculum "model based": Gateway and Foundation with existing models developed by one or more of their members, SUCCEED with its objective of an integrated Curriculum 21. The other three were "theme based": ECSEL, Synthesis, and Greenfield were more conceptual in their basis, although oriented toward curriculum reform and development. B. Study Methods At the mid-term of the Program's prospective life span, NSF desired a review that would synthesize a broad range of information in order to provide a description of the progress of the Coalitions Program. The intent was not to evaluate either the Program overall, or any individual Coalition or its members, but to capture an overview of the progress of the coalitions collectively in toward achieving the Program goals. Consonant with the complex character of the Program, SRI International, working with COSMOS Corporation, conducted a multidimensional study that uses a variety of information gathering tools to derive a comprehensive picture that would meet NSF's requirements. The elements of the study consisted of: 1)Archival review; 2)Field visits to the lead and one other member institution in each Coalition; 3)Interviews and focus groups of students using Coalition- developed products; 4)Telephone interviews with the Coalition leader at institutions not visited; 5)A survey of faculty participants in Coalition projects at the various member institutions; 6)A survey of deans of engineering schools and colleges not involved in a Coalition. A study strategy had to be devised to provide an intellectual framework that would pull the wide-ranging aspects, activities, outputs and their impact of this extremely complex program. This involved developing an overview of program inputs, activities, outputs and their impacts. To accomplish this, a networking meeting was held early in the project, bringing together the study team, representatives from each of the Coalitions, and NSF Program staff. The objective of the meeting was to lay out the overall dimensions and context of the program being assessed, along with its varied outputs and expected impacts, aiding in the identification of contextual issues by forcing a description of the attribution of causality among the sequences of interventions and desired outcomes of the program.2 This report is organized around the various data gathering methods used. Generally speaking, each method considered the identified program elements: - Inputs, such as Coalition building efforts and faculty participation in Coalition-sponsored projects; - Outputs, including curriculum materials and instruction techniques, and assessment efforts; - Outcomes with students, faculty and changes in academic culture; and - Impacts, in terms of the Program's goal of introducing comprehensive changes and restructuring of undergraduate engineering education. With the first Coalitions having been funded in 1990 for an initial period of five years, the time frame of the data gathering aspects of the study occurred during a period shortly after NSF began the mid-term reviews of the Coalitions for their second five-year term of funding. Specifically, these took place during the following periods: 1) Field research visits (including student focus groups): April, 1997-February, 1998 - most in the fall of 1997; 2) Faculty survey: May 19, 1998-March 11, 1999; 3) Engineering school dean survey: September 29, 1998-March 22, 1999. While the full range of data collection required nearly two years, it should be emphasized that the study could only capture a picture of the progress of the Coalitions within a limited period near the mid-term of the program. Moreover, a "snapshot" of the Coalitions was all that was intended in carrying out the study: it was in no way intended to evaluate the program overall, the individual Coalitions, or any of the members of the Coalitions. II. FINDINGS FROM FIELD RESEARCH A. Curriculum and Instructional Delivery Two types of products were of special interest to NSF and therefore the main topics of data collection: curriculum and related practices (courses and instructional delivery systems) and assessment practices (see next section). The main question of interest in terms of curriculum and related practices was the extent to which the Coalitions extended the state-of-the-art of engineering education, and whether the extensions appeared to be modifications, incremental advances, or truly novel innovations. The Coalitions supported the development of many new curriculum practices, mainly taking the form of revised or new courses in engineering. In general, curriculum practices and other related activities at the Greenfield Coalition must be considered apart from the other five Coalitions. Greenfield supports development of a new curriculum specifically in manufacturing engineering, using computer-based instruction modules in an institutional setting that is distinct from their participating institutions. The other Coalitions are trying to innovate within existing institutional settings. Aside from the unusual organization of the Greenfield Coalition, among the other five Coalitions, all of the new courses demonstrated the variety of features desired by NSF: e.g., students working in teams rather than independently and including cooperative learning, especially in the earlier undergraduate years; increased use of contemporary educational technology, with computer-based methods of delivering courses increasingly taking the place of traditional lectures; and integration of engineering with other disciplines, such as biology, writing courses, and social science and humanities more generally. Many of the courses resulted in increased interactions with industry, with firms sponsoring courses and providing equipment, supplies, and guest teachers. One enhancement is `just-in-time' teaching, whereby concurrent lecture and laboratory courses are sequenced so that lecture topics are covered just as they become needed in the laboratory, whereas in the past the lecture and laboratory courses might have proceeded almost independently. Of all these changes, perhaps the most pervasive innovation across Coalitions was that of providing meaningful engineering experiences for freshmen, ranging from `professional level' laboratory facilities to realistic design projects. (Traditionally, most engineering programs have postponed this experience until the junior or senior year.) Some of the new courses received national recognition for their quality. However, although changes in the freshman year inevitably have some impact on the teaching of upper-level courses, the Coalitions' new courses were only beginning - as of the time of the assessment - to reflect larger strategic shifts and the evolution of large-scale upper-level and cross-disciplinary curriculum changes. Most of the new pedagogical practices were `within campus' innovations, not involving substantial collaboration at the Coalition level. Except for the changes in the introductory courses, most also were limited to one or two departments in engineering (especially mechanical engineering), not the whole school. To this extent, the innovations in curriculum and instructional delivery practices to date may still be considered incremental. Individual faculty have invested a lot of time in improving specific courses and employing Coalition- like principles in doing so; however, the institutional reward system for undertaking such efforts does not appear to have changed, so that pressures for a faculty member to spend more time in doing research do not appear to have diminished. (See Section [VI] for a discussion of the impact of Coalitions on academic culture.) B. Coalition-building Efforts The study considered five types of Coalition-building strategies that were identified in the study's initial meeting with key Coalition participants and NSF staff: the extent and manner of communications among Coalition members; use of principles of experimental design and guidance by concrete operational plans; the dynamics of institutional commitment; reassessment of goals in light of Coalitions' experiences; and the dimensions and complexity of tasks. 1. Communication Across Coalitions Across Coalitions, the diversity of processes and circumstances precludes easy generalizations. Some of the Coalitions worked hard to develop and operationalize such a vision (e.g., Gateway and SUCCEED); others appear to have interpreted the communication goal mainly as an attempt to build bridges across institutions. Even intense cross-institutional communication, in the absence of such a vision, is not likely to advance the interests of the Coalition as a whole. The desired operationalization of the vision usually took the form of some plan for changing the undergraduate curriculum, based on increasing agreement over the key concepts within the vision. 2. Concrete Operational Plans Explicit operational strategies for dealing with the spread and institutionalization of change within individual universities were markedly absent from the Coalition-building process. In theory, the scope of the desired undergraduate reform would suggest that changes cannot be limited to single or a few engineering disciplines (departments); that nearly all engineering faculty must become part of a changed way of doing business; and that some critical changes must occur in schools outside of engineering as well as for an institution as a whole. The field visits and other sources revealed little in the way of explicit plans for this type of spread or institutionalization process, other than as implied by a Coalition's curriculum planning (e.g., lower-level courses first, then upper-level courses). Only a minority of the engineering faculty at any given institution seem to have been aware of their Coalition's breadth and vision, fewer still participated in the Coalition's activities. 3. Institutional Commitment Coalition-building also needs to be fueled by sufficient institutional commitment. There were instances in which one or more deans had made early and sustained commitments to a Coalition (e.g., ECSEL and SUCCEED). Several Coalitions had instances of external funding that supplemented NSF's funding. However, it was difficult to ascertain the extent of "true" commitment by an institution or even an individual. Thus, there were occasions when the field visit team was told that faculty were not encouraged to participate in a Coalition's activities because it detracted from their research productivity. Whether member institutions will continue with the Coalition's work (full reform, going beyond the revisions of individual courses) after NSF funding ends seems questionable. Industry has definitely been involved and appreciative of Coalition efforts, but its level of commitment and involvement is limited. In-kind contributions of equipment and some funds were noted during field interviews, but 42% of faculty survey respondents had worked with individuals from industry in teaching courses and laboratories. Greenfield's direct ties to the automobile industry in the Detroit area was the strongest industrial connection evident, but, as indicated elsewhere in this report, Greenfield has a number of characteristics unique among the Coalitions. 4. Reassessment of Goals in Light of Experiences Feedback about the Coalition-building process did not occur early through formal means, such as evaluation or assessment. In general, evaluation and assessment activities appeared to be isolated (occurring within a single course, department, or institution, but not uniformly across an entire Coalition in any systematic manner). More detailed comments about the Coalitions' assessment experiences are contained in Section C, below. 5. Dimensions and Complexity of Tasks With regard to the dimensions and complexity of the tasks confronting the Coalitions, it should be noted that the Coalitions were confronted with three difficult processes simultaneously: scaling up (if an innovation appeared promising); institutionalizing (within a single institution); and coalescing plans and actions across institutions. Other conditions such as geographic distance and the disbursing of funds through a single institution on behalf of an entire Coalition may have made it harder to carry out these processes well. Further, all of these processes are extremely time- consuming, and it is to the Coalitions' credit that, given extreme dedication on the part of certain groups of faculty and participants, the Coalitions have made the progress exhibited thus far. C. Assessment Practices NSF's intent in requiring Coalitions to institute assessment practices was to ascertain the quality and outcomes of any new curricula and institutional changes initiated by the Coalitions. Such assessment does not appear to have been either an explicit goal from the Belmont Conference or of the Coalitions Program as originally designed by NSF, although it had become an explicit requirement by the second competition. Because of lack of previous exposure to education or social science outcome assessment, some of the Coalitions initially thought that the desired "assessment" was actually quality control as the term is used in industry. This misconception had to be overcome, and, in recent years, both the Coalitions and the broader engineering education community have come to understand what NSF originally intended. The desired type of student outcome assessment is now articulated and emphasized in the new Engineering Criteria 2000 by the Accreditation Board for Engineering and Technology, Inc. (ABET), which will be in use for all accreditation actions by the year 2001. This includes assessment of student learning and employment outcomes. However, NSF also intended that Coalitions conduct assessment of new curricular materials and instructional delivery systems while in the pilot stage and use the assessment results for continuous improvement prior to dissemination beyond the originating faculty developers. The data show that assessment has been slow in starting and a spotty activity among the Coalitions. Nearly all of the available assessments have been conducted within institutions or even within single courses, with little overall strategy for synthesizing lessons learned across institutions or courses to get a broader perspective on a Coalition's work. The paucity of assessment information suggests that it is minimally used for continuous improvement. Further, whether within institutions or courses, the assessments have not been well implemented because the member institutions have generally been unable to overcome such conditions as: - Difficulty in defining common data collection instruments across courses or institutions; - Unacceptably low response rates, whether from faculty members or students; - Infrequent use of or lack of access to archival sources (e.g., student records) that might overcome the previous two conditions better than newly-defined surveys or focus groups; - Inability to define non-Coalition groups of students or faculty for comparison purposes; and - Lack of useful databases to capture more easily Coalition and non-Coalition students' demographic and course performance information. Even when such conditions have been overcome, the assessment results cannot be easily interpreted. For instance, one of the more advanced assessment teams, at Arizona State University (Foundation Coalition), found that more freshmen enrolled in the Coalition program than freshmen enrolled in the traditional program in Fall 1994 were still in engineering two years later (Spring 1996); however, the results were reversed for the subsequent cohort of students (freshmen enrolling in the Fall 1995). When GPAs were compared, those of the comparison group exceeded those of the Coalition group3. The evaluators have been perplexed by these findings and have not yet been able to explain them. Among other possibilities, the baseline equivalence of the two groups needs to be carefully scrutinized. Similarly, some assessment of specific efforts, such as SUCCEED's STEPUP program, cannot be readily interpreted. The local evaluation found that minority students' retention rates and GPAs were similar to those of non-minority students; however, whether this represented a positive or disappointing result was not clearly stated. Possibly typical of the general challenges confronting assessment teams have been the experiences reported by another of the more advanced assessment teams, working with the ECSEL Coalition. The team, led by an excellent group of investigators from one of the Coalition universities (Penn State), admitted to being unable, during the first five years of their work, to collect the type of quantitative data that had been desired across the Coalition's institutions. As a result, an adaptation being tried in the second five years is to pursue a meta-analytic strategy, whereby different instruments and measures may have been used but a common statistic, such as effect size is calculated to aggregate quantitative findings across campuses. However, even within an institution, the ECSEL team encountered difficulty in collecting quantitative data on outcomes, with most of the information coming from a handful of open-ended interviews or from focus groups of students. Whether even good qualitative data (e.g., case studies of the Coalitions) are being collected also may be questioned, however. For instance, only rarely did the study team encounter examples of efforts to track policy changes in the institutions and the role of Coalitions in stimulating such change. For example, one such policy change was reported by the assessment team at Arizona State University without noting specific Coalition influence. This exemplifies the need for more Coalition goal-related qualitative data that could have been tracked more thoroughly by all of the assessment teams. The change was a new policy of `post-tenure review' adopted by the Arizona Board of Regents, whereby all faculty, even those with tenure, will be reviewed annually for their contributions to teaching, research, and service.4 A similar change took place at the University of Florida, where the state established a Teaching Improvement Program (TIP) incentive for engineering faculty. Faculty receive a $5,000 increase over base salary for excellence in teaching. During the past three years, 66 engineering faculty at the University of Florida have received TIP awards. If such policies are faithfully implemented and become more common across institutions, a more general culture change favoring the values of the Coalitions Program may occur. AT THE SAME TIME, AN IMPORTANT CAVEAT IS THAT THE DIFFICULTIES ENCOUNTERED BY THOSE DOING ASSESSMENTS OF ALL KINDS SHOULD NOT NECESSARILY BE CONSIDERED SPECIFIC TO THE COALITIONS PROGRAM. ASSESSMENTS HAVE BEEN SUCCESSFUL, BUT ONLY UNDER SPECIAL CIRCUMSTANCES, SUCH AS A FOCUS ON A SPECIFIC COURSE OR INNOVATION AND WHERE ALL PARTICIPANTS AGREE TO BE PART OF A `STUDY.' FOR UNIVERSITY-WIDE ASSESSMENTS, GOING BEYOND COMMONLY TRACKED GRADE, ENROLLMENT, AND GRADUATION DATA IS A NOTORIOUSLY DIFFICULT TASK, WHETHER WITHIN OR OUTSIDE OF ENGINEERING. EFFORTS SUCH AS THE COLLABORATION BETWEEN GATEWAY AND SUCCEED SINCE 1996 TO WORK TOGETHER TO DEVELOP ASSESSMENT MEASURES FOR NEW EDUCATIONAL INNOVATIONS MAY THEREFORE REPRESENT MOVES IN A PROMISING DIRECTION. D. Impact on Students "Outcomes" represent the first meaningful effects of Coalitions, and these have been examined in terms of three categories: the impact on students, the impact on faculty, and changed academic culture. This section discusses the impact on students - the impact on faculty and academic culture follow in the next section. Student outcomes show many examples of the learning behaviors valued by the Coalitions program - e.g., students gaining exposure to practical problems related to industry, students learning to work as part of teams, students gaining hands-on experiences with computers and other equipment, and students gaining exposure to engineering design during their freshman and sophomore years. At the same time, the listing of outcomes oversimplifies some of the operational challenges in putting the desired processes into practice. For instance, for cooperative learning among students, teaming was done in many institutions but followed different approaches regarding how individuals are assigned to teams; how individuals are coached to work on teams; whether all team members will get the same grade; and how to overcome scheduling problems for teamwork, especially in environments where some students are also maintaining outside jobs. Among the student outcomes of high priority, the Coalitions do not appear to have tracked, systematically, the level of minority and women students and their level of satisfaction or likely success in progressing through the engineering curriculum. Most of the Coalitions have only begun to emphasize new outreach efforts, aimed at recruiting such students in the first place, but are not yet ready to deal with issues of progression and retention. Information about student outcomes tended to come from single institutions or single courses, covering a variety of outcomes potentially related to but not directly covering retention and progression. Figure II-1 lists these outcomes and their illustrative examples. Among the six Coalitions, Greenfield and any outcomes attributed to this Coalition may need to be considered apart from those of the other five Coalitions. Completing the Greenfield program can lead to any of three degrees: an associate degree, a bachelor of engineering degree, or a bachelor of science in manufacturing engineering technology. The first 11 students in the bachelor's programs completed their degrees in the summer of 1998. Early feedback to the evaluation team indicated that the initial graduates were offered exceptional positions and valued highly by the corporate partners, because they had engineering degrees and also had attained machinist level qualifications. While its modus operandi assures a high rate of women and minority enrollment, the attrition rate from the three programs has been about 50 percent. The Coalition claims, however, that nearly all candidates leaving before graduation have been enticed by attractive job offers from industry and not confronted by academic difficulties. Figure II-1 Illustrative Student Outcomes from Coalition Courses, Modules, and Labs E. Faculty and Academic Culture Outcomes In addition to student outcomes, the study design posited that the Program should result in outcomes reflected in changed faculty approaches to teaching and in the overall academic culture of the member institutions. In the study, the faculty outcomes of interest were observations of any shifts from traditional, passive, lecture-based instruction to newer models of active, collaborative learning (e.g., coaches rather than all- wise dispensers of knowledge). The desired academic culture upon which the Coalitions Program is based calls for a shift away from an exclusive focus on individual faculty achievement, disciplinary specialization, and wholly research-based reward structures, broadening the focus to include team contributions, integration and not just specialization of knowledge, and educational (not just research) scholarship and innovation. Evidence of such changes in culture was spotty and possibly premature to expect at this stage. No assessments yet involve case studies or other qualitative examination of such shifts. Further, complete study of cultural shifts also involves collecting data from non-engineering or non-university perspectives (such as industry representatives). Nevertheless, although evidence of desired cultural changes at the time of the study was limited, the field visits and surveys did provide some evidence of changes taking place. These could hardly be characterized as strong trends, but there seemed to be desirable, if relatively recent, developments in four areas. The first and possibly most important pattern across Coalitions was the increased valuing of contributions to the practice of engineering. SUCCEED's Quality Improvement Partnership project called for individual university members each to form a partnership with a local firm, with a preparatory total quality systems course followed by summer internships at the sponsoring industrial site. Students worked in multi-functional and multidisciplinary teams on real production and development problems and created a record of producing significant cost savings for the plants. Similar benefits are reported in SUCCEED's Integrated Product and Process Design course, with feedback from industrial recruiters suggesting that the students have the equivalent of two years' job experience. ECSEL's upper level product engineering and manufacturing course also has resulted in students having their redesigns actually adopted by the partnering firm. A second consistent theme across all Coalitions was the development of cross-institutional and not just cross- disciplinary perspectives and interactions. These include the Synthesis Coalition's mechatronics initiative, as well as the benefits derived from sharing equipment and facilities at other institutions, as in Gateway's scanning tunneling microscope and rapid prototyping machine. SUCCEED's integration of biology and engineering promoted cross-institutional collaboration by deliberately calling for joint proposals from multiple institutions to develop new courses and avoid duplication across campuses. Such collaboration also included cross-disciplinary team-teaching and laboratory development. The articulated transfer of engineering courses from a community college to a university in the Foundation Coalition required inter- institutional collaboration. In a similar manner, the Synthesis Coalition helped one of its members, a historically black college or university, to gain ABET accreditation for its newly established engineering program. Given the overall pattern of the relatively narrow and largely intra-institutional scope of interaction during the Coalitions' early years, these are useful developments. A third potential cultural shift involves outreach to non- engineering students and departments. These include: recruitment of women and minorities into engineering, as in Foundation's mathematics, engineering, and science achievement project; a mix of physics and engineering students enrolled in an upper-level course in the Gateway Coalition; and SUCCEED's biology and engineering collaboration. Foundation's integrated engineering program also involves interaction with the university's English department, and some laboratories, although created mainly for engineering, are being used by a large number of students in the liberal arts, with a number of examples of applications in fields other than engineering. Despite these instances, however, outreach beyond engineering schools has not been one of the stronger elements of Coalition efforts. A fourth shift is evidenced by isolated examples (and, indeed, counter-examples) of shifts away from a research-only culture of promotion and tenure. In ECSEL, one university has adopted a long-range plan moving the engineering school away from a program emphasizing engineering science to one that is more oriented toward the practice of engineering in a broad context. At another ECSEL university, promotion and tenure processes are being changed to include recognition of teaching and involvement in undergraduate engineering. To balance excellence in research with excellence in education, a Synthesis Coalition institution plans to use "excellence in teaching" awards as part of the promotion and tenure process. On the other hand, there also were clear reports of untenured faculty being given the message that tenure decisions will be based primarily on research rather than including education contributions III. FINDINGS FROM CAMPUS-LEVEL PRINCIPAL INVESTIGATOR INTERVIEWS For the purpose of broadening the study's understanding of the impact of Coalition participation on campuses that were not included in the field visits, the perceptions of Coalition leaders at schools not visited were sought by telephone interviews with the campus PI or leader. Where changes in leadership had occurred, the individual with the broadest "institutional memory" was sought out for this purpose. In order to limit the demands of the interview, efforts were made to focus the discussion on: 1)important aspects or special developments in the history of that institution's involvement in the Coalition; 2)interactions within the institution, the Coalition, with other Coalitions, and outside the Coalition community; 3)dissemination efforts; 4)important challenges encountered in the past and those seen in the future. The historical elements having been used in conjunction with other parts of this report, only the last three and some issues that broadly affected nearly all Coalitions are discussed here. A. Interactions Responses concerning the types and focus of interactions engendered by the Coalitions Program usually represented personal observations and experiences, but there were some consistent themes - both positive and negative - and they were generally consistent with the picture that emerged from the field visits. On the positive side, a number of respondents pointed to the utility of travel funds to actually observe what was going on elsewhere. This might be to observe a course, courseware and its classroom use, or computer laboratories prior to attempting to adapt and/or adopt these outside the developing institution. It was also useful for Coalition faculty to observe where cultures of change had emerged at other institutions. This was especially important when the member institutions were widely dispersed. It was reported, for example, that there appears to be more interaction among SUCCEED institutions and the institutional members of other Coalitions in the southeast than with more distant members of other Coalitions: i.e., SUCCEED's geographic focus is the basis for interaction - not the Coalition program. Campus-level funds also permitted Coalition members to share facilities - Cooper Union used Columbia's Coalition-driven computer laboratory, other schools have used the University of South Carolina's Communications Center, New Jersey Institute of Technology, and Ohio State provided access to rapid prototyping facilities to Gateway members. (However, in the last case, the practice was abandoned due to its high costs.) On the down side, it was clear that both intra-Coalition and inter-Coalition interaction was difficult to achieve outside of meetings arranged by NSF, or professional meetings such as the ASEE or FIE. Most that did occur appeared to represent two-way relationships, although opportunity to build a network of acquaintances with similar interests at other schools was often cited as something of great value - another observation consonant with field visit interviews. However, a few campus leaders conceded that their interaction had been largely passive. In particular, one HBCU PI indicated that his campus had done little in the way of carrying out projects under the Coalition, largely evaluating and, in some cases, modifying and adopting Coalition- developed materials. Campus PIs reported that interaction with institutions outside the Coalitions Program was nearly non- existent. B. Dissemination Interviewees were asked about dissemination activities and results. The issue of dissemination was raised in the PI interviews because other sources of information made it clear that dissemination is a serious problem and that it is not simply a problem of gaining broader "buy-in" beyond the originating institution. The telephone interviews replicated observations from the field visits with repeated admissions that dissemination was slow to come, behind schedule, and generally disappointing. The most obvious problem noted during the field visits was the difficulty of providing documentation that permitted transfer. Faculty enthusiasts enjoyed projects developing new courses, projects, and other materials. The problem of getting these documented derived from many factors. The conflicting pressures on faculty, the small amounts of money provided individual projects at the end of the pipeline, and faculty skills at writing and presenting materials outside the classroom all contributed to dissemination problems. Despite the frequent mention of publications and conference presentations in field visits and PI interviews, a paper at a conference does not provide coursework in and of itself. CD-ROMs and computer modules may be informally distributable, but without documentation are neither easily implemented as part of courses nor "commercializeable." The results of the survey of non-Coalition engineering schools seems to confirm that, aside from a couple of textbooks and a few CD- ROMs, Coalition materials seem idiosyncratic and hard to adapt. One problem identified in the interviews is that the Coalitions Program has produced its own Catch-22: it has legitimized scholarly publication concerning curriculum reform, but this takes precedence in the reward structure and tends to preempt any efforts that most faculty are prepared to undertake to work up Coalition products for dissemination. The phone interviews confirmed these problems, as well as suggesting a broader range of issues concerning the problems involved in dissemination. Perhaps the broadest, most important, and frequently noted point is that dissemination must be seen as a process - not just the distribution of a product. One campus PI formulated the process as one of generating awareness first, followed by a need for adaptation, and only then hope for adoption by other institutions. An allied comment was that there was a lot more communication going on than dissemination. Several emphasized the need - and difficulty, even on their own campuses - to get people to recognize the benefits realized by students to just incremental changes, much less "systemic reform." Reform efforts can suffer because of the need, even on a single campus, to mesh iterative efforts to improve courses. The constant "tweaking" of one first-year course was cited as an example of why it was difficult to move on to the next level. What was conceived as a sequential developmental process was actually a parallel one. The team that kept trying to improve the freshman course frustrated faculty and led to an unwillingness to adopt the efforts of the team working on the sophomore course. Continuous improvement can hamper efforts to overcome academic inertia. There is also simple parochialism-the "Not Invented Here" factor. It is hard enough to gain buy-in in one's own institution, much less to get an outside institution to enter into the process of adopting things developed elsewhere. Cost inhibits dissemination. Coalition courses and other materials require additional faculty time, new classroom configurations, computers, and other equipment, often placing additional burdens on students, as well. Greenfield indicated that some of its products (sequences of linked instructional modules) require nearly $1 million per year to maintain, given the need to coordinate, professionally present, and update units. C. Challenges The challenge most commonly identified was that significant, widespread reform of engineering education is harder than originally thought, especially in the time-frame provided by the Coalitions Program. It was difficult to get the Coalitions up and running, particularly the early ones, hard to get buy-in, and hard to scale things up and institutionalize them. This is closely related to the theme concerning dissemination as a process that involved incremental change, not mass adoption of new approaches. Coalition courses are typically expensive to maintain. This makes them hard to transfer, as noted under dissemination, and they may prove hard to sustain after NSF Coalition funding ceases. Administrators are not eager to pick up these expenses. None of the PIs interviewed mentioned the new Action Agenda initiative as a potential vehicle for this, possibly because their continued funding by the Coalitions Program meant that they had not turned their attention in that direction. Beyond cost, there was a broader unease about the prospect of a breakdown of the Coalition culture and a falling away of people with the loss of focus that the Program gave. PI telephone interviews reinforced frequent complaints heard in field visits concerning the problems of administering funds. Projects were proposed to a Coalition and accepted. They often then had to be started up "on spec" since the funds were slow to arrive - and sometimes did not show up until the course or project was over. There was broad agreement that the Coalitions were slow to move on the assessment requirement. Many got off to a bad start and lacked the internal resources of ECSEL's Penn State Center for the Study of Higher Education, or the participation of the Columbia Teachers College analysts. The prospect of ABET 2000 helped as both a spur and a challenge, with frequent comments concerning how helpful Coalition participation had been in preparing for that assessment process. Most Coalitions encountered difficulties in persuading an "old guard" of the necessity of change in the undergraduate curriculum, and most had activities concentrated in a very limited number of departments: the emphasis on design usually meant that the Department of Mechanical Engineering was the key participant. This led to another cost barrier when trying to scale-up or institutionalize new courses, and to move up to more advanced courses, which often involved other departments. The phone interviewees also noted the difficulty in dealing with the Program's goal of increasing the number of degrees awarded to women and underrepresented minorities. For some it was a matter of student constituencies or state populations that made this difficult. For others, the problem was one of developing the skills needed for outreach and retention. For HBCUs, already focused on a minority constituency, it was irrelevant. Complaints were frequent concerning the NSF pressure at a time when affirmative action programs are losing public support, and some state laws putting some schools in a legal bind. The remarkable number of undergraduate women at MIT - about 40% - was primarily attributed to an initiative of the Admissions Office that predated the Coalition. Outreach to K-12 was similarly difficult in terms of skills and other pressures on faculty, although some schools viewed Coalition summer programs as being their best recruitment devices. Some local K-12 involvement was feasible, but broad efforts were precluded by other pressures, especially in research universities. D. Broad Issues and Special Concerns Both phone interviews and field visits pointed to the variety of converging influences that made it difficult to isolate the influence of the Coalitions. ABET was cited as an especially strong influence, but there had been a general sense of the need for reform at the time the Coalitions Program emerged and that sense was embodied in other programs funded by a variety of organizations. However, given the complex problems facing undergraduate engineering education when the Program began and the changes that have occurred since then, the Coalitions Program was widely viewed by the campus PIs as the right thing to do at the time, providing the critical incremental element to get things moving and legitimizing change and pedagogical scholarship. There is, in fact, widespread concern about how the momentum initiated by the Program will be sustained when Coalition funding is phased out The telephone interviews were consistent with survey and field research information concerning issues involving younger, untenured faculty. Experiences ranged from an institution's deliberately discouraging participation by untenured faculty to being able to cite cases in which individuals were specifically helped in obtaining tenure through their Coalition work. Initially, one Coalition as a whole was reported to have deliberately limited participation to senior faculty, largely with the goal of establishing an early track record. In another, initial discussions about joining the Coalition left only two faculty members, both junior, interested enough to move the school into the Coalition, and they were discouraged from participating. One prospered as a result, the other at minimum did not benefit. Pressures are particularly strong at schools with ambitious administrations that place emphasis on advancing the school's research reputation. At one such institution, however, Coalition membership was cited as an important influence in holding back an administration-sponsored rush from the classroom to the laboratory. The need for enthusiastic, energetic, and charismatic leadership was constantly mentioned as critical to getting a Coalition started and maintaining its momentum. However, such leadership is not sufficient without support, both in terms of policy, encouragement, and supplementary funds from both engineering school and university administrators. IV. FINDINGS FROM STUDENT FOCUS GROUPS The response of students to their experiences in courses influenced by Coalition activities was investigated by conducting student focus groups at each field visit campus. These meetings were not "focus groups" in the formal sense of the term because the student volunteers could not always be assembled in a manner that permitted a consistent duration of structured topical discussion. A number of consistent themes emerged, however, among the student groups interviewed at the eleven campuses visited . A. Coalition Courses The most common observations were that Coalition-related courses were considered to require more work, could be hard on Grade Point Averages, and could put a separate Coalition cohort at a perceived disadvantage in comparison with their classmates. Such observations about workloads require some type of baseline and were thus limited to campuses where the local operational approach made courses or cohorts identifiable. At the same time, it was common for students who made such observations to insist also that, hard as the effort had been, they recognized what it had done for them and that it often provided them with educational advantages over their classmates that outweighed the costs in terms of time, effort, and GPAs. Students became aware that Coalitions had learning curves. The inferiority of early efforts to later developments was noted. Those making such remarks had often suffered the initial experimentation, moved on through the curriculum, and were observing the improvements with a degree of envy. Nor were students shy about noting that some faculty were more adept than others in their efforts to deliver improved courses - something especially evident when more than one section of a course was being taught. The student network was well informed and highly judgmental. The students found that variations among the participating faculty were important. Their reactions to courses varied, depending, for example, on whether a student had taken the course in a section with professor A or professor B. Usually, this reflected differences in personality or the degree to which the faculty member's commitment and/or teaching style were consistent with the Coalition goals. It was not a matter of individual professors being assigned to teach courses not having bought in on the Coalition values: even committed volunteers could vary greatly in their ability to translate their interest into classroom enthusiasm and results. Particularly at research- intensive universities, there was a conflict between the faculty member's interest in curricular change and a university culture that was at odds with the reality that developing and using Coalition materials and teaching Coalition courses required extra time and effort in the classroom, especially in computer-based courses. B. Computer-Based Instruction The use of computers in instruction in general, as well as reactions to specific commercial programs, was frequently the subject of criticism by students, but Coalition-designed modules were seldom a negative topic. Perhaps one of most interesting themes was the students' concerns about the effort that it took "learning to learn programs." This took various forms. At minimum, they felt that the faculty were inattentive to the time and effort needed to learn a new application. In other cases, the problem was that time was taken to "learn" an application like Excel early in a course or course sequence, but that the content of the training was trivial or irrelevant to its potential uses. When they really needed to use the program, they did not have an adequate background to tackle the problems posed. Specific programs were liked, disliked, critiqued, or thought to be out of synch with industry. The math instruction program, MAPLE, seemed to be very commonly used and much disliked. Some CAD/CAM programs common in the universities were viewed with contempt by students who had co-op, summer job, or intern experiences: they often felt that what was being used in their courses did not represent the direction in which industry was moving. Students would point to changes in versions of software that the university seemed unable or unwilling to upgrade. Coalition money often represented the means for substantial investment in computer equipment and software, but the pressures of obsolescence are constant. The students showed great acuity in their software reviews and expressed frequent frustration with the inability of new curricula to keep up with changes or new developments in commercial programs. C. Student Recruitment And Retention In Coalition Programs Students were recruited into Coalition programs, courses, and other activities in a variety of ways. For example, Drexel had spent some years developing its E4 lower division curriculum, which was to serve as the main basis for Gateway's efforts to reform and institutionalize new curricula, first at the lower division level, then the upper. Since matriculating Drexel students were automatically enrolled in the E4 courses, the school had a head start on other Gateway institutions that were often less willing or able to move quickly to put the curriculum in place. Foundation and SUCCEED worked on the basis of initially taking a limited number of students into their changing lower division courses, so that there were parallel cohorts within a given class, some identifiably in the Coalition curriculum, others in the traditional one. Faculty interviews at field visits usually described screened invitational processes, where at least some entering freshmen were invited to presentations and asked if they would be interested in the alternative curriculum. Some students considered there to be a coercive element in the process that almost obligated them to go the Coalition route. Several subsequently found greater satisfaction in more traditional courses. One orientation program for entering freshmen produced oddly symmetric results: one student entered at his parents urging on the grounds that the hands-on, team- oriented approach described would be good for him, while another opted for the program over his parents objections to its experimental nature. The Coalitions have varied in the degree to which they pursued and achieved a specified "Coalition" curriculum, as well as the levels to which reforms had penetrated (lower division more likely than upper, senior project or design courses more likely than other upper division courses). Some students were recognizable on campus as "Coalition" students and able to compare their experiences (generally favorably) with classmates who were not. In other cases specific courses were recognizably Coalition-derived by the degree to which they had been changed to include Coalition elements like design, teaming, and hands-on experiences. Synthesis was almost invisible at Berkeley: the amount of influence it had on "Synthesized" courses depended upon which faculty member taught the course and how much Coalition developed material was included. At California Polytechnic, San Luis Obispo, on the other hand, the Coalition was generally a visible entity, but its objectives were so compatible with the school's tradition and culture as an undergraduate teaching environment that Coalition project implementation blended into the scenery. An important issue related to student retention and to institutionalization emerged at several schools - the need for documentation, or a "road map" of related programs. Few schools have institutionalized Coalition-based reform beyond a number of common lower division courses. This represents a particular difficulty where articulation problems arise when an engineering school receives a substantial influx of its enrollment from community or junior colleges. It seemed clear that if the Program's goal was "systemic reform," students who liked the reforms wanted a map of the "system" (i.e., curriculum sequence) - just as those who preferred more traditional approaches felt they needed to know how to exercise their preferences and still meet degree requirements. E. Design Projects The themes of design and "engineering up front" were of great appeal to most of the students. Many of the group participants were quite articulate about the role that these had played in their recruitment and retention in engineering. For example, one drop-out who had returned to engineering after several years of military service, thought he would have been less likely to have dropped out had he had more of the Coalition- type courses to begin with that he experienced on his return. Some cynicism was evident about the apparently trivial character of some of the freshman design experiments used to implement these concepts. One group pointed to the "game-like" character of design projects involving bagel and hot dog "launchers," involving materials unlikely to be encountered in the real world. Yet the same group expressed enthusiasm about this freshman design course, regarded as the only real engineering that they would see for some time, and found it a useful entry on their resumes when seeking summer employment. Many saw the need for "real world" and "hands-on" opportunities early in the curriculum in order to hold their interest through the numerous tedious, more theoretical courses required later in the curriculum. F. Teaming Efforts Teaming was widely recognized by the students as an essential part of their education and something that was a real asset in their encounters with industry. It was also one of the more difficult aspects of Coalition goals to implement, and it was implemented in a variety of ways that often evoked spirited discussion of common complaints. These were often offset by one or more of the group pointing out that the real world was a place in which work-mates were not always congenial team partners. Coalition faculty put teams together in a variety of ways. Some used personality tests, GPA distributions, or, especially on campuses with a large heterogeneous commuter population, age groups. Some picked teams randomly, while others tried to ensure a mix of skills. Gender often played at least an initial role. Many faculty were leery of having but one woman on an otherwise male team and assigned two per team as far as possible, often abandoning this concern once students had had some teaming experience. Within limits, self selection was sometimes permitted. The students did not seem to have any sense of which method worked best, although they liked least those in which they viewed themselves as having been manipulated. Given the problems that most had encountered in their teaming efforts, students appreciated efforts to prepare them for it. Within one Coalition, a field visit observed a class being braced for the social nuances of team interaction in one of a series of class periods devoted to this. The student group at the other campus in the same Coalition complained about the lack of any preparation. There were frequent complaints of the difficulty of gaining faculty intervention when serious problems within a team surfaced. The teaming problems most subject to complaint were consistent. Establishing leadership and how to deal with slackers dominated student comments. Teams appear to have usually been left on their own to deal with their internal organization, division of labor, and scheduling within the project deadline. In that sense, the experience lacked the real world hierarchical assignments that would usually be made in industry. The problem of slackers was a major irritant, and most of the groups cited examples of team members who essentially disappeared as the project was supposed to be moving forward. Special problems could arise in the context of commuter campuses or community colleges. Students often had job and/or family obligations that made scheduling team meetings difficult. Heightening the tension was the problem that most faculty appear to have adopted a policy of collective grading: i.e., the project grade was given collectively to all team members. Greater satisfaction with the grading process was evident when some sort of internal team evaluation could influence the grade distribution in a way that reflected perceptions of relative contributions. >From the student groups, one got an impression that teaming was readily accepted into Coalition courses, but that faculty were loath to deal with most of the problems that arise in the educational environment. Most took a laissez-faire attitude once the rules on grading were established and their personal method of establishing the teams set. Faculty input during field visits seemed quite complacent in terms of awareness of problems compared to student reactions to the ways in which teaming was handled. Despite the problems involved in teaming, student responses were overwhelmingly favorable. The overall thrust of the groups was one of great consciousness of the importance of the experience for their long-term career interests - their "preparation for the real world" being a phrase often used. Also, a number of the student groups noted that the team approach often spread among their fellow students. The value of team efforts was becoming apparent in the sense that not only students involved in engineering courses requiring teaming were picking up on the cooperative approach. Coalition engineering students continued to engage in informal teams for a variety of courses and problems, and the practice spread to non-engineering students in some cases. F. Industry Response The student groups were positive about the ways in which their institution's involvement with industry had increased, as well as the response of industry to the impact of new Coalition curricular elements on their skills. The groups were enthusiastic endorsers of the impacts that their Coalition work had had on their efforts to find jobs and internships. In job interviews, industrial recruiters were almost always impressed by students displaying a variety or some combination of factors reflecting the Program's goals - working in teams, communications skills, and the like. Communication skills were an important factor: students reported that their experience in making presentations in Coalition courses was recognized by industry in their job-seeking activities. The down side in terms of curriculum development was a clear discontent with the ways in which Coalitions attempted to provide those skills by linking technical courses to review by humanities instructors, linking grades to judgments of faculty members trained in writing and other communications skills, but lacking understanding of the technical content of the material involved. The implementation of training in communications was not always sensitive to the tension between the standards normally applied by humanities teachers and the technical content involved, especially in grading. G. Students' Overall Coalitions Experience An effort was usually made to ask the students to assign a quantitative ranking to their "Coalition experience." That phrase was used because of the variety of experiences in Coalition's context that students had. The comings and goings of students during focus group sessions sometimes made this difficult, and the results, should be taken as a non-rigorous set of ratings. Despite the candid critiques voiced in the student groups, when asked to rank their experience on a scale of one to five (five being the top), they gave the Coalitions high marks. >From the top down, one received a rating of 5, two 4.4s, one 4, and two 3.9s. V. FACULTY SURVEY In order to broaden the study's perspective on the activities of the many faculty who had participated in the variety of Coalition activities that had taken place at member institutions, a sample survey was conducted of faculty participants across the six Coalitions. The random sample of 302, drawn from lists of participating faculty provided to SRI by the Coalitions (a total of 1151 faculty names, the sample representing 26%), was given a choice of responding via the web or printing out the survey to preview it and/or submit it in hard copy, or, through an SRI contact number, to receive a hard copy by mail. The overall response rate was slightly over 50% when the survey was closed. From the 152 responses received to the survey, 18 (12%) indicated that they had never participated in Coalition activities. Eight of those (5% of those responding) were not even aware of the Coalition Program's existence. A sample of non-respondents was contacted, and this revealed no significant differences in their replies from the returned responses, so no weighting of responses was required. The survey covered five general areas about each respondent: - Personal demographics; - Involvement in the Coalition; - Involvement in activities involving course design and educational product development; - Involvement in instructional implementation of Coalition materials; - Assessment of the impact of the Coalition on teaching, students, and institutional culture. A. Coalition Inputs Of the 132 valid responses from individuals involved in the Coalition program at their respective institution, 117 (89%) held professorial rank at some level, while the remainder held administrative or other non-teaching positions. With more than 75% of the respondents holding normally tenured positions, the data suggest that tenure-track junior faculty do not appear to be significant participants in Coalition activities. In terms of time involved in the Coalition, 17% had spent one year or less on Coalition related activities, 45% had spent between two and four years, and 37% had spent five or more years. Over half of all respondents reported that they remained active in the Coalition. Based on duration in the program, the data suggest that Coalition participants have exhibited no small degree of commitment to Coalition activities. Respondents were asked to describe the types of activities in which they participated as part of their involvement in the Coalition. The vast majority (79%) described participating in the design or development of courses or educational-related materials. There was also a great amount of participation in activities concerning information-exchange with fellow faculty members; roughly three-fourths had attended meetings involving other schools from their Coalition, two thirds had collaborated with faculty from both their own Coalition and other Coalitions, and half had presented papers on Coalition-related topics at professional meetings (see Table V-1). Only about a fourth of the respondents had participated in student-related activities, such as recruitment and retention. About 60% of faculty were teaching courses that were either new to the curriculum or had been modified as a result of the Coalitions program. Less than 5% had been involved in assessment-related activities. Respondents were asked to compare their involvement in activities on the basis of participating either with other faculty from their own Coalition or with faculty from different Coalitions. Almost two-thirds (64%) of faculty members reported redesigning or modifying courses or educational materials with faculty from within their Coalition, while only about a fourth (24%) had done so with individuals from other Coalitions. There was a somewhat smaller difference relative to faculty who reported teaching or learning new instructional techniques: 58% participated in this activity within their Coalition, while about a third (30%) did so with faculty from other Coalitions. Aside from more informal exchanges, most collaboration took place within each Coalition. Industry involvement in Coalition curricula most often took the form of a visiting lecturer in courses or labs. Among those teaching courses, 42% of the faculty reported having a guest speaker from industry, but only 13% had included industry guests in Coalition-related labs. Other participation by industry representatives included reviews of student work or presentations (31% courses, 13% labs), generating or participating in design projects (28%, 11%), and providing internships or co-op experiences (12%, 7%). B. Curriculum And Instructional Delivery An important objective of the Coalitions Program was to affect the engineering curriculum at member institutions. Faculty were asked a series of questions to assess the impact that their Coalition has had on developing, designing and revising courses, courseware and other educational materials at their institutions. A total of 79% of respondents had been involved in these activities through their Coalition. Half of these individuals helped modify or develop labs, while almost all had done so for courses (many participated in both). Collectively, these individuals were involved in reshaping 234 courses and 99 labs in total. Combined, surveyed faculty taught 146 single-discipline courses and labs, 94 involving multidisciplinary engineering classes, and 93 cross-disciplinary courses and labs involving both engineering and non-engineering areas in conjunction with their Coalition participation. Project teams were the most common component found in the courses and labs that respondents reported designing or developing (see Table V-2). More than two-thirds (70%) of those involved in course design and development reported that project teams had been part of courses, while about half (53%) said they were elements in labs. Faculty reported that the people they dealt with most often in their work on Coalition courses and labs were faculty from their own institution (80% in courses and 68% in labs). Significantly less collaboration occurred with faculty from other institutions within the respondents' Coalition (47% and 37%), as well as with faculty from other Coalitions (15% and 18%). Faculty reporting interaction with industry personnel was relatively low at 34% for courses and 23% for labs. Less than half had interacted with evaluation and assessment experts. Respondents were asked to identify the different types of products they helped develop for these courses and labs (see Table V-3). Two-thirds said they helped develop instructional modules for courses, while nearly half did so for labs. About half indicated they participated in developing a curriculum for a whole course. Curricular materials produced by faculty in the survey employed an array of instructional media. While nearly two- thirds had worked on products that took hard copy form (e.g., a textbook or a paper module), substantial numbers had been involved in developing computer-related materials. Nearly half had worked on developing Web-based materials, and roughly a fourth had incorporated e-mail, computer diskettes, and C-ROMs in instructional materials they had produced. Components of courses and labs taught by Coalition faculty followed the same pattern as components of curricula designed or modified by faculty. Open-ended problem solving was cited most often, with 75% of faculty who taught Coalition courses reporting its use in their course, and 50% of faculty who taught Coalition labs having used it in their labs. Project teams (72% and 57%, respectively) and design projects (70%, 46%) were the other most frequently reported components of Coalition courses and labs the faculty were involved in teaching. Less commonly-cited elements of Coalition courses included discovery-based learning (34%, 30%), using a business context in problem-solving (34%, 15%), and student recruitment and retention efforts (20%, 4%). Hardcopy (e.g. textbooks or paper modules) remains the dominant type of media used in Coalition labs and courses, with 77% of faculty reporting their use in Coalition courses, and 63% using them in their labs. E-mail is also used by more than half of the faculty in courses (60%), as are World Wide Web sites (52%). Multimedia modules, computer diskettes, and video are all used by over 20% of the faculty teaching Coalition courses. Dissemination of materials across Coalition institutions was relatively uncommon. Only 17% of faculty utilized World Wide Web sites developed at another Coalition school in the courses they taught, with 15% reporting their use in labs. Course curricula developed at another Coalition institution were used by 16% of the faculty in their courses and by 9% in their labs. Respondents were asked to elaborate on their use of the World Wide Web in the courses and labs they taught as part of their involvement in the Coalition. The most frequent response was the use by faculty to gather information on courses or labs, with 53% and 28% reporting this respectively. Faculty also made relatively frequent use of the Web in obtaining materials for student use, with 48% reporting this use in courses and 24% in labs. Student use of the Web for courseware occurred slightly less often at 40% and 24%. Less frequent usage was found in interdisciplinary exposure (23%, 15%) and assessment-type activities (15%, 9%). C. Student Outcomes The most systematic information on student outcomes produced by the study derives from the survey of participating faculty. About half of the faculty (49%) had taught more than 100 students in their Coalition courses and labs, while 13% taught under 30 students. Faculty were asked to compare the students in their Coalition classes or labs at the beginning term with their performance after a month to six weeks of exposure to the Coalition materials. Generally, once given time to get used to the media, the content, and the learning environment, the students appeared to gain a firm grasp of the courses. Clearly, the students had a fair amount of acclimating to do: a minority of faculty (40%) reported that, at the beginning of the course, most or all of the students knew what was expected of them. Four to six weeks into the course, nearly three quarters of the faculty reported that most students understood the course expectations. The number of faculty who said that most or all of their students understood the material easily doubled after four to six weeks in the course, going from 15% to 30%. The Coalition curriculum also had some impact on students' comfort with team work: faculty who reported that most or all of their students were comfortable working in groups rose from 29% at the beginning of a course to 51% four to six weeks later. Coalition students also compared very favorably to their non- Coalition counterparts in terms of achievement. Among faculty reporting (Table V-4), 82% said that the Coalition students performed better in interdisciplinary projects. Coalition students were also rated higher at working in groups (79%) and understanding and pursuing hands-on learning (61%). Overall the faculty rated the Coalition students higher than the non- Coalition students on all seven criteria listed. The student workload for Coalition courses and labs was considered by faculty to be significantly higher compared with non-Coalition classes. The majority (69%) thought that students had either much more work or somewhat more work in the new classes. However, faculty generally believed that Coalition courses had had a positive impact on students. More than three- fourths (79%) stated that Coalition courses had had a positive impact on students working in teams. Roughly two-thirds reported a positive impact of these courses on students' preparation for employment (69%), communication skills (68%), and breadth of understanding of subject matter (63%). Half or more also thought that the Coalition courses had had a positive impact on students' depth of understanding of subject matter (58%) and retention in engineering majors (50%). A considerably lower percentage (38%) thought that Coalition courses had had a positive impact on students' academic performance. Very few identified negative impacts. Faculty were asked to discuss the aspects of Coalition courses or labs that students enjoyed the most as compared with more traditional non-Coalition classes. A few themes dominated their responses. Students really enjoyed the opportunity to work in teams. They learned how to solve problems collaboratively, something they didn't get to do as often in non-Coalition classes. It also allowed them to interact with people they normally wouldn't, including students and professors from not only outside of their class but from other disciplines as well. Students also preferred the hands-on/active learning structure in Coalition courses. Passive listening to lectures gave way to active involvement in projects, which clarified some difficult ideas. Included in this hands-on approach is the chance to incorporate cutting-edge technology into the coursework. Finally, students appreciated, in the view of faculty, the real- world focus provided by projects that had practical applications, as opposed to involving more abstract concepts, especially while still in their first year. When asked what students liked the least about Coalition courses or labs, the over-whelming response from the faculty was the amount of work. Clearly the courses and labs were much more demanding of the students' time than traditional classes. Another major complaint of the students was the same as what they identified as liking most about Coalition classes: working in groups. Students got frustrated when conflicts inevitably arose over which direction the group should take towards completing a project. According to faculty, another student complaint about working in teams was that some members didn't do their share of the work, leaving it up to others instead, and working with students from other disciplines could also initiate problems. Ironically, aside from the workload, faculty reported that what students enjoyed most about Coalition courses and labs were the same things with which others had difficulties. D. Outcomes: Faculty and Academic Culture The survey of participating faculty asked respondents whether participation in the Coalitions had affected their involvement with undergraduate students in terms of teaching, advising/mentoring, research, and outreach/recruiting. In each case, more than half of the faculty reported no change from prior to their involvement with the Coalition. The area in which increased involvement with undergraduates was reported by the highest number of respondents was teaching (43%), followed by advising/mentoring (35%), research (28%), and outreach/recruiting (19%). One out of ten (11%) faculty members reported decreased involvement with undergraduates in their research activities. Faculty were asked in an open-ended question to describe other types of cultural changes they identified as having taken place in their institutions or departments during the past five years. Increased faculty interest in teaching was the most frequently listed change (17 respondents), followed by increased industry interaction (14), interdisciplinary collaboration (12), student mentoring (10), and student teamwork (7). Three-fourths of the participating faculty (75%) reported that the Coalition had led to improved teaching. A small majority of faculty (53%), thought that the value of teaching compared with research in their departments had remained relatively the same since prior to the Coalition. However, over a third (35%), thought that the value of teaching had increased somewhat relative to research, and an additional one out of ten (11%) thought that it had increased a lot. Most participating faculty (83%) reported that research has always been and remains more highly valued than teaching in promotion and tenure decisions. A handful (6%) reported, however, that Coalition activity had led to teaching being valued as highly as research. Faculty were also asked whether the weighting of their participation in education reform activities had changed in promotion and tenure decisions since before the Coalition. While most reported that innovative teaching and curricular reform activities are currently valued about the same as before the Coalition, about a third thought that innovative teaching is currently valued more than prior to the Coalition in promotion decisions (36%) and in tenure decisions (28%). Roughly one out of four or five thought that curricular reform activities are currently valued more than prior to the Coalition in promotion (27%) and tenure decisions (18%). VI. SURVEY OF DEANS OF SCHOOLS OF ENGINEERING The Coalitions have collectively produced many examples of outcomes reflecting Coalitions program goals. The field visit evidence suggests that the impact of these outcomes falls far short of the desired systemic reform. Of the various types of additional information that were collected, the most systematic comes from a survey of engineering school deans, focused on the types of reforms that have been instituted at their institutions and their awareness of the Coalitions' efforts and the use that has been made of Coalitions' materials at their schools. The survey was administered to deans of the schools or colleges of engineering at all U.S. colleges and universities that have undergraduate engineering programs except those that are part of an Engineering Education Coalition. Of the 193 Deans to whom questionnaires were distributed by a combination of e- mail and web-based administration, 126 valid responses were received, for a response rate of 65%. The questionnaires were addressed by name to the deans of the schools or colleges of engineering, noting that the dean might wish to direct the questionnaire to another individual in the university administration with special knowledge of engineering curriculum issues, such as the academic affairs officer. Two-thirds of the responses (65%) were completed by the deans of engineering themselves, and another fourth (25%) were completed by the associate/assistant deans for academic affairs. The remaining ten percent were completed by vice presidents/vice provosts for academic affairs, associate/assistant Deans for engineering, Deans of the undergraduate college or academic programs, and in one case by the provost. Respondents were asked to identify the types of undergraduate education reforms that had been initiated in their school/college of engineering during the last five years. About a third (36%) of these schools of engineering had initiated the development and management of large scale curricular changes. In terms of less sweeping changes, virtually all of the schools of engineering (98%) had undertaken course design, development, or revision, with two-thirds or more having initiated most of the range of other activities is shown in Table VI-1. Of the 25 that had introduced changes aimed at the adoption of an engineered systems approach, more than half were institutions in which an NSF Engineering Research Center has been established. Most of the above reforms had been initiated at the level of the school/college of engineering (89%) or department (87%), but a substantial number (44%) had been university-wide. One of the primary factors that has motivated reforms at engineering schools over the last five years has been ABET 2000; roughly nine out of ten (91%) survey respondents indicated that this had been a factor motivating change. Faculty interest has also been a strong factor motivating reform, indicated by about 8 out of 10 (81%) survey respondents. Other factors are listed in Table VI-2. With wide scale reform of engineering education the basic goal of the Coalitions Program, institutional commitments to teaching in promotion and tenure decisions are an important measure of the extent to which such reforms are being established. Innovative teaching is given greater weight in promotion and tenure decisions than five years previously at roughly forty percent of colleges and schools of engineering. While in many cases it is not receiving any greater weight than previously, almost no cases were reported in which it is being weighted less than five years ago. Roughly twenty-five to thirty percent of the schools also reported that curricular reform activities and efforts are being weighted more heavily in promotion and tenure decisions than five years ago. While again, the vast majority reported no differences from five years previously, almost none reported that such activities are weighted less than five years earlier. Evidence of important change might include interdisciplinary interaction arising in schools outside the Coalitions Program. Respondents to the Deans survey were asked what proportion of their faculty engage significantly in interactions beyond their home departments. Faculty interaction with other departments is not uncommon, with 61% of the schools of engineering reporting that about a third of their faculty interact across departmental lines and another 27% reporting that more than half of their faculty engage in cross-departmental activities. Interaction with non-engineering colleges is much less common, with 50% of respondents reporting that less than one out of ten of their faculty engage in cross-college activities, another 43% reporting that only about a third of their faculty are significantly engaged in these activities, and only 7% reporting that more than half of their faculty interact with other colleges. Faculty interaction with industry, another part of the Coalitions Program agenda, is more common in colleges of engineering than is interaction across departmental lines. Close to four out of ten of the schools surveyed (38%) reported that more than half of their faculty have significant interactions with industry and another five out of ten (51%) reported that about a third of their faculty are involved in such interactions. About half of the colleges of engineering that responded to the survey (48%) reported that their college had collaborated with other universities or colleges in the development of courses, material, or student assessments during the past five years. More than a third (38%) of these other universities and colleges with which education-related collaboration had taken place were said to have been members of an NSF Engineering Education Coalition. By far the most commonly reported sources for learning of new developments in curricular materials were ASEE publications, conferences, and meetings, listed by over 50 respondents in response to an open-ended question. Other frequently cited sources of information included journals, articles, and other publications (19); the Internet and the Web (17); national conferences (17); professional meetings (16); ABET meetings and publications (16); NSF Engineering Education Coalitions materials and publications (13); NSF reports and publications (8); and publishers and vendors (7). The field research noted that Coalitions reported a large amount of ASEE media as means of dissemination. With ASEE sources listed as the top source of learning of new developments, it is hard to assess the reason that Coalition materials, themselves, ranked so low in respondents' awareness of Coalition efforts and materials, but it is possible that the respondents read about Coalition-related efforts in ASEE publications without being aware of the relationship to the Program. This is especially true because roughly eight out of ten of the Deans' survey respondents reported that they were familiar with NSF's Engineering Education Coalitions Program. SUCCEED was known of by 56% of respondents, ECSEL by 47%, Gateway by 44%, Synthesis by 37%, Foundation by 36%, and Greenfield by 21%. When asked whether they were familiar with any products and/or materials (e.g., textbooks, CD-ROMs, World-Wide Web sites) developed by an Engineering Education Coalition, however, less than half (40%) answered in the affirmative. Specific products with which survey respondents were familiar are listed in Table VI-3. About one-third of respondents were familiar with specific Coalitions products. As shown in Table VI-4, 46% of these respondents considered the Coalitions products with which they were familiar very accessible, 40% considered them very user friendly, and 34% considered them very useful. Respondents were also asked whether, based on their understanding of those in their school or college of engineering who had reviewed Coalitions products or materials, any shortcomings in these materials had been found. Highly pertinent to understanding some of the difficulties involved in disseminating Coalition products is that while roughly a third of the respondents indicated that no serious shortcomings in Coalitions products had been found, about a fourth indicated that the materials require too much faculty time to use properly and a fourth also indicated that the materials are not ready for transfer beyond the originating institution. Several comments about transferability implied that the materials had limited utility outside the Coalition group of institutions because they were too campus-specific, too internally focused, or limited by differences in size and culture of the institutions involved. Of the 50 respondents who indicated that they were familiar with Coalition materials and products, only 22 indicated that any of these materials or products had been used in their school or college of engineering. This is about 18% of all institutions that responded to the survey. Eleven institutions are using methods for incorporating new technology in courses or labs (e.g., video or distance learning), 9 institutions are using instructional modules for courses or labs, and 6 are using other methodological materials. Only a few institutions are using a curriculum for an entire degree, a complete curriculum for an individual course or lab, or a curriculum for a set or sequence of courses or labs. The most common media through which these Coalition materials and products have been obtained are Web sites, hardcopy textbooks or paper modules, and CD-ROMs. Eleven institutions reported that Coalition materials have been enthusiastically embraced by a small number of faculty, 6 reported that the materials are considered to be no different from those already available, and 10 reported that faculty are not sufficiently aware of the materials to have an opinion. Only fifteen respondents reported awareness that individuals from their institution had looked at the National Engineering Education Delivery System (NEEDS). Most of those who had not looked at the system reported that they were not aware of it. Of those who had examined the system, most found it somewhat useful but none found it very useful, a roughly equal number found it somewhat or very accessible, and a roughly equal number found it somewhat or very user friendly. Eleven respondents reported knowledge of someone in their institution having downloaded courseware or other materials from NEEDS, 5 reported someone having tested such materials in a course or lab, and 4 reported someone having evaluated such materials in a formal sense (e.g., individual or committee review). In fact, relatively few of the respondents in the Faculty Participant Survey (9%) had used NEEDS postings to disseminate products, as opposed, for example to conference presentations (57%). The Deans survey respondents, especially in questions indicating detailed awareness and/or attempts to use Coalition outputs, indicated that the impact of the Coalitions and their outputs has been mixed and limited. The results reinforce information from field research pointing to a serious problem with dissemination of Coalition materials to the wider engineering education community, as distinct from actually adopting Coalition curricula and materials in non-Coalition institutions. VII. CONCLUSIONS AND RECOMMENDATIONS A. Overall Findings 1. Progress towards Reform Thus Far The Engineering Education Coalitions Program was established to create broad-based institutional collaborations aimed at establishing "bold, innovative, comprehensive, and systemic new models for undergraduate engineering education." "Comprehensive Curriculum Restructuring" was emphasized, along with widespread dissemination and adoption of Coalition products at both Coalition and non-Coalition schools of engineering. The objective of this study was to assess the progress that the Coalitions had made during the first five years of the program toward these goals at roughly the midpoint of the Program's prospective life span. A large number of Coalition-supported "products" have been generated, both in terms of curricular materials and instructional practice. All are responsive, typically, to more than one of NSF's desired characteristics: a teaming approach, open-ended learning, new instructional technologies, etc. The most pervasive of these was found to be the presence of "engineering up front": the exposure of freshmen to hands-on, real-world engineering practice early in their undergraduate education. Both dissemination and program evaluation activities were found to have been much slower to develop than the Program had intended. Senior design projects with industrial partners were highly valued and efforts are being made to expand them. Both faculty and students saw Coalition emphasis on a team approach as a difficult experience, replete with problems associated with team composition and organization, ways of dealing with slackers, and grading methods. Students nonetheless rated it highly as a valuable "real world" experience that gave them an advantage in the job market. Because too few students had graduated from degree programs heavily influenced by Coalition efforts when the study was conducted, no component of following their experience in industry was included in the study. However, available student feedback indicated that Coalition experience was entirely advantageous in terms of the reactions of industry personnel during internships, coop or summer jobs, and senior job interviews. The teaming experience, hands-on understanding, and communications skills derived from Coalition-influenced courses were recognized and valued by industry as reported by students interviewed at the Coalitions. Preexisting characteristics of the institutions involved had an important impact on the degree to which "buy-in" and the consequent inception, breadth, and institutionalization of Coalition efforts was received. Some disciplines in a given institution had long since developed a team culture, others resisted it. Support, both in terms of leadership and resources from administrators in both the engineering school and the larger institution, had a significant impact. Some institutions were active developers of materials, others passive recipients and/or adaptors. Coalition efforts had to compete with other priorities over and above research agendas within schools and departments. Coalition acceptance was aided by and, in turn, aided schools in dealing with ABET 2000 efforts. In fact, it is difficult to assess the progress of the Coalitions because there are so many other convergent forces pressing for the reform of engineering education. A clear limitation of Coalition initiatives has been their breadth in relation to the full range of engineering disciplines. Even at a single institution, much less across institutions, much of the progress thus far has been within certain disciplines such as mechanical engineering or at lower undergraduate levels, such as freshman coursework. Most of the Coalitions have had only a limited impact on the other major engineering disciplines or at the upper levels of the undergraduate curriculum. Although many examples of desirable change have been described in this report, the examples are largely isolated observations of the desired behavior, covering only a small minority of courses, faculty, or students. Some possible reasons for the lack of stronger signs of systemic reform to date follow. 2. How Long Should Reform Take? One obvious response to the limitations on reform observed thus far is that the Coalitions have been functioning for a relatively short time, and the focus of the first stages of reform has tended to concentrate to date on lower division courses. Such a response may be entirely reasonable, because the time needed for broad-based reform to occur is not well understood. System-wide reform of undergraduate engineering education in the United States certainly takes more than five or even ten years. For example, for some types of systems change, an entire generation of people (in this case, faculty) may need to turn over before institutions are fully reformed. From this broader time perspective, the fact that the Coalitions have been able to make significant changes in some disciplines, and, at the lower undergraduate levels in particular, can be taken as a positive sign of progress. 3. Conditions Impeding Reform In terms of developing broad-based reform of the U.S. undergraduate engineering education system, study findings suggest that several conditions exist that may be impeding reform. First would be that the Coalitions Program strategy lacks sufficient breadth. Based on 1996 data, the Coalitions collectively cover only 32% of the "teaching" faculty in undergraduate engineering in the United States, and only 24% of the undergraduates enrolled in engineering. Even though the Coalitions include many of the leading engineering institutions, change across the entire country will still require substantial dissemination and diffusion into practice of the Coalitions' successful experiences. A second potential impediment is that some of the factors underlying the concerns leading the original Belmont Conference have not changed greatly. For example, large-scale funding for academic engineering research, whether from industrial or governmental sources, does not appear to have diminished substantially. At the same time, while the funding of the Coalitions collectively seems substantial, the small amounts allocated to individual projects could make them appear to be more trouble than they were worth. Faculty who are oriented toward research can still pursue their interests and either resist change in undergraduate teaching practices or be distracted from it. In the face of such conditions, institutions as a whole have less pressure to change. 4. Conditions Accelerating Reform Despite possible impediments, one force strongly favoring reform is the gradual implementation of ABET's Engineering Criteria 2000. Beginning in the year 2001, the criteria will apply to all accreditation actions in the United States. ABET reviews about 25% of the approximately 330 schools in the U.S. with undergraduate engineering programs (about 305 schools currently have accredited programs). Thus, by 2005, nearly all schools desiring ABET accreditation will have had to comply with the criteria. As this report has noted, some Coalition schools were helped in meeting the ABET criteria by their Coalition involvement. In addition, some Coalitions provided assistance to non-Coalition schools preparing for their own accreditation reviews. Their example suggests that adopting some of the practices and innovations already tested by the Coalitions should be an attractive strategy for other schools in efforts to meet the new criteria. In this sense, the work of the Coalitions will then have provided a valuable contribution toward the overall movement for engineering education reform. B. Building on the Progress of the Coalitions The study suggests that there has been limited success in achieving the Coalitions Program's perhaps overly ambitious goal of comprehensive changes and reform in such a short time-frame of five years. There is much still to be accomplished along such lines as changing academic cultures to integrate engineering education and research, enhance interdisciplinary and team- oriented teaching, and provide greater institutionalization of reforms that provide early experience with design and engineering practice. Nevertheless, much has been accomplished. As the Program moved into its second and final phase, it had: - developed a significant base of enthusiastic human resources, dedicated to and experienced in the effort to provide innovative courses and instructional materials; - produced a wide range of pedagogical outputs that range from computer-based instructional modules through complete new courses to integrated curricula aimed at meeting NSF's goals as well as ABET 2000 criteria. Moreover, while ABET 2000 has been and will continue to be a driving force that influences schools of engineering to adopt programs and materials consistent with the goals of the Coalitions Program, the Program has meanwhile played an important role in preparing schools for ABET review. It also has legitimized to some degree pedagogical research as a part of the reward system of a number of universities. The intent here is to suggest ways in which the Coalitions Program can most effectively leverage these outputs during its final years of investment in the reform of engineering education. To do so is a form of innovation. Exemplary program outputs need to be disseminated, adapted, and adopted widely for systemic reform to be achieved. What we know about innovation says that dissemination, at least in the sense of communication about the availability of something, does not necessarily result in technology transfer (adaptation and adoption), and this is the critical strategic issue for the Coalitions Program to confront. How can it most effectively promote the transfer and adoption of the best work that it has sponsored since 1990? Diffusion of educational innovations faces a number of barriers. Dissemination was shown by the study to be a significant problem for several reasons related to it being an intermediate step, not an end point. 1) Most innovative developments were limited to intra- institutional adoption among Coalition members, and even within institutions, expanding "the choir" has not been easy. 2) Documentation is a serious problem. The most enthusiastic faculty participant is still not inclined by disposition, available project money, and other aspects of the reward structure - including his or her regular research agenda - to invest substantial time in providing the documentation necessary to translate the originating classroom experience into something immediately transferable to other teachers, curricula, and institutions. 3) Dissemination methods used thus far have failed to distinguish between knowledge transfer and technology transfer. The Program has facilitated travel to meetings, and its legitimization of pedagogical research has resulted in the publication of numerous papers in ASEE and other related education journals. However, just as few patents are ever licensed for application, survey evidence from the study suggested that few presentations or publications had resulted in true transfer - adaptation and adoption - of Coalition outputs beyond the boundaries of their developing institution or, in some cases, the Coalition more broadly. These problems with dissemination suggest that the Coalitions Program needs to establish a process that will bring together the products that have been produced and, beyond where they are already in use, the human resources who are committed to needed reforms and seeking innovations to adopt. The process needs to start with a strategic assessment of the products for their quality, adaptability, and marketability in the domain of engineering education. Without demonstrative empirical proof of the effectiveness of an innovation, it is unlikely to be accepted in the market place. This is not enough to bring about adoption, however. Instruction modules and courses need to documented sufficiently in order to be usable in different types of institutions. They must also be sufficiently generic in context to be usable in other settings: if products aren't immediately usable in other institutions they must be adaptable. Broader multi-course curricula face more difficult barriers: they need buy-in from larger and multi-department constituencies. Courses need to be evaluated for their potential adaptability and for their potential for institutionalization, not brief experimentation. Finally, marketability would be enhanced if products were potentially commercializable through publishers, software vendors, and related sources. One of the most serious transfer barriers for Coalition courses and products is that they are typically more costly to implement than traditional teaching methods, both in time and monetary investment: efficiencies of scale or other economies are needed. At least two mechanisms might be considered in order to commence this process. 1) Some type of primary screening of both available materials and the related human resources - their developers, or faculty who have taught courses developed by the Coalition or using Coalition-developed materials. Even the best "products" will have difficulty finding acceptance without "champions." 2) Selectivity Workshops: Priorities need to be established for efforts to push the best of the best across and out of the Coalitions. Such workshops can be topically focused. They need to include the developers and prospective champions of the materials, engineering deans or other academic representatives from outside the Coalitions, representatives from industry who are not Coalition insiders, and "commercializers" (publishers, etc.). Following the establishment of selective priorities, a concerted effort at transfer to other institutions can follow to complete the process. This could include: 1) The development and marketing of commercial products (textbooks, documented modules and computer programs, etc.). 2) Training workshops - perhaps summer sessions of one or more weeks in which faculty from non-Coalition schools are brought together with Coalition champions of the selected innovations to provide an overview of their effectiveness and training in their use or adaptation. 3) Exchange programs intended to leverage the human resources developed by the Coalitions by bringing outside faculty into a Coalition environment containing innovative instructional techniques and cultural shifts, as well as putting product champions into other schools to teach and show others how to implement Coalition courses and materials in non-Coalition contexts. The latter two are aimed at moving beyond forms of secondary interaction, such as conference presentations and the reading of published papers to the most assured approach to innovation: primary human interaction on an extended basis that is intended to equip others to use Coalition products successfully. Finally, if such activities are undertaken, each Coalition should be given the responsibility of providing NSF with a tracking of its activities in the process and an assessment of the effectiveness of those activities in contributing to the broadening of reform toward the goal of more systematic and pervasive reform of undergraduate engineering education. The National Science Foundation (NSF) funds research and education in most fields of science and engineering. Grantees are wholly responsible for conducting their project activities and preparing the results for publication. Thus, the Foundation does not assume responsibility for such findings or their interpretation. NSF welcomes proposals from all qualified scientists, engineers and educators. The Foundation strongly encourages women, minorities, and persons with disabilities to compete fully in its programs. In accordance with federal statutes, regulations, and NSF policies, no person on grounds of race, color, age, sex, national origin, or disability shall be excluded from participation in, be denied the benefits of, or be subjected to discrimination under any program or activity receiving financial assistance from NSF (unless otherwise specified in the eligibility requirements for a particular program). Facilitation Awards for Scientists and Engineers with Disabilities (FASED) provide funding for special assistance or equipment to enable persons with disabilities (investigators and other staff, including student research assistants) to work on NSF-supported projects. See the program announcement or contact the program coordinator at (703) 306-1636. The National Science Foundation has Telephonic Device for the Deaf (TDD) and Federal Relay Service (FRS) capabilities that enable individuals with hearing impairments to communicate with the Foundation regarding NSF programs, employment, or general information. TDD may be accessed at (703) 306-0090 or through FRS on 1-800-877-8339. The National Science Foundation is committed to making all of the information we publish easy to understand. If you have a suggestion about how to improve the clarity of this document or other NSF-published materials, please contact us at plainlanguage@nsf.gov. Publication Number: NSF 00-116 _______________________________ 1 National Science Foundation, Engineering Education Coalitions: Program Solicitation, NSF 89-107, Directorate for Engineering and Directorate for Science and Engineering Education, p.1. 2 R.K. Yin, "The Case Study Method as a Tool for Doing Evaluation," Current Sociology, 40: 121-137 (Spring 1992). 3"Year 4 Report of the Foundation Coalition Program at Arizona State University" (October 26, 1997), p.24. 4Op. cit., p.34.