The Cultural Context of Educational Evaluation
The Role of Minority Evaluation Professionals
June 1 - 2, 2000
These proceedings are an edited version of a two-day meeting with minority evaluation professionals. Session summaries were prepared by invited participants, several of whom were also asked to prepare papers/presentations to frame the discussion. Dr. Elmima Johnson organized the conference and served as editor of this document. We would like to acknowledge the contributions of all attendees, including members of the Directorate for Education and Human Resources who gave invited comments or served as workshop discussants. We also would like to acknowledge the assistance of Joy Frechtling and Martine Brizium, Westat, Inc., in conference planning and logistics under Contract No. REC-9412965.
The National Science Foundation (NSF) Directorate for Education and Human Resources (EHR) sponsored a two-day workshop on the cultural context of educational evaluation June 1-2, 2000, at the Arlington Hilton and Towers in Arlington, Virginia. Invited participants included 15 nationally recognized minority evaluation professionals as well as EHR staff. The meeting served as a platform for presentation of seven invited papers and talks. It also provided a forum for examining a host of issues associated with evaluation.The NSF Assistant Director for EHR and the Director of the Division of Research, Evaluation and Communication delivered opening remarks and framed the meeting's purpose within NSF and the Directorate. Three workshop sessions followed, organized around two major themes:
Invited papers and presentations stimulated dialogue. Invited discussants, experienced educators familiar with program accountability issues across the educational continuum, offered their views.
The workshop represents the opening of a dialogue and will serve as a reference point for the Directorate as it determines its role in building capacity within the field of educational evaluation. This report presents the formal workshop papers and presentations as well as highlights of the discussions. Appendices list the workshop agenda and participants.
Dr. Sunley welcomed participants, noting that they were a distinguished group with expertise and experience in the evaluation of education programs. As such, she said, they are aware of the issues and needs that pertain to evaluation and capable of proposing effective strategies. The participants also were charged with considering the challenges of training NSF staff and grantees as well as other professionals.
She offered a contextual framework for the meeting and its importance to the Directorate, noting that the requirements of the Government Performance and Results Act (GPRA) have increased the need for substantive evaluation of NSF's activities. Dr. Sunley stressed the importance of having the "numbers" reflect what is happening in the field in a consistent manner in order to provide guidance to program managers and help them describe project performance relative to program objectives. She also noted that the meeting was a follow-up to the Evaluation Program's January 2000 workshop on training evaluators to work in mathematics and science. That workshop suggested the need for more SMET (science, mathematics, engineering and technology) evaluators, including those from traditionally underrepresented groups.
She suggested two major meeting tasks. First, participants were asked to consider, when discussing equity and diversity, whether it matters who formulates and asks questions and conducts the evaluation. The second task pertained to a broader issue: how to connect interim and ultimate program outcomes. Because long lead times are common, NSF has difficulty tracing causal pathways between expenditures and outcomes. Focus on interim outcomes can help address this challenge.
Dr. Sunley closed by committing support to evaluator training and related activities, including attention to the participation of minority evaluators. She agreed to consider carefully the recommendations of the workshop, particularly because of the projected need for a substantial number of new evaluators who are familiar with mathematics and science as well as evaluation techniques.
The issues raised at this meeting - building the capacity of the national education evaluation enterprise and, specifically, increasing the presence and participation of culturally attuned and minority-represented evaluators - collectively reflect a strategic national concern. As former director of NSF's Comprehensive Regional Center for Minorities in Chicago in the early 1990s, I am deeply aware of the difficulties involved. We labored long and hard to devise and implement a pipeline of programs for K-12 youngsters in the city that would bring them into the urban higher education centers prepared to excel in science and engineering. We also sought to build capacity in the school system to make sure more minority youngsters had the opportunity to successfully navigate the gatekeeper courses of middle school and high school mathematics. But we were always functioning ad hocas we generated the formative evaluations that would allow for continuous, robust improvement of the program pipeline. Our problems were a microcosm of the broader national need to elevate education evaluation, especially as it pertains to minority-focused programs. On behalf of NSF and the REC Division, please accept our welcome to this effort to help build deeper leadership and participation by minorities in evaluation. Their involvement is critically needed.
Unlike days past, evaluation has become an important issue for NSF's programs and projects. Once, evaluation was considered an add-on. Program officers dropped it from budgets as they struggled to fund a few more grants. Now, all programs require the inclusion of evaluation plans in proposals. Also, many programs require a percentage of each award (often 10%) to be directed to evaluation. Under these circumstances, it is critical to have trained evaluators available. Often, we hear complaints from Principal Investigators who say they cannot locate evaluators to work with them. Clearly, there is a shortage of evaluators - particularly minority evaluators - with mathematics and science backgrounds and experience.
Although our budget is limited, the Evaluation Program has tried for several years to have an impact on this problem. In 1993, we supported the publication of a User-Friendly Handbook for Project Evaluation, developed by Westat, Inc. It provides our Principal Investigators with the basics of evaluations. In 1997, we published a companion document, The User-Friendly Handbook for Mixed Methods Evaluation. To apply these materials, we also sponsored a series of short workshops, primarily for Principal Investigators and their staff.
In addition, for doctoral-level evaluation training, we support the work of the American Educational Research Association. It coordinates four programs for training evaluators with strong mathematics and science backgrounds. We also support the Evaluation Center at Western Michigan University to provide short, intensive evaluation training. By supporting month-long summer institutes for faculty and advanced graduate students, we also sought to provide evaluation that goes beyond awareness, but also is less than full-scale doctoral training. Finally, we initiated an Online Evaluation Resources Library developed by SRI, International. It provides materials for evaluators to use in their work on NSF projects.
In January 2000, we held a workshop for 40 invited specialists in evaluation training. Its purpose was to discuss and assess what was known about the impact of our efforts and how to proceed with them. Our planning continues. However, one issue that clearly emerged was the need to prepare minority evaluators. That finding led to this conference. Another issue that has arisen is how we can work most effectively with professional evaluation associations, particularly the American Evaluation Association. Work in that area is still in the planning stage. However, EHR's Evaluation Program intends to pursue a vigorous training effort and it welcomes input about how best to proceed.
|EHR Home | nsf.gov
||| About NSF | Funding | Publications | News & Media | Search | Site Map | Help|
|The National Science Foundation
4201 Wilson Boulevard, Arlington, Virginia 22230, USA
Tel: 703-292-5111, FIRS: 800-877-8339 | TDD: 703-292-5090