The Cultural Context of Educational Evaluation
The Role of Minority Evaluation Professionals
June 1 - 2, 2000
This session included a brief presentation of two background papers by their authors. It is followed by a group discussion, a summary of the discussion and identification of issues to be addressed in the future by the discussant.
The paper by Rodney Hopson presented the problem of the lack of available mechanisms to identify the current population of minority evaluators. It took the position that: "sustainable substantive interventions and mechanisms are needed to increase the number of minority evaluators with advanced training and experience in the field of educational evaluation, particularly those with expertise in science and mathematics education" (p. 1). The paper's theme was "that any proposals garnered to address the paucity of minority participation in educational evaluation were necessary but insufficient unless they both provided structural and adequate nurturance to a pipeline of scholars and created conditions whereby minority evaluators sought to advance knowledge production in the field for the purpose of emancipation and social justice" (p. 1).
Hopson explained that the role of professional organizations in addressing this problem was just getting started. He stated that organizations such as AEA [American Evaluation Association] were beginning to build diversity among the evaluation community including the development of mentoring programs within the organization. He further suggested that to be effective efforts to increase the numbers and participation of minority evaluators should include the use of multiple strategies.
Two major questions framed the paper by Stafford Hood: 1) "What ought to be the motivation for increasing the number of minority evaluators with advanced training and experience in the field of educational evaluation?" and 2) "What might be the benefits of including minority evaluators in the evaluation of science and mathematics education programs?" Hood believed that the motivation to increase the number of minority evaluators stemmed from practical considerations, since urban districts have increasing numbers of minority students. Yet very few minorities served as external or internal evaluators of educational programs in these districts.
Hood further stated that graduate programs have not done enough to address this problem, yet there were many contributions to be made by minority evaluators. Among the potential contributions were: better understanding of the programs serving minority students, value of the programs for those intended to be served, refinements to improve the benefits of the programs, interpretation of the meaning based on "who's doing the looking?" and observation and translation of the non-verbal communication that occurs in interactions. The author also raised the issue that, according to the Program Evaluation Standards, it is necessary to include stakeholders in the evaluation, particularly minorities. However, in the case of minority evaluators, racial similarity should not be the only requirement considered beyond competency. The paper ended with an invitation to look at covariates of success for achievement in mathematics and science by children of color.
The group discussion began with agreement that more minority evaluators are needed. Dr. Judith Sunley, Interim Assistant Director of EHR, clearly recognized this need in her welcoming remarks. The group expressed the belief that the identification and description of minority evaluators are important steps to justify the creation of programs to increase their numbers. As part of this discussion, it was suggested that NSF conduct a survey to determine the number of minority evaluators and their work settings. There also was a proposal to develop a framework and plan for a database of the current pool of minority evaluators in school districts, state education agencies, colleges and universities, as well as in professional organizations, as part of the identification and description process.
In terms of strategies to recruit minority evaluators, some group members suggested that we think across the educational pipeline to identify and train minority evaluators. We also need to make a list of the "cogs" of the wheel such as universities, government agencies and professional organizations that can help to identify and to produce minority evaluators. Another suggestion was that we develop recruitment strategies at the undergraduate level to tap future evaluators before they enter graduate school. Two additional suggestions were to recruit an advisory team of education research and evaluation faculty from minority-serving institutions to plan a pipeline program and to create an interagency working group to develop a paper on research and evaluation needs.
Several strategies to train minority evaluators were introduced. The need to emphasize the importance of building cultural sensitivity at multiple levels and to consider cultural context in the evaluation of programs was stressed throughout the discussion. It was proposed that colleges and universities (including predominantly minority serving institutions), government agencies and professional organizations develop training programs for prospective minority evaluators; their proponents could look at training centers for models for these programs. Another strategy was to identify the best methods to recruit and train minority evaluators. Because it is so difficult to find evaluators who can be solely responsible for mathematics and science projects, the group debated whether the training of minority evaluators should be based on content or on strategies and suggested that individuals should be trained across disciplines and across levels (i.e., social sciences, undergraduate, master's level). Further, an intern program for minority evaluators was presented as a training alternative, and the need for incentives for students and institutions to support the training of minority evaluators was highlighted.
In addition to identifying and training potential minority evaluators, design strategies were suggested to keep these evaluators working in the field. The group proposed four major strategies to address this need: 1) have senior faculty at universities serve as mentors and role models; 2) identify mechanisms to nurture a critical mass of minority evaluators within universities; 3) identify potential employment opportunities for minority evaluators and make this information available to eligible candidates; and 4) utilize evaluation teams that are culturally diverse and include subject matter content experts.
The group addressed the importance of supporting minority evaluators from multiple types of agencies. It was stated that agencies should fund programs that will prepare evaluators - especially minority evaluators- effectively. It was further suggested that agencies need to build on current successful efforts, particularly where a critical mass of individuals is currently involved in the training of evaluators, particularly minority evaluators. It was suggested that NSF could become the leader in increasing training opportunities for minority evaluators among federal agencies.
The discussion and subsequent written comments by the participants raised the following questions for future consideration:
The discussant concluded with several reflections. First, the discussant pointed out similarities between the issues related to the identification and training of minority evaluators and those faced by many educational reform efforts. She stressed the importance of identifying potential future minority evaluators early in their careers since, if students do not make it through the pipeline, they cannot become evaluators later on. She emphasized the importance of using a multi-prong approach to identify evaluators so that specific programs target a critical mass of people, while other interventions focus on work with individuals. She suggested that program designers look beyond incentives when making decisions about ways to identify and train minority evaluators; for instance, redesign program structures within higher education institutions and other organizations. She urged the participants to look beyond NSF to other agencies that could also address this important problem of the evaluation field.
New Look at an Old Question*
This paper provides a starting point for our conversation about several provocative and critically important questions related to the evaluation of educational programs in general, with special attention given to mathematics and science programs. The two primary questions that frame this paper are: What ought to be the motivation for increasing the number of minority evaluators with advanced training and experience in the field of educational evaluation? And what might be the benefits of including minority evaluators in the evaluation of science and mathematics education programs? The paper further suggests the merit of looking beyond the dismal failure of African American, Hispanic and Native American students on standardized measures of mathematics and science achievement by mounting a systematic study of students from these groups who have achieved or are presently achieving in mathematics and science in secondary schools. I turn now to my first point.
In my opinion, the motivation for increasing the number of minority evaluators with advanced training in educational evaluation is a practical one. We live in a time when most of our major urban school districts have predominant enrollments of students of color, and evaluators of color are generally absent in the evaluation of educational programs that serve these students. To make this point I do not think it is necessary to provide numbers, but rather ask the reader to rely on personal recollection. You are simply asked to recall the number of African American, Hispanic and Native American evaluators with whom you have come in contact at research and evaluation units in central school district offices, state departments of education, the U.S. Department of Education and private foundations. How many African American, Hispanic and Native American evaluators have you seen as members of external evaluation teams evaluating educational programs that target students of color or even directing such evaluations? If your experience parallels mine, your answer to these questions will be "very few."
One of the major reasons for the limited number of trained program evaluators of color is that those graduate programs with the capacity to train program evaluators have not done enough to rectify this situation. The most telling symptom is the dearth of doctoral degrees awarded to African Americans and other groups of color by institutions capable of doing so. My on-going monitoring of the National Center on Education Statistics Integrated Postsecondary Education Data System's data, reporting doctoral degrees awarded by institution, race, and program areas within the field of education at major research universities, supports my observations (see Hood and Freeman, 1995). These data are not perfect for making this point but they are enlightening. The need for more trained program evaluators of color is what they can contribute to"understanding" in the evaluation of programs serving students from this population.
In program evaluation the production of clear, useful and objective knowledge and the pursuit of understanding seeks to determine program worth. In this case, I emphasize the importance of the evaluation resulting in an "understanding" of the program, its value for those who are intended to be served and its refinements to improve the benefits. I would argue that an evaluator's understanding of a program, as it functions in the context of culturally diverse groups, is the most critical dimension for evaluating programs that serve these populations.
We must honestly assess whether current evaluation practice concerning diverse people does not systematically ignore potentially important aspects of diversity. When evaluators attempt to derive meaning from data gathered in cultural contexts they do not understand, their efforts are limited at best and potentially hurtful at worst. We must safeguard against producing evaluative knowledge "that seems counter intuitive to the [culturally diverse stakeholders] and seems to contribute little to our understanding of the people" (Gordon, 1998) and the programs which intend to serve them. Gordon referred to the work of anthropologist Michael Jackson, who queried "whether the lived experience is a necessary condition for valid observations." It was his view that "there was a possibility of our inability to understand the experience of the other." In my opinion, central to the observation is the meaning of what has been observed. Who is doing the looking is central to the process of evaluation.
When observing African American participants, for example, much in the program under evaluation can be lost long before reaching a summary "understanding" of the merit and worth of what has been observed. For example, too often nonverbal behaviors are treated "as error variance" by the observer and therefore ignored. Akbar (1975) asserts that African Americans "[rely] on words that depend upon context for meaning and that have little meaning in themselves.. [while also]...using expressions that have meaning connotations." Therefore the review of interview transcripts without the ability to interpret meaning based on these unwritten rules would likely result in interpretations that are more frequently wrong than right, thereby limiting communication and ultimately understanding between the African American participant/stakeholder and the evaluator. I would expect that these concerns regarding "observations" and "translation" are equally true for other groups of color. A brief look at the program evaluation standards may also be instructive.
The second edition of the Program Evaluation Standards: How to Assess Evaluations of Educational Programs by the Joint Committee on Standards for Educational Evaluation (1994) has been offered to provide guidance to effective evaluation. The standards are organized around four major areas: Utility; Feasibility; Propriety; and Accuracy. The utility standards suggest that evaluators are required to "acquaint themselves with their audience, define the audience clearly, ascertain the audience's information needs, plan evaluations to respond to these needs and report the relevant information clearly and in a timely fashion." One of the ways to achieve this objective is through the identification of stakeholders.
The standards suggest that: 1) it is necessary to include "less powerful groups or individuals as stakeholders, such as racial, cultural or language minority groups;" 2) determine how the respective stakeholders view the evaluation's importance, would like to use the results and what information will be particularly useful; and 3) include the clients and stakeholders in designing and conducting the evaluation. Very few of the program evaluations with which I am familiar in American urban schools and other settings have shown a significant level of participation by the less powerful stakeholders. Additionally, I found it interesting that the standards did not consider the importance of these stakeholders assisting the evaluator to interpret the evaluation's results.
I find the standard on evaluator credibility to be important for culturally responsive evaluation. This second utility standard requires that: 1) the evaluators be trustworthy and competent to conduct the evaluation; 2) they be knowledgeable of the social and political forces affecting the less powerful stakeholders and utilize this information in the design and conducting of the evaluation; and 3) the work plan and composition of the evaluation team be responsive to key stakeholders.
Very fine guidelines - I have found little evidence of them being implemented in the evaluation of American urban schools. There may in fact be different views regarding evaluator credibility. Grace (1992) offers a different perspective on evaluator credibility when working with African American communities. She indicates that "in many cultures the age, race, sex, and credentials of the evaluators may have a significant impact on the evaluation process...[in the case of African Americans] - all things being equal - the most influential and respected members of the evaluation team are likely to be older individuals with academic credentials related to their expertise as evaluators."
Grace and I agree that every effort should be made to include African American evaluators who are positively identified with the black community among the members of the external evaluation team, but racial similarity should not be the only requirement considered beyond competency. Grace also suggests that potential evaluators should be interviewed using questions designed to tap culturally relevant knowledge, attitudes and skills as the best indication of a candidate's suitability for the job as an evaluator. Something for us to think about.
I trust that the need for increasing the supply of trained program evaluators of color has been established. I trust further that the benefits of including program evaluators of color throughout the evaluative process have been established to the reader's satisfaction. If this is the case, then we may turn to the ultimate question for mathematics and science educators, and evaluators as well.
What do we know about the antecedents and conditions that are needed for meritorious academic achievement in mathematics and science by children of color? I'll provide the answer: very little. Every experienced evaluator has discovered children of color who have distinguished themselves in either mathematics or science. Why these and not others? Further, even a modest historical search will reveal that African American educators, for example, have achieved remarkable success as scholars in science and mathematics. How was this accomplished? Similarly, there is a documented trail of African American educators who have contributed to the evaluation literature. In a word: America has produced high achievement in mathematics and science by people of color. How? Rather than participate in yet another examination of the correlates of failure by learners of color in the mathematical, biological and physical sciences, I envision a study of the covariates of success. To that end, we can begin by increasing the supply of trained people of color in educational evaluation and their participation in the evaluation of educational programs in urban schools.
Toward Participation and Liberation in Educational Evaluation: Developing Educational Pipelines to Increase Minority Evaluators
Rodney K. Hopson
Liberation is a value worthy of science. That should be the perspective from which the minority scientist seeks to advance knowledge, always in the spirit of respect for logical canons, multiple perspectives, and methodological rigor; not for the purpose of simply predicting, controlling and understanding, but for the purpose of emancipating (liberating) the bodies, minds, communities, and spirits of oppressed humankind (Gordon, Miller and Rollock, 1990:19).
Introduction and Problem Identification
The comments in this paper related to participation of minority professionals in educational evaluation directly address the mechanisms - or more accurately, lack thereof - available to identify the current population of minority evaluators*. This paper takes the position that sustainable, substantive interventions and mechanisms are needed to increase the number of minority evaluators with advanced training and experience in the field of educational evaluation, particularly those with expertise in science and mathematics education. The theme of this paper is that any proposals garnered to address the paucity of minority participation in educational evaluation are necessary but insufficient unless they both provide structural and adequate nurturance to a pipeline of scholars and create conditions whereby minority evaluators seek to advance knowledge production in the field for the purpose of emancipation and social justice.
The lack of mechanisms to confront the participation of minority professionals in educational evaluation, especially those with experience in science and math education, are symptomatic of larger and more systemic issues. That is, understanding the dearth of minority evaluators means putting into perspective the condition of education for blacks, Latinos, Native Americans and other underrepresented groups much earlier in the educational pipeline.
Reporting on the situation for blacks and Latinos, the most recent National Center for Education Statistics (NCES) edition of The Condition of Education highlights well-recognized disparities (USDOE, 1999). Despite a pattern of declining gaps in National Assessment of Educational Progress (NAEP) performance scores reported between the early 1970s and mid to late 1980s, the average performance for both blacks and Latinos in math, science and reading have remained lower than those of whites throughout elementary and secondary schooling. By the time these same groups are ready to enter the next level of schooling, gaps between whites and blacks and between whites and Latinos in their rates of enrolling in college immediately after graduating from high school have increased. College completion rates, moreover, depict a similar story as black and Latino rates are lower than their white counterparts.
These patterns of achievement that manifest throughout the educational pipeline for blacks and Latinos are hardly novel; they continually speak to the worsening crisis in the education of these particular Americans. In fact, as Claude Steele has pointed out (1992), these disparate trends are the rule in most grade, high, undergraduate and graduate schools (including the most elite) as they pertain especially to African Americans. His message is alarming, but clear: despite beginning school with test scores fairly close to the test scores of white students their age, the longer blacks stay in school, the further they fall behind.
Framing the Problem of the Participation and Pipeline of Minority Evaluators
The problem of the participation and pipeline of minority professionals in educational evaluation cannot be fully explained and understood by the supply problem (i.e., the lack of "qualified" minority professionals is due to the small numbers of graduate level students) nor by deficit theories (i.e., the problem of academic achievement among poor and underrepresented groups stems from disadvantaged home and environmental backgrounds). Rather, the problem of the participation and pipeline of minority evaluators most directly lies within the arena and structure of schooling in the United States.
Within this schooling structure, as many critical educationists suggest (Apple, 1996; Bourdieu and Passeron, 1977; Giroux, 1983; House, 1999; MacLeod, 1995; Spring, 1997; Yeakey, 1981; Yeakey and Bennett, 1990), mechanisms and ideologies are constructed to prevent lower and working class individuals (and marginalized or underrepresented groups) from advancing into the upper strata of the social class structure. The very history of educational program evaluation surrounding achievement tests, as Edmund Gordon reminds us, has long promoted deficit-oriented explanations pertaining to minority and disadvantaged populations (1977:31).
Building Mechanisms to Increase Participation and Pipeline of Minority Evaluators: Focus on the American Evaluation Association
Increasing the participation and pipeline of minority professionals in educational evaluation inevitably involves increasing the number of minority scholars, particularly from underrepresented groups, in the discipline of educational evaluation. The role of professional associations, such as the American Evaluation Association (AEA), in addressing participation and pipeline issues by increasing the number of minority evaluators and building the capacity of all evaluators to work cross-culturally is now only beginning to take shape (AEA, 2000).
Due perhaps to the impetus from several important persons in key leadership positions in AEA, including the attention to greater inclusiveness of marginalized groups by major plenary speakers at recent annual conferences (Chelimsky, 1998; Kirkhart, 1995; Mertens, 1999; Stanfield, 1999; Weiss, 1998), the major evaluation association in the United States is taking an active role in building diversity among the evaluation community. The Building Diversity Initiative Phase I proposal, to extend from June 2000 - June 2001, suggests a number of tasks upon completion of its first year, including a directory of minority evaluators, a survey of evaluation training programs, the development of a diversity-building plan, publication of guiding principles for evaluators to work across cultures, and other expected deliverables. Questions remain, however, over the sustainability of the Building Diversity Initiative project after its first year of Kellogg funding, the nature of Phase II activities, and articulated efforts to address the educational pipeline for minority evaluators.
Another potentially promising mechanism to address the participation and pipeline of minority professionals in educational evaluation includes the nascent development of a mentoring initiative within the Minority Issues in Evaluation Topical Interest Group (TIG) of AEA*. With AEA board support to provide three $300 travel award scholarships for students to attend this year's annual conference, the executive committee of the Minority Issues in Evaluation TIG has begun steps to organize a mentoring activity pairing the travel award winners (i.e., protege) with more experienced members of the TIG (i.e., mentors). Questions here concern the one-time nature of the travel award scholarships from the AEA Board, the support and/or endorsement of the Board for a coordinated, mentoring effort, and the exact nature and activities of the mentoring program. As Hood and Boyce (1997) point out from experience in developing mentoring activities within professional organizations, structured efforts to mentor future educational researchers must be multifarious, from awarding fellowships and other financial awards to developing organizational structure and policies to reflect a level of commitment in achieving a culturally diverse membership within the association and the profession.
Promoting the active participation of minority professionals in educational evaluation is an important goal for professional associations and the larger field of evaluation. Inherent in the efforts to build mechanisms to increase minority evaluators should be the concomitant realization of nurturing budding evaluators toward sustainable diversity and leadership within the ranks as well as promoting social justice and liberating evaluation frameworks.
|EHR Home | nsf.gov
||| About NSF | Funding | Publications | News & Media | Search | Site Map | Help|
|The National Science Foundation
4201 Wilson Boulevard, Arlington, Virginia 22230, USA
Tel: 703-292-5111, FIRS: 800-877-8339 | TDD: 703-292-5090