The Cultural Context of Educational Evaluation

The Role of Minority Evaluation Professionals

June 1 - 2, 2000
Arlington Hilton and Towers Hotel
Arlington, Virginia

WORKSHOP RECOMMENDATIONS and CLOSING REMARKS

WORKSHOP RECOMMENDATIONS

At the close of Session Three, the Session Chair asked participants to reflect upon two days of discussion and provide guidance to NSF. Participants identified broad themes, outreach mechanisms and products for further study. The following is a list of recommended actions.

1. NSF's User Friendly Handbook for Project Evaluation should be updated to encompass and respond to the points specified below:
• Cultural awareness of the environment from which the participants are drawn must be emphasized.
• Test results must be reported with context data.
• Disaggregation of program data should include, as appropriate, factors such as, but not limited to, race, gender, socioeconomic status and opportunity to learn.
• The level of implementation of a program/intervention must be coupled with achievement results.
• Evaluations must recognize that the culture of students influences how they respond to the assessment process and assessment items.
• Consideration must be given to the impact of teachers' attitudes, beliefs and behaviors on student achievement because low-income and minority students are more teacher dependent.
2. Non-minority evaluators should be trained to evaluate programs that target minority students.
3. NSF should fund training where the greatest pool of potential minority evaluators can be located and trained.
4. Training of minority evaluators should be conducted by a team that includes minority education evaluation faculty/trainers.
5. NSF should fund evaluator internships to eliminate barriers to advanced training:
   • Regular students should be funded at the prevailing wage level for internships.
   • Non-traditional students should be funded at the level of their current salary.
6. Evaluation training must include traditional evaluation courses such as: methods and statistics; non-traditional courses such as multi-cultural, socio-cultural and linguistic cognitive factors; a holistic and systemic approach that considers both internal and external factors that influence the process and focuses on components and the relationship between them; practical experiences of conducting evaluation; and support from mentors.
7. NSF should fund the identification and development of a database of practicing minority evaluators, to be added into an existing database.
8. NSF should establish collaboratives between university evaluation training sites and consortiums of school districts having administrators and staff interested in becoming evaluators.
9. NSF should fund a research study that captures from minority evaluators those experiences that led them to become evaluators.
10. NSF evaluation training program sites should have a critical mass of trainees so that support mechanisms can be planned and implemented.
11. NSF evaluation training should be provided at locations conducive to reaching Native Americans,perhaps at Haskell Indian Nations University in Kansas.
12. NSF should fund evaluations/studies of successful programs that are encouraging involvement of minorities in math and science, such as MESA.

Back to Top

CLOSING REMARKS

Elmima C. Johnson
Staff Associate
EHR/NSF

This two-day meeting has raised more questions than it answered. However, it did clarify the questions to be asked and sharpen our focus. As a result, I can list several potential activities that would respond to the discussion and recommendations emanating from the Workshop.

Directorate actions will begin with the compilation of the proceedings of the workshop. This document, which will include the papers and presentations as well as summaries of the discussions and recommendations, will serve as a reference and blueprint for designing a strategy to respond to participant concerns regarding capacity-building and NSF's role in evaluator training and education. To date, EHR capacity-building activities have been primarily ad hoc and none have focused specifically on the issue surrounding culturally relevant evaluation. These have been relatively small efforts suggested by the field, and informal feedback suggests they were successful. We recognize that a specific rationale and framework should guide future efforts. Additionally, any proposed efforts should be defined as part of a more comprehensive approach.

Before we can develop the framework we need additional information in several areas. First, we need a comprehensive picture of NSF-supported training opportunities. To this end we are supporting a project to provide a detailed description of our funded efforts and their success. We are also exploring other formal evaluation training approaches and related efforts that can be adapted for our needs. This is being accomplished through a broad-based literature review to identify other evaluation models and potential models, i.e., from disciplines other than mathematics and science education.

Second, we need to determine the demographics of the current population of "evaluators," including minority evaluators. Two manpower surveys are under consideration. One would survey graduates of formal evaluation training programs, i.e., university-based efforts. The second survey would target practicing evaluators without formal training in evaluation methodology. The first task will be to define the two populations and there probably will be some overlap. Both would include a special effort to identify minority evaluators.

At some point we will need feedback from our stakeholders. We will utilize the recommendations offered by workshop participants regarding evaluator training and updating NSF evaluation publications to include a focus on the multicultural context of evaluation. We also plan to talk with representatives of national evaluation associations. For example, we could hold discussions with officials of the American Evaluation Association (AEA) regarding their efforts to diversify the pool of evaluation professions. Another issue to consider is how to attract more of the minority social science graduates into the field of evaluation.

These steps, when completed, should prepare us to set goals and priorities and define the parameters of future NSF efforts. The framework developed must be flexible enough to accommodate new practices and new Federal mandates regarding accountability stemming from the Government Performance and Results Act (GPRA). The question to be answered is what does the directorate want to accomplish in addressing the need for culturally relevant evaluation within the GPRA and diversity contexts.

Back to Top

 

<-- Back

Next -->