| NSF Org: |
DRL Division of Research on Learning in Formal and Informal Settings (DRL) |
| Recipient: |
|
| Initial Amendment Date: | September 5, 2023 |
| Latest Amendment Date: | September 5, 2023 |
| Award Number: | 2334631 |
| Award Instrument: | Standard Grant |
| Program Manager: |
Amy Baylor
abaylor@nsf.gov (703)292-5126 DRL Division of Research on Learning in Formal and Informal Settings (DRL) EDU Directorate for STEM Education |
| Start Date: | September 15, 2023 |
| End Date: | August 31, 2024 (Estimated) |
| Total Intended Award Amount: | $200,000.00 |
| Total Awarded Amount to Date: | $200,000.00 |
| Funds Obligated to Date: |
|
| History of Investigator: |
|
| Recipient Sponsored Research Office: |
107 S INDIANA AVE BLOOMINGTON IN US 47405-7000 (317)278-3473 |
| Sponsor Congressional District: |
|
| Primary Place of Performance: |
902 W. New York St Indianapolis IN US 46202-5155 |
| Primary Place of
Performance Congressional District: |
|
| Unique Entity Identifier (UEI): |
|
| Parent UEI: |
|
| NSF Program(s): | ITEST-Inov Tech Exp Stu & Teac |
| Primary Program Source: |
|
| Program Reference Code(s): |
|
| Program Element Code(s): |
|
| Award Agency Code: | 4900 |
| Fund Agency Code: | 4900 |
| Assistance Listing Number(s): | 47.076 |
ABSTRACT
Large language models (LLM) represent a new and rapidly changing technological advancement for K12 STEM learning. It is critical at this point in time to investigate and provide pathways for including justice, equity, inclusion, and community cultural capital and wealth in designing LLM-based educational systems. In the context of developing an AI chatbot, this RAPID project will research the ways in which teachers can plan universally designed and culturally relevant and responsive K12 STEM learning activities and environments. This research will contribute to an improved LLM that will incorporate novel methods to incorporate community data and reinforce knowledge from user communities into the LLM that is largely missing from large text corpora on which LLMs are usually trained. The AI chatbot will increase teacher capacity to create more inclusive STEM activities and support career pathways by facilitating the inclusion of more underrepresented learners in STEM careers. This proposal was received in response to the Dear Colleague Letter (DCL): Rapidly Accelerating Research on Artificial Intelligence in K-12 Education in Formal and Informal Settings (NSF 23-097) and funded by the Innovative Technology Experiences for Students and Teachers (ITEST) program, which supports projects that build understandings of practices, program elements, contexts and processes contributing to increasing students' knowledge and interest in science, technology, engineering, and mathematics (STEM) and information and communication technology (ICT) careers.
The project will convene a series of focus groups to elicit training data for the chatbot, adopting a community-based participatory action research approach. This approach recognizes that AI can foster and grow community well-being by including the community in the design, orienting the AI to address community issues, and adopting an interdisciplinary and systems-based stance. The AI training will be sensitive to cultural nuances and techniques like sentiment analysis will help to understand the context and ensure culturally appropriate responses. Human-centered AI methods will be used to continuously incorporate user feedback by deploying the chatbot and actively seek responses from the diverse set of participants. This process will reinforce knowledge from user communities that is largely missing from content on which AIs are typically trained, producing an AI system that will generate more culturally aware text. Importantly, the project will create a model for developing community sourced AI LLMs that can continue to be refined and researched. A beta-level chatbot will be made available for teachers to improve their lesson plans, activity structures, and learning environments by the end of the project year for further research and development.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
We conducted research to begin the development of CATpc: Critical Activity Teacher Planning Companion. CATpc is a generative AI (genAI) chatbot or "copilot" for teachers to support them in adapting and creating culturally focused and universally designed STEM activity structures and learning environments while honoring, sustaining, and building upon the cultural capital and wealth of communities (Habig et al., 2021; Yosso, 2005) that students bring into the classroom.
Our research explored two intertwining research questions: 1) What are the directions, features, and dispositions that educational researchers, teachers, and families of marginalized and underrepresented students to design a genAI chatbot to support teachers create culturally relevant activities and lessons 2) What are possible technical approaches to train and tune a genAI chatbot that reflects and represents these features and dispositions?
To explore question 1, we convened a series of virtual listening sessions with four groups: 1) educational and AI experts from across the US; 2) teachers; 3) families; and 4) a combined group of teachers and families, a subset of the original teachers and family member groups.
The teachers and families were affiliated with an elementary school in a district that is a member of the Council of Great City Schools in an urban neighborhood of a large city in the Midwest.
The results of the expert panel highlighted emerging tensions, such as:
- Accuracy vs. Errors, a concern to genAI generally, that while a chatbot’s output is useful, it can include errors that look correct (often referred to as hallucinations);
- Efficiency vs. Critical Thinking, that the genAI may facilitate increased efficiency and speed in creating educational materials, but lessen the teacher’s tendency to question the purposes, enactments, and outcomes of the learning experiences;
- Ideal Case vs. Reality, that the genAI may generate outputs that are aimed at an ideal case of a classroom that does not exist, rather than the actual dynamic and diverse conditions of real classrooms in real schools; and similarly,
- Local Wisdom vs. Universal Assumptions, that despite the intents, given genAI’s tendencies to create outputs that convey broad, generalized, and surface-level statements, locally- and culturally-situated knowledge and practices, particularly in STEM, will be further hidden from view in the classroom.
The teachers and families, who are closer to the students and their communities, sometimes reinforced the experts’ ideas and concerns, and at other times viewed them as opportunities. The teachers and families emphasized:
- Instilling in the teacher a genuine love for students, honoring each individual student for who they are, where they live, and what they carry with them;
- Attending to the socioemotional needs of individual students, such as engaging students who may need to talk about events in or out of school in a supportive conversation, alerting the teacher of the student’s emotional state; and
- Managing auxiliary and mundane classroom tasks, such as pulling up presentations for teachers, facilitating station activities, or completing attendance, freeing the teacher up to teach.
To address question 2, a preliminary version of the AI chatbot is currently in the data preparation and fine-tuning phase. This phase involves leveraging a custom script with the open-sourced Llama 3.1 model to generate culturally inclusive STEM lesson plans, refine conversation guidance, and address knowledge gaps to optimize performance. We are designing and testing prompt scripts to guide the Llama 3.1 8B model in assisting teachers with creating culturally relevant lesson plans. The adaptable script inquires about classroom context, lesson goals, and educational standards, prioritizing cultural inclusivity. The testing phase ensures the model's responses are accurate and aligns with educational outcomes. We also initiated the creation of a structured dataset from research paper summaries to fine-tune a conversational model on culturally relevant pedagogy. Additionally, we developed external data for RAG model design and testing, extracting text and metadata from documents and converting them into vector embeddings stored in a PostgreSQL database. The RAG model augments user input by combining retrieved chunks with the query to generate contextually accurate responses. Overall, we successfully enhanced the AI's cultural responsiveness and demonstrated the RAG model's capability to generate culturally inclusive responses, with implementations offering flexibility and scalability.
This research has compelled us to investigate the opportunities and risks further. We seek to examine innovative ways to navigate the tensions highlighted by the experts and to implement the ideas of the teachers and families more diligently in future versions of the genAI chatbot, while maintaining rigorous and accurate STEM content.
Last Modified: 08/29/2024
Modified by: Jeremy F Price
Please report errors in award information by writing to: awardsearch@nsf.gov.
