Award Abstract # 1943530
EAGER: Seeding an Assessments Hub and Catalyzing a Community of Educators for Student Success in CS (SUCCESSinCS)

NSF Org: CNS
Division Of Computer and Network Systems
Recipient: LOOKING GLASS VENTURES LLC
Initial Amendment Date: August 22, 2019
Latest Amendment Date: October 15, 2020
Award Number: 1943530
Award Instrument: Standard Grant
Program Manager: jeffrey forbes
CNS
 Division Of Computer and Network Systems
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: October 15, 2019
End Date: June 30, 2022 (Estimated)
Total Intended Award Amount: $249,999.00
Total Awarded Amount to Date: $299,950.00
Funds Obligated to Date: FY 2019 = $249,999.00
FY 2020 = $49,951.00
History of Investigator:
  • Shuchi Grover (Principal Investigator)
    shuchi.grover@gmail.com
  • Shivram Venkatasubramaniam (Co-Principal Investigator)
  • Bryan Twarek (Co-Principal Investigator)
Recipient Sponsored Research Office: Looking Glass Ventures, LLC
801 W 5TH STREET
AUSTIN
TX  US  78703-5465
(650)380-3627
Sponsor Congressional District: 37
Primary Place of Performance: Looking Glass Ventures, LLC
202 Sequoia Avenue
Palo Alto
CA  US  94306-1043
Primary Place of Performance
Congressional District:
16
Unique Entity Identifier (UEI): HJCKFH4BP161
Parent UEI:
NSF Program(s): CSforAll-Computer Sci for All,
IUSE
Primary Program Source: 01001920DB NSF RESEARCH & RELATED ACTIVIT
01002021DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 023Z, 7916
Program Element Code(s): 134Y00, 199800
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

This project launches crucial exploratory research and development activities to (1) birth a Community of Practice of K-12 CS teachers in the US centered on a shared need for quality assessments and building CS teachers' assessment capabilities and (2) lay the foundation for a sustainable technology hub (created for and by teachers along with researchers and technologists) of rich, innovative assessments organized by CS standards, grade, concepts, curriculum, and other relevant criteria for intuitive use by teachers of all experience levels.

The project will be carried out by an interdisciplinary leadership team from Looking Glass Ventures, LLC (creator of the National Science Foundation funded assessment authoring and delivery platform "Edfinity") and Computer Science Teachers Association (CSTA), in collaboration with K-12 CS teachers, the CSForAll community, and experts in the field (as advisors). Key activities include: (a) bringing together teacher experts to aggregate, create, taxonomize CS assessments; (b) creating a classroom assessment training module for use in teacher training and workshops; (c) planning and conducting multiple teacher workshops co-located at conferences and other venues where the K-12 CS community convenes; and (d) conducting research on the development of a community of practice and classroom use of CS assessments on Edfinity. The research and development activities will be guided by the following questions:

(RQ1) To what extent does the Edfinity platform and its features support development of an assessments hub of rich formative and summative CS assessment that covers CS standards, grade levels, and topics, and can be created and used by teachers for sharing and classroom teaching?

(RQ2) What is the value of the workshops and the assessments hub for teacher community engagement and for supporting shifts in teachers' vision for, and participation in, an assessment-driven community of practice of CS teachers?

(RQ3) (a) To what extent does the use of the assessments hub and the classroom assessment training help teachers' confidence around, and understanding of, formative assessments of CS? (b) How do teachers' classroom practices change with the use of formative assessments? (c) How does the use of formative assessments in the hub help teachers better track student understanding and their ability to get timely feedback about students?

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Grover, Shuchi "Toward A Framework for Formative Assessment of Conceptual Learning in K-12 Computer Science Classrooms" 52nd ACM Technical Symposium on Computer Science Education (SIGCSE 21) , 2021 https://doi.org/10.1145/3408877.3432460 Citation Details

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

This CSForAll:RPP EAGER effort led by Looking Glass Ventures (Shuchi Grover, PI, Shivram Venkatasubramaniam, Co-PI) and CSTA (Bryan Twarek PI) focused on assessment of and for student learning-- building capacity (in terms of infrastructure and teacher preparation) to provide K-12 CS teachers with the knowledge and resources needed to measure student learning of CS (especially programming) with a focus on improving learning outcomes. In order to address the crucial need for quality formative assessments for classroom feedback to improve classroom teaching and learning of CS, we aimed to:

  1. Create a rich repository of assessments for and by teachers on Edfinity.com, an NSF-funded technology-rich platform for assessing learning, and examine its use by teachers. 
  2. Build capacity and support CS teachers in using formative assessment through the design and implementation of a multi-session assessment literacy-focused teacher PD "Formative Classroom Assessment for Teachers (FCAT)".   

Over the course of the project the CSAssessmentsHub was created and seeded

  1. It includes over 1200 items in the "K-12 CS repository" that were added to Edfinity by teachers who participated in the project. 
  2. Many researchers, teachers, and curriculum providers (such as Code.org, MobileCSP, Beauty & Joy of Computing, UTeachCS) contributed their items to this effort. 
  3. These items are organized by a robust taxonomy that was created and iterated upon with the help of our teacher partners for intuitive organization by CSTA standards, curricula, grade level, CS topics, and other relevant criteria (such as those for AP CSP and AP CSA). 
  4. There are free format tags that can be added to each problem such as the programming language in which the problem is coded, the curriculum it maps to, etc.
  5. The search and discovery capabilities in Edfinity for the CSAssessmentsHub were enhanced to search on these tags as well.
  6. Edfinity.com innovated to create several various new problem types that are inspired by K-12 CS education research. Thare new types are engaging, interactive, and intuitive compared to the usual multiple choice and multiple select types.  
    1. Sequence - to support Parson's problems
    2. Hotspot interaction - to support items that include responses corresponding to part of an image (like code snapshots)
    3. Point & Click - a variation of the Hotspot type
    4. Matching Column - to support Code.org courses that embed matching column formative assessments
    5. Code Correctness - supporting Python unit-test items
    6. Label image - to support subgoal-labeling type problems
  7. We also created master courses for AP CS A, and generic high school and middle school topics covering the typical topics for those grade levels. 

Three teachers each in years 2 and 3 of the project used Edfinity and provided feedback on their experiences. About 300 students in all were involved in the trial use of Edfinity in secondary grades. Teacher feedback was largely positive but there also provided suggestions on how to improve the CSAssessmentsHub both in terms of content and user experience. 

In all, about 15 teachers were engaged as "teacher partners" and "teacher experts" on this project for building out the CS taxonomy, seeding the assessments hub, dissemination activities, and participation in classroom research.  

 

Another noteworthy accomplishment was the creation and piloting of a first-of-its kind Formative Classroom Assessment for Teachers (FCAT) "PD-in-a-Box" that we have made freely available on the project website https://csassess.org. FCAT is designed to deepen teachers' understanding of how to adapt and use formative assessment in K-12 CS classrooms. In this multi-part series, we present a synthesis of research and many examples, plus interactive activities to practice evaluating, adapting, and responding to assessment items. Examples cover content related to algorithms and programming, with examples across grades 3-12. We also created supporting community resources covering granular learning goals and a misconceptions corpus for introductory programming topics. Teacher feedback to the workshops was overwhelmingly positive and indicated that FCAT filled a much-needed gap.

 

Dissemination & Outreach to Teachers. An early blueprint of the approach in this project was articulated as a Framework for Formative Assessments for K-12 CS that was presented as a paper at SIGCSE 2021 and at the Raspberry Pi Foundation CS education seminars. We also presented our work on innovative assessments design in Edfinity st SIGCSE 2022. Besides the FCAT module pilots, we conducted online workshops alongside SIGCSE (2020), and CSTA (2020, 2021), and presentations at the CSTA Chapter Leaders Summit (2021, 2022).  Through these outreach activities we directly reached about 700 teachers and and through the ~100 CSTA chapter leaders we reached thousands of CS teachers in the US

 

Covid-19 Impact. The work of this project was conducted in 2020, 2021, and 2022. Needless to say, teacher availability and engagement was a big issue during these years due to Covid-19 disruptions and challenges. Given these constraints, we believe the project managed to accomplish its proposed goals, although engagement with the teacher community might have likely been higher had it not been for Covid.


Last Modified: 10/20/2022
Modified by: Shuchi Grover

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page