The Computer Science in Secondary Schools (CS3) project designed assessments for a high school computer science (CS) course and used them to conduct research on the factors that influence student learning. K-12 CS teachers face challenges because they have to learn new ideas and technologies (i.e., programming languages and Web design tools) and implement new teaching practices. Well-written curricula such as Exploring Computer Science (ECS) with its supportive professional development (PD), can help but are not a guarantee. To ensure that curricula like ECS succeed, the CS3 research project studied relationships among teacher characteristics, teaching approaches, and curriculum adaptations to see how these factors influenced student learning of computational thinking practices.

The project involved two years of iterative development and pilot testing of measures, followed by two years of data collection with up to 40 teachers and 1133 students. Led by SRI Education in partnership with the authors of ECS, researchers studying ECS, and school districts teaching the curriculum, the project developed and validated measures (tests) of students’ proficiency with computational thinking practices. Validity evidence included student test scores, student interviews, and teacher feedback, and this evidence was used to improve the assessment tasks. In the study, students were given a pre and posttest to measure learning gains. They also took 4 end-of-unit assessment tests, for a total of 6 assessments of student learning.

The project created ways to measure teaching quality (what teachers do that helps students learn? To what extent do teachers’ instructional approaches reflect the underlying principles of ECS) and curriculum enactment (i.e., Are all lessons and units completed in order? Are lessons modified?) and increased our understanding of what helps teachers effectively teach computer science. Across multiple surveys, teachers were asked about their background, the PD they were given, and their experience with each unit. 

The objectives of the research were to understand what factors enhance or impede the successful implementation of ECS and how implementation relates to student outcomes.  Characterizing the relationship between curriculum implementation and student outcomes required a statistical analysis of the data collected in order to integrate teacher and learning context attributes, teaching approaches, and curriculum adaptation. We used the end-of-unit test as an outcome measure, and student scores on the pre-assessment were used in the model as well to account for variation among students. We systematically explored all factors to see which were significant.

The factors that related to student outcomes varied by the CS topic covered in the unit. The most significant and salient results from the analysis were:

–  Students whose teachers had higher unit inquiry (for Unit 3) and/or collaboration practices (for Unit 1) performed significantly better on the assessment for that unit, after accounting for student- and classroom-level covariates.

–  More experience teaching CS (but not ECS) was related to lower student performance

–  More experience teaching ECS was related to higher student performance

–  CS presence in the school aided learning of basics about computers and human-computer interaction

We confirmed that teachers who had difficulty accessing technology modified or skipped lessons.  Modifications of Units 1 and 3 was tied to lower student performance, while skipping topics in Units 2 and 4 was tied to lower performance. This indicates that care should be taken when deciding to modify or skip a lesson, because then students may then not have the opportunity to achieve the expected learning goals.

This study contributes, for education researchers, a model of the factors that can be considered when examining CS curriculum and instruments to measure these factors. The model links indicators of teaching quality and curriculum enactment to key attributes of the ECS teachers and learning contexts. This study demonstrates that the factors for success in an introductory computer science course may not be the same across all CS concepts or CS classrooms and points to a complex relationship among teacher/classroom characteristics, teaching quality, curriculum enactment, and student success.

Our 4 end­-of-­unit and cumulative assessments are still being used by teachers of ECS, judging in part from activity on the CSForAllTeachers.org website. Links to overview documents, research articles, and presentations are available at the project website, PACT.sri.com. The assessments and instructions for scoring them are available to ECS teachers via the CSForAllTeachers.org community web site. With the assessments, teachers are now better able to understand what their students are learning, and to better prepare them for future coursework in computer science, thus creating pathways for students to careers in computer science. Working with the ECS curriculum has enabled us to do this for populations that are typically underrepresented in computer science.

Last Modified: 08/09/2019
Modified by: Daisy W Rutstein