The primary goal of MOSART-HSPS was to develop rigorous assessment tools that can generate evidence-based measures of teacher and student understanding of high school-level physical science concepts in chemistry and physics. The project achieved this goal by first developing close to 1,000 distractor-driven multiple-choice items based on the grades 9-12 Next Generation Science Standards (NGSS) for Physical Science (divided into the domains of chemistry and physics), using the related peer-reviewed research on student misconceptions. Items passed through several development stages, including review by scientists, pilot-testing using crowd-sourced subjects, and national field-testing with high school students. Multiple 30-item assessment instruments were constructed to represent a range of difficulties, have high discrimination, low gender bias, high misconception strength, and cover all disciplinary core ideas in the Next Generation Science Standards (NGSS). Chemistry and physics assessment instruments are available on the newly revised MOSART Self-Service website for use at no-cost by teachers, professional development providers, researchers, and other science education stakeholders (https://waps.cfa.harvard.edu/mosart/).

Secure versions were developed for use by researchers and employed by our team to study effectiveness of teaching science concepts with a national sample of 6,000 chemistry and physics students. Administered as pre/post-tests at the beginning and the end of the school year, teachers answered the same content questions as students to determine their subject matter knowledge (SMK). Teachers also predicted the most common wrong answer among their students for each item to identify their knowledge of student misconceptions (KOSM). Additionally, students identified the classroom pedagogies that teachers chose to employ.

The relationship between teachers’ subject matter knowledge and student gains.
Student learning has long been thought to depend on the extent of teacher knowledge. We found this to be the case; students with teachers who had high SMK achieved more. By building a model that predicted students’ likelihood of answering a posttest item correctly as a function of the misconception strength of that item, teachers’ knowledge (SMK & KOSM, SMK-Only, No-SMK), and student pretest correctness of the item, we found that student learning to be highly dependent on both teacher knowledge of subject matter and of student misconceptions. This was particularly evident for items with “strong misconceptions,” those that are conceptually most difficult for students. The general pattern shows that students’ correct rate is low when teachers’ SMK is low and declines drastically with misconception strength in the mid-range. As the misconception strength increases, students whose teachers do not have the SMK for a specific item will find the item more difficult in the posttest (controlling for the pretest); students whose teachers have only SMK but not KOSM will be minimally affected by the item misconception strength, and, most interestingly, students whose teacher have both SMK and KOSM will find the item easier. When the misconception strength is low, teachers in the SMK & KOSM and the SMK-Only group yielded similar students’ correctness in the posttest.

Pedagogical practices associated with student gains
We also examined the association between a variety of classroom pedagogies and student gains for both chemistry and physics. An example of such an activity is “asking students to make predictions.” A comparison of gains of students whose teachers asked them to make predications vs. those of students whose teachers did not ask them to make predictions indicates that, while both groups’ pretest scores were almost identical, the group that made predictions had higher posttest scores than the group that did not make predictions.
Following this example, we estimated the effect of each of 31 pedagogies on students' gains by running a list of regression analyses that predicted students’ posttest scores as the function of each pedagogy, while controlling for background information such as gender and race/ethnicity. There are both similarities and differences in the pedagogical effects between high school chemistry and physics. An interesting similarity is that field trips and guest speakers had negative effects, while giving lectures, making connections to life or other disciplines, having students ask or answer questions, in addition to making predictions, had positive effects in both disciplines. The most noteworthy difference between the two disciplines was that hand-on activities, collecting data, doing homework, and memorizing facts had a positive effect on students’ gain in high school chemistry, but not in physics.

Last Modified: 12/30/2022
Modified by: Philip M Sadler