The Formative Assessment for Computer Science (PFACS) project is designed to support middle school teacher?s formative assessment practices of students? ability to understand and analyze data as it relates to the domain of computer science. To address this goal the project used a combination of evidence-centered and participatory design approaches to develop a formative assessment game, Beats Empire. In the game, students take on the role of a music studio and make decisions about which artist to hire and what type of songs to record using data about listeners? interest in various music genres, topics, and song moods. The game focuses on CS concepts and practices related to data collection, storage, visualization, and interpretation. In addition, the PFACS project developed a dashboard that provides teachers with actionable information about how students are playing the game, as well as a set of unplugged formative activities and teacher guides for these activities. 

A key design outcome of Beats Empire was the development of a game system that allowed students to make personally meaningful choices throughout the game, choices that could also be assessed for their connection to target concepts. Rather than standardize the game experience, or require all players to ?correctly? answer questions or perform tasks, Beats Empire invites players to make predictions about music trends with in-game data. Playes then indicate which data they used to make these predictions, allowing the teacher and game system to evaluate whether these decisions are data driven. This system ensures assessment moments make sense in the game?s narrative and that these in-game actions are useful and meaningful to students? play.

To obtain feedback on the game the team conducted cognitive interviews with students during game play, interviews with teachers after they had used the game in their classroom and several small pilot studies. The formative feedback sessions with students confirmed many of our foundational assumptions about middle school students? interest in music and games.  We found students? game literacy was very high, with little direction or help needed to understand how to play this game about data and the music industry. We also found that students had a strong interest in music and the music industry, and that music was a core part of their social experience. This finding highlights the importance of developing games that aligns with the experiences of young people and made explicit the need to ensure the game?s representation of the music industry was authentic.

Results from the pilot studies showed overall high engagement of students during gameplay. Students overwhelmingly enjoy the game and are enthusiastic about engaging meaningfully with the game mechanics. By collecting clickstream data of how players progressed through the game, and how they used data to make predictions in the game, we were also able to differentiate students that used data differently in the game. This log data revealed that students ranged from those who rarely used data, to those who consistently were able to draw meaningful inferences from data. The game was also able to detect those students that changed their use of data over time. We found that students were less likely to engage with aspects of collection and storage in the game, but this may be because students were able to progress without this level of engagement. These findings indicate play in Beats Empire can help to distinguish between students that have varying capabilities using and drawing inferences from data, and that this information can be communicated clearly to teachers through the included dashboard. 

When examining the relationship between developed "unplugged" activities and game play we found that the use of the classroom activities had some effect on how students played hte game. For instance, after students engaged with an activity in class that had them comparing information found from line graphs and bar graphs, we found that students were more likely to use the line graph in the game. We saw similar behavior when students engaged with activities around data storage and collection. 

Results from the teacher interviews indicated that teachers found the game to be a useful tool. Teachers felt that Beats Empire is a good way to engage students with the concepts of data and analysis. Teachers also indicated that the unplugged activities have value and they saw the connections between the two and how they could use both the game and the activities in future instruction. 

Overall, we believe that this project demonstrated the potential for the use of formative assessment games to measure students? engagement with data science practices.  We found that designing a formative assessment game required a balance of incorporating ways to keep students engaged with the game, while also collecting information that would allow you to make inferences about students evolving knowledge and skills.

Last Modified: 11/07/2022
Modified by: Nathan Holbert