This research focused on emotionally-laden experiences with algorithmically curated content. Across six studies, we investigated the interplay of human and computational understandings of data with a focus on when these understandings breakdown. The major goals were (1) to understand how people navigate sensitive algorithmic encounters; (2) how these encounters impact people; and (3) what social concepts algorithms are failing to understand.

Our studies fall into two different focus areas:

Sensitive Algorithmic Encounters

Encounters with Ex Romantic Partners: We conducted a study of upsetting algorithmic encounters on social computing systems focused on encounters with ex-romantic partners. We found that upsetting encounters were exacerbated by the complex social networks and data that enable inferred connections around otherwise explicit (and often explicitly terminated) relationships -- a concept we termed as ?data peripheries.? We show how designing for peripheries can allow technologists to conceptualize differences between explicit and inferred connections, and how to design agency into systems based on each. Next, we conducted a study focused on encounters with digital objects and relationships following a break-up. We found that people took different actions towards their possessions and connections in service of creating a post-break-up identity. We found that existing tools are ill-designed to support competing desires to present authentic past and future online identities and produced design suggestions to address this tension. 

Encounters through Technologically Mediated Reminiscence (TMR) Tools: Recommender systems sometimes inadvertently curate content that users may find emotionally intense, such as a picture of an ex-partner or of a now-deceased family member. To explore this issue, we examined Facebook's Memories feature, a Technology-Mediated Reflection (TMR) system that shows users content about their pasts in order to prompt reflection. We interviewed 20 people who had recently seen sensitive curated content through this suite of features. We found that they wanted to see ?bittersweet? content, but they preferred to see it when it was expected and viewed in a context they felt was appropriate, and when they were able to make sense of why the recommender system curated the content. We recommend that designers engage in three practices to meet users? needs: (1) draw inspiration from no/low-technology artifacts, (2) use empirical research to identify which contextual features have negative impacts on users, and (3) conduct user studies to determine how users are doing sense-making around the perceived affect of recommender systems.

Everyday Evaluations of Social Media Metrics: The Like button is simultaneously a means of social interaction and a tool to evaluate social media content. Based on in-depth interviews with 25 artists who use Instagram, we identified three overlapping orientations to the Like button: affective, relational, and infrastructural. We found that the flexibility of the button creates ambiguity around the meaning of a Like that incentivizes an economic approach to evaluation that crowds out other value schemas, shaping how artists use the platform, make art, and even understand themselves.

Algorithmic Infrastructure and Bias

AI systems have been critiqued for designs that cannot capture the nuance of human identity. However, there was scant empirical work at the time. To address this gap, we conducted a study of how gender is operationalized into commercial facial analysis services. We conducted a two-phrase study: (1) a system analysis of ten commercial facial analysis and image labeling services and (2) an evaluation of five services using a custom dataset of diverse genders using social media images with gender labels provided by their creators. We found that services performed consistently worse on transgender individuals and were universally unable to classify non-binary genders. We found that bias is the result of more than just engineering practices and training data. We identified bias in how gender is codified into the data standards that surround (and typically pre-date) classifiers. We demonstrate how the cloud based services we studied provide black-boxed infrastructures to many third-party developers that can include subtle and invisible forms of bias. 

Next, we investigated race and gender bias in computer vision by examining the databases with which models are built. Race and gender have long sociopolitical histories of classification in technical infrastructures?from the passport to social media. In this study, we specifically focused on how race and gender are defined and annotated in image databases used for facial analysis. We found that the majority of databases rarely contain underlying source material for how those identities are defined. Further, when they are annotated with race and gender information, database authors rarely describe the process of annotation. In our publication we discuss the limitations of these approaches and argue that the lack of critical engagement renders databases opaque and less trustworthy. Our work encourages and provides guidance for database authors to address both the histories of classification inherently embedded into race and gender.

Last Modified: 08/29/2021
Modified by: Jed R Brubaker