The primary goal of this project was to improve language generation technology for multilingual users. Most people are multilingual, yet language generation technologies (like chat bots) assume that users will prefer to communicate in a single language - often English, which is not even the most common native language in the world.

In the first phase of the project, we aimed to better understand why bilinguals switch languages (or "code-switch") when conversing with other bilinguals, and to improve computers' ability to accurately predict when code-switches will take place. We trained state-of-the-art deep-learning models to predict code-switch points within an already-existing corpus of Spanish-English conversations, testing several new approaches to improving such predictions (see Figure 1 from Ostapenko, Wintner, Fricke, & Tsvetkov, 2022). Our main findings were that (1) predictions were better when models were given information about speakers' language background and preferences, and (2) our speaker-aware models were also able to learn useful code-switching-related linguistic cues (which the non-speaker-aware models missed).

Next, we began developing a new method for collecting our own large-scale data sets of multilingual conversation. To collect data related to the needs and preferences of multilingual users, we adapted a well-known task from the linguistic literature, the Map Task, for use in text conversations with Spanish-English bilinguals. The Map Task has often been used by linguists for studying patterns in human conversations, because it results in natural, conversational speech while still being somewhat constrained. In the standard version of the task, one participant (the "direction giver") has a map with a path passing through various landmarks, while the other participant (the "direction follower") has a map with no path. The goal is for the direction giver to successfully direct the follower along the correct path, without either participant being able to see the other's map.

In Berríos, Swain, and Fricke (2023), we explained how we developed our maps using extensive testing with English and Spanish speakers. We verified that our maps resulted in linguistically interesting dialogs, and also documented the types of vocabulary items they elicited. We provided a thorough overview of methodological considerations for researchers wishing to develop their own Map Task, and described the task's utility for researchers working in a variety of research areas.

In the next phase, we used our maps to collect a large corpus of English-Spanish human-machine text dialogs (see Figure 1 from Geckt, Fricke, & Wintner, under review). To this end, we also developed a chat bot capable of completing the Map Task while conversing in code-switched English and Spanish (see Figure 2a from Geckt et al.). In the course of developing our bot, we tested how participants responded to different "strategies" of code-switching; that is, we programmed our bot to code-switch following different linguistic patterns, to see whether such differences affected participants' task performance or satisfaction.

We found that users strongly preferred code-switching patterns that aligned with their previous experience and expectations. When our bot switched languages randomly, users reported lower task enjoyment and more difficulty communicating with the bot. When our bot consistently used a grammatical structure that has previously been described in linguistic studies of Spanish-English code-switching, users completed the task faster and more accurately; when the bot used an analogous structure that has not been attested (and is therefore quite unexpected), users reported lower task success and enjoyment, and greater difficulty communicating.

Our findings underscore the importance of language technology "getting it right". If chat bots are unable to produce the linguistic patterns expected by their users, then user satisfaction, ability to communicate and understand, and even non-linguistic task performance will suffer. Language technologies must therefore strive to be maximally adaptable to their users, and importantly in our highly multilingual world, this must include the ability to adapt to multilingual users. Our work has already produced a large data set (1,556 dialogs) that can help researchers understand multilingual users' language preferences during human-machine interactions, and we have made this data set publicly available, along with the computer code and documentation necessary to implement our multilingual chat bot (https://github.com/HaifaCLG/MapTask). While our corpus alone makes a valuable contribution to the fields of bilingualism, corpus linguistics, and language technology, by documenting and sharing our methods, future researchers will be able to build on our progress and develop their own multilingual chat interfaces and large, multi-useful corpora.

Finally, by taking steps to improve language generation methods for multilingual speakers and those who use English as a second language, our project not only helps improve technological accessibility for potentially marginalized populations; it also advances the state-of-the-art more generally by moving towards more adaptable, person-aware language technology that can benefit all users.

Last Modified: 02/05/2025
Modified by: Melinda Fricke