Many information systems engage with their users through the following loop of interactions: the system receives a context as input (e.g. query, user profile), responds with a context-dependent action (e.g. ranking, recommendation, ad), and then receives some explicit or implicit feedback on the quality of the action (e.g. star rating, following a search result, clicking on an ad). While ubiquitous and plentiful, log data from this interaction loop does not fit the standard mold of supervised learning, since the feedback is both biased and partial -- the system determines through its actions where it gets feedback, and even for the chosen actions it typically doesn't observe all feedback (e.g. missing clicks on relevant results in ranking).

This project addressed the question of how this logged data can nevertheless be used for evaluating and learning new systems. The potential upsides of reusing the existing log data are evident. For evaluation, the use of historic log data enables engineers to rapidly evaluate many new systems offline (e.g. new ranking functions, recommendation policies), without the weeks of delay and the potential negative impact on user experience implied by online A/B testing. For learning, it similarly enables offline reuse of existing data instead of slowly collecting new data through an online learning algorithm. This can greatly speed up the machine-learning development cycle, since model selection, feature selection, and eventual quality control can happen offline before any learned policy gets deployed to the users. Reusing existing log data is particularly important for small-scale information systems (e.g. scholarly search), where it is often the only type of potential training data that is readily available in sufficient quantity.

The project developed principled machine learning methods that enable information systems to reliably learn from logs of the partial and biased feedback they produce. As the theoretical basis for these methods, the project uncovered connections to counterfactual and causal inference, exploiting the analogy between logs and controlled experiments with actions as treatments and the current system as the assignment mechanism. In this way, the project developed new ways for answering the question of how a new system would have performed, if it had been used instead of the system that logged the data. In particular, the project developed new counterfactual estimators specifically designed for the action spaces typically encountered in information systems (e.g. rankings), new propensity models, and new counterfactual policy learning algorithms that incorporate both. To validate the real-world effectiveness of the research, the project built the Localify system (https://localify.org/), which provides local music-event recommendations and personalized playlists. 

In addition to producing new methods and changing how learning from partial-information feedback is done for many online platforms in industry, the project helped grow a research community around off-policy learning and evaluation. In particular, the PI organized the REVEAL/CONSEQUENCES workshops at RecSys 2019, 2020, 2021, 2022, and 2023, and by now off-policy learning and evaluation have become mainstream with its own conference session at RecSys 2024. Furthermore, the project offered research and career developments to a diverse group of undergraduate and graduate students.

Last Modified: 12/06/2024
Modified by: Thorsten Joachims