Many settings in everyday life require making decisions by combining the subjective preferences of individuals in a group, such as where to go to eat, where to go on vacation, whom to hire, which ideas to fund, or what route to take. In many domains, these subjective preferences are combined with moral values, ethical principles, or business constraints that are applicable to the decision scenario and are often prioritized over the individual preferences of the agents making the decision. The potential conflict of moral values with subjective preferences are keenly felt both when AI systems recommend products to us and when we use AI enabled systems to make group decisions. This research seeks to make AI more accountable by providing mechanisms to bound the decisions that AI systems can make, ensuring that the outcomes of the group decision making process aligns with human values. To achieve the goal of building ethically-bounded, AI-enabled group decision making systems, this project takes inspiration from humans, who often constrain their decisions and actions according to a number of exogenous priorities coming from moral, ethical, or business values. This research project has taken concrete steps to address the current lack of principled, formal approaches for embedding ethics into AI agents and AI enabled group decision support systems by advancing the state of the art in the safety and robustness of AI agents which.

This project developed a number of innovative algorithmic techniques to address these issues, leading to scientific publications, code, and datasets that accomplish the primary goals of the project and lay the groundwork for future development of AI systems that are responsive to the diverse needs of decision makers and software systems that enable those decisions. Specifically this project has accomplished the following concrete results.

Modeling and Evaluating Ethical Principles We have extended principles from social choice theory and fair division to include more fine grained models of individual preferences in their decision making. We proposed and evaluated new models of consensus decision making, taking ideas from liquid democracy and integrating them into a formal model of group consensus. In addition we developed reinforcement learning based frameworks to disentangle various objectives in environments modeled as Markov decision processes. These algorithms and tools are available in open source libraries including a number of new datasets hosted at www.preflib.org.

Learning Ethical Principles From Data We developed novel machine-learning frameworks to learn individual ethical principles, aggregate them into an overall group preference, and put them to work in Markov Decision Processes, a general model of decision making under uncertainty. This resulted in novel algorithms to learn from trajectories in these environments, and model the underlying constraints formally. For instance, when navigating a simulated driving / grid environment, we are able to separate travel preferences from rules like “don’t travel through certain color squares.” We used this framework along with human experiments on Amazon Mechanical Turk to collect and test these algorithms with real world data. All these code and algorithms have been released in open source libraries.

Embedding Ethical Principles into Group Decision Support Systems Combining results from the first two areas our algorithms are able to take data from human decision makers, disentangle the (often) conflicting preferences and constraints, and then deploy them in real environments. In addition we proposed and evaluated a new framework, called Social Mechanism Design, that is able to more flexibly define both goals and constraints in group decision making tasks.

In terms of intellectual merit, this project led to 18 academic publications, more than 10 research talks, and more than 10 public engagement opportunities where the work was shared with experts and non-experts. In terms of broader impacts, this project funded two PhD students at Tulane University (both advanced to candidacy 2025, one female), one masters in computer science (graduates Spring 2025), and one undergraduate student (female, graduated Spring 2025).

Last Modified: 06/16/2025
Modified by: Nicholas Mattei