This project studied the intent estimation, planning and control, and perception problems for an autonomous vehicle (AV) to interact with human-driven vehicles (HV) in a safe and socially adept manner. The research team ("we" thereafter) formulated the AV-HV interaction problem as an incomplete-information differential game, explored learning methods to solve incomplete-information differential game, and developed semantically driven and robust sensing techniques for vehicle sensing.

In an incomplete-information differential game setup, the AV and HV estimate the intent of each other simultaneously. We developed a novel intent inference scheme, called empathetic intent inference, which explicitly considers the mutual intent inference between the AV and HV. Simulation studies showed that the new intent inference algorithm allowed the AV to dynamically negotiate with the HV like another human driver, which improved the safety and efficiency of future transportation. We further studied the utility of the proposed empathetic intent inference algorithm, and results showed that empathy helped agents choose policies that led to higher social values when agents had common beliefs inconsistent with the ground truth. Considering the computational intensity for empathetic intent inference, we developed an intermittent intent inference algorithm triggered by reinforcement learning. Using an uncontrolled intersection case, we showed that the proposed new intent inference mechanism led to safe interactions between an AV and HV while significantly reducing the computation load.  To study corner cases such as unprotected left turns and uncontrolled intersections where traffic accidents often happen due to misunderstanding between drivers, we focused on extending the game-theoretic framework for robotics applications in two important directions.

First, we studied two-player zero-sum differential games with both incomplete information and state constraints. Our study was the first to prove value existence for such games. As a byproduct, the proof led to the subdynamic programming principles that governed the behavioral strategies of informed and uninformed players. In comparison with studies on imperfect-information games, our theoretical development revealed unique structures of values and strategies for games where the incomplete information is on player payoff types.

Second, we studied the open challenge in approximating values for complete-information differential games with Markov payoffs and state constraints. For games either without constraints or without continuous payoffs (e.g., reach-avoid), the curse of dimensionality can be alleviated through Monte Carlo methods such as physics-informed neural networks (PINN). When both payoffs and state constraints are present, the value of the game becomes discontinuous in which case standard PINN fails to converge. To this end, we developed a Lagrangian PINN that forced explicit learning of discontinuous co-states along with the value. We showed that our method led to significant improvement in value generalization and safety performance compared with existing PINNs for Lipschitz-continuous partial differential equations (PDEs).

For situation awareness sensing research thrust, in our project's first stage, we leveraged traffic monitoring cameras as integral tools for traffic management and autonomous driving systems. We developed CAROM, or "CARs On the Map," a framework for vehicle localization and traffic scene reconstruction. This framework processed traffic monitoring videos to generate anonymous data structures detailing vehicle type, 3D shape, position, and velocity, facilitating accurate scene reconstruction and replay. In the second stage, we aimed to enhance sensing models without relying on detectors. Introducing ViTCAP, a pure vision transformer-based model utilizing grid representations, we incorporated a Concept Token Network (CTN) to predict semantic concepts and improve end-to-end sensing performance. CTN, based on a vision transformer, effectively predicted concept tokens, enriching the sensing process with valuable semantic information. In the last stage, we addressed visual feature selection crucial for Visual Odometry (VO) and Simultaneous Localization and Mapping (SLAM). Our novel self-supervised VO method, VOCAL (Visual Odometry via ContrAstive Learning), intelligently selected and normalized features using contrastive learning based on associated 3D camera motions. VOCAL eliminated the need for intricate feature extraction and preprocessing steps, demonstrating robust and generalizable performance in experiments on the KITTI challenge dataset compared to baseline methods. VOCAL presents a promising alternative pathway for achieving accurate VO through self-supervised contrastive learning methodologies.

This project has generated positive societal impacts by improving the safety and intelligence of AVs when operating around human-driven vehicles. The research team have disseminated their research findings in various workshops and presentations in control, robotics, and computer vision conferences, and they have shared their vision on future AVs and transportation in outreach events (e.g., Phoenix Mobile Festival) to local communities. Results on vehicle perception from this project led to a new start-up company by co-PI Yang. All the publications resulting from this project have been made available in the NSF Public Access Repository.

Seven Ph.D. students and one M.S. student have been involved and at least partially supported by this project, and one Ph.D. graduate is now working in academia. Eight undergraduate students have participated in this project, including five as Research Experience for Undergraduates (REU) trainees.

Last Modified: 03/12/2024
Modified by: Wenlong Zhang