Overview: This CPS project focused on enhancing the reliability and robustness of perception systems by employing formal languages to specify and evaluate their quality. By developing new logics and computational frameworks, the research aimed to improve perception accuracy in applications such as autonomous driving and robotic systems. Throughout the project, significant progress was made in formalizing and applying spatio-temporal logics to real-world scenarios.

Intellectual Merit: The research introduced Spatio-Temporal Perception Logic (STPL) as an advanced method for evaluating and monitoring perception systems. This effort resulted in several major contributions, including the development of STPL as an extension of Timed Quality Temporal Logic (TQTL) with spatial operators, allowing for a more precise assessment of perception quality. Algorithmic advancements were made with the design of efficient offline and online monitoring algorithms for STPL, enabling real-time data stream analysis. The project demonstrated the practical application of these methods in both simulated and real-world driving environments, leading to improvements in object detection under adverse conditions. Another key achievement was the introduction of Latent Space Energy-based Neural ODEs (ODE-LEBM), a novel continuous-time sequence modeling technique that integrates an expressive energy-based prior to enhance perception models, improving both interpolation and extrapolation capabilities. The outcomes of this research were disseminated through high-impact journal publications, including papers presented at IJRR, WACV, EMNLP, and TMLR, which highlighted advancements in perception quality evaluation, object detection, and neural modeling.

Broader Impacts: The research significantly contributed to the advancement of autonomous systems by improving perception robustness, thereby enhancing safety in dynamic environments. In the educational sphere, the project supported multiple PhD students, two of whom successfully completed their doctorates and secured influential roles in industry and academia. Principal Investigator Yezhou Yang actively engaged in educational outreach by conducting courses at Arizona State University and delivering public lectures, including an invited talk on "AI and the Joy of Living" at the Mirabella invited talk series. Additionally, a high school intern was involved in research efforts, focusing on the analysis of perception metrics for AI-driven applications. The project also contributed to open-source development through the release of the PerceMon Python toolbox, which enables researchers and developers to apply STPL-based monitoring techniques in their work. Collaboration with Toyota further enriched the research, as the company provided robotic platforms for testing and validation.

Key Findings and Achievements: A fundamental contribution of this project was the formalization of STPL, which provided a mathematically rigorous approach to evaluating perception systems. A trust-aware control framework was developed, integrating trust-based monitoring into self-driving systems. The research led to improvements in object detection through the development of IA-YOLO, an enhanced object detection model that increases accuracy in challenging environments. The introduction of latent space ODE models significantly advanced continuous-time perception modeling by offering a novel energy-based neural ODE framework. These advancements were successfully implemented and tested in both simulated environments, such as CARLA, and real-world robotic systems, demonstrating their practical effectiveness.

Future Directions: The research conducted in this project establishes a foundation for further advancements in the field. Future work will focus on extending STPL to multi-modal perception, combining vision and LiDAR to improve environmental understanding. Additional efforts will be directed toward increasing AI interpretability through formal logics that explain and verify deep learning-based perception models. The project also paves the way for enhancing perception in dynamic environments by applying STPL techniques to complex multi-agent systems beyond autonomous driving. These contributions will continue to influence AI-driven perception, with applications spanning autonomous navigation, robotics, and intelligent transportation systems.

Last Modified: 03/23/2025
Modified by: Yezhou Yang