The project focuses on advancing UAV capabilities to intelligently explore complex environments using recent advances in deep learning and miniature hardware. We aim to create a dynamic multi-UAV system that enhances self-awareness and generalizability by actively integrating and processing visual data across space and time. The system optimizes view plans and path planning for energy efficiency, with applications spanning environmental monitoring, search and rescue, autonomous vehicles, smart health, and manufacturing.

To develop advanced attention models capable of integrating video data from multiple UAVs comprehensively, we curated extensive attention and reasoning datasets. Our novel attention model considered both correct and incorrect attention patterns to refine machine learning. Additionally, we conducted the first research on predicting scanpaths during diverse tasks, achieving superior accuracy in capturing human attention's spatio-temporal dynamics compared to existing methods. We introduced novel datasets and models to facilitate the integration of multi-modal inputs, enhancing decision-making transparency and performance across varied applications. Furthermore, our development of a trustworthiness predictor enhances prediction accuracy by distinguishing distributions of positive and negative examples, thereby optimizing decision-making under complex real-world conditions.

To enhance the adaptability and performance of attention models in dynamic environments, we integrated external knowledge directly into our models. We introduced the first explicit and explainable visual reasoning method that integrated external knowledge through dynamic scene graphs and functional programs, augmented by a novel reinforcement learning method. We also defined new attention mechanisms tailored for embodied settings and developed methods that tightly integrated perception with action for embodied navigation. Experimental results demonstrated substantial improvements in navigation across diverse, previously unseen environments. 

We have developed cutting-edge view and path planning algorithms tailored for UAVs to enhance the efficiency and quality of image capture. Our research focused on optimizing view planning in complex scenarios such as disaster response and urban environments, ensuring high-quality 3D geometry and texture representation while minimizing path length deviations. By converting quality requirements into mathematical constraints, we introduced adaptive viewing planes that adapt naturally to scene geometries, significantly reducing reconstruction errors compared to traditional methods. Addressing scalability challenges, we innovatively transformed the coverage problem into a cone-based variant of the Traveling Salesman Problem (cone-TSPN), achieving polynomial runtime and constant memory usage for large-scale city coverage, a significant improvement over previous techniques. Additionally, we developed algorithms that compute visual coverage trajectories optimizing total length while maintaining specified detection probabilities. Finally, we introduced a novel route-finding approach integrating perception and travel costs, employing an entropy-based viewing score to generate diameter-bounded viewing neighborhoods. 

Furthermore, this project also involves developing methods to enhance online learning in dynamic environments. We measured agreement between new and prior knowledge, implementing Direction Concentration Learning (DCL) to optimize updates in deep neural networks. Additionally, our gradient-based few-shot learning methods enabled efficient knowledge transfer from large-scale datasets to domains with limited samples. We proposed a learning approach leveraging privileged information from multiple modalities during training, enhancing testing performance without necessitating one-to-one modal relationships. Finally, our Gradient Adjustment Learning technique optimized gradients by incorporating knowledge from previous iterations, demonstrating significant improvements across diverse applications.

The project has shaped artificial intelligence and robotics by enhancing autonomous systems' capabilities to navigate and adapt in complex environments. It has also established frameworks for transparent decision-making and improved generalizability across diverse applications, focusing on attention, generalization, and adaptive action planning. Public accessibility of datasets, analyses, and code fostered transparency and community engagement, reflected in 26 publications at conferences like CVPR, ECCV, ICCV, ICRA, and NeurIPS. The project involved seven PhD students in computer science and robotics, fostering their expertise and skills in these fields.

Last Modified: 06/28/2024
Modified by: Qi Zhao