This project studied how to facilitate effective and efficient collaborative perception on top of a set of independent and multi-modal data generating agents. The project also studied how and where to best deploy UAVs to augment existing information content to achieve the highest possible collaborative perception across the spatio-temporal window of interest. The team worked on both the accuracy of the data representation and retrieval, and the scalability to large collections of data, which may be static or streaming.

A key innovation is the development of methods to jointly model text, image, video, spatiotemporal and graph data, which allows effectively searching multi-modal data sources, such as social media data. The project also created methods to query collections of documents using LLMs with minimum incurred computational cost, and by considering the user’s interaction history with the query system. Methods to scale the processing of big data were also developed, with a focus on publish-subscribe systems. The project studied how to perform robot planning to facilitate active intelligence collection. We further developed methods for sensor-based autonomous and human-in-the-loop mobile robot navigation to collect multi-modal information for scene understanding.  In autonomous navigation, we introduced both data-driven and classical perceptual planning methods.  In contrast, human-in-the-loop navigation leveraged newly developed vision-language foundation models to enable fluent interaction between the robot and a human user.  The resulting algorithms were validated both in simulation and via physical experiment using a range of mobile (wheeled and aerial) robots.

Seven undergraduate students were supported by this project. These students were involved in the research process, and were trained on machine learning and data science principles. Five PhD students graduated. The findings were presented in scientific forums in the form of publications or tutorials. Optimizing the management of big data may cut the storage costs and the energy consumption in data centers and  allow users to manage more data with the existing cloud-based hardware infrastructure. The research achievements of this project  may increase the utility of social and video surveillance data, specifically for tasks that require situational awareness. In the area of disaster response, responders may have a more integrated and holistic view of the situation, so they can better allocate their resources.

Last Modified: 09/10/2025
Modified by: Evangelos Christidis