Award Abstract # 2144764
CAREER: Machine-centered Cyberinfrastructure for Panoramic Video Analytics in Science and Engineering Monitoring

NSF Org: OAC
Office of Advanced Cyberinfrastructure (OAC)
Recipient: GEORGE MASON UNIVERSITY
Initial Amendment Date: January 11, 2022
Latest Amendment Date: July 9, 2025
Award Number: 2144764
Award Instrument: Continuing Grant
Program Manager: Juan Li
jjli@nsf.gov
 (703)292-2625
OAC
 Office of Advanced Cyberinfrastructure (OAC)
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: June 15, 2022
End Date: May 31, 2027 (Estimated)
Total Intended Award Amount: $500,000.00
Total Awarded Amount to Date: $395,476.00
Funds Obligated to Date: FY 2022 = $293,426.00
FY 2025 = $102,050.00
History of Investigator:
  • Zhisheng Yan (Principal Investigator)
    zyan4@gmu.edu
Recipient Sponsored Research Office: George Mason University
4400 UNIVERSITY DR
FAIRFAX
VA  US  22030-4422
(703)993-2295
Sponsor Congressional District: 11
Primary Place of Performance: George Mason University
4400 UNIVERSITY DR
FAIRFAX
VA  US  22030-4422
Primary Place of Performance
Congressional District:
11
Unique Entity Identifier (UEI): EADLFP7Z72E5
Parent UEI: H4NRWLFCDF43
NSF Program(s): CAREER: FACULTY EARLY CAR DEV
Primary Program Source: 01002627DB NSF RESEARCH & RELATED ACTIVIT
01002526DB NSF RESEARCH & RELATED ACTIVIT

01002223DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 1045, 079Z
Program Element Code(s): 104500
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

Video analytics plays a pivotal role in science and engineering monitoring. Monitoring videos captured by remote cameras are typically live streamed to servers for analysis because of the limited computational capabilities of camera devices. From wildlife tracking and coastline event detection to airport suspect recognition and victim search in disaster response, such automated video analytics systems have been deployed widely to assist human operators. The recent advancement of 360 degree cameras enables a new paradigm of panoramic video analytics that can cover the 360 degree surroundings of a monitoring site and can address the errors in and missing analysis abilities of traditional 2D video analytics. However, realizing this vision requires live streaming massive panoramic video data to servers for online analytics, which cannot be supported by the current cyberinfrastructure (CI). The mismatch between the 360 degree video bit rate and available network bandwidth can cause lagging or failed analysis, diminishing the benefits of panoramic video analytics. This project will create a framework of video compression, streaming, and recovery for achieving the vision of panoramic video analytics in science and engineering monitoring. The new CI will allow scientists and engineers to conduct online panoramic video analytics and enable innovative applications that are otherwise unattainable. The research outcomes will support the development of a remote learning tool for imaging analytics, course curriculum and undergraduate research in media computing, and educational videos for public outreach.

This project investigates a machine centered video computing framework in order to enable online panoramic video analytics. Unlike traditional human centered video frameworks where pixels are processed to preserve extensive aesthetic details for human viewing, the proposed CI compresses, streams, and recovers feature points for machine analytics. Because of this fundamental change, the proposed framework is able to greatly outperform legacy video CIs and support panoramic video analytics. To this end, a deep learning based 360 degree video codec will be built to distill the spatiotemporal characteristics of video features and optimize both compression ratio and analytics accuracy. Second, an adaptive 360 degree video bitrate streaming system will be designed to ensure continuous delivery of full 360 degree video frames by prioritizing regions of interest preferred by machines. Third, a 360 degree video recovery scheme will be developed to restore noisy and delayed video data while considering the time constraints in the online analytics models. Finally, interdisciplinary collaboration will be done with application area scientists and engineers to carry out the project plans for evaluation and validation of the panoramic video framework on real world problems.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Chen, Bo and Guo, Hongpeng and Wu, Mingyuan and Yang, Zhe and Yan, Zhisheng and Nahrstedt, Klara "ImmerScope: Multi-view Video Aggregation at Edge towards Immersive Content Services" , 2024 https://doi.org/10.1145/3666025.3699324 Citation Details
Chen, Bo and Wu, Mingyuan and Guo, Hongpeng and Yan, Zhisheng and Nahrstedt, Klara "Vesper: Learning to Manage Uncertainty in Video Streaming" , 2024 https://doi.org/10.1145/3625468.3647621 Citation Details
Chen, Bo and Yan, Zhisheng and Han, Bo and Nahrstedt, Klara "NeRFHub: A Context-Aware NeRF Serving Framework for Mobile Immersive Applications" , 2024 https://doi.org/10.1145/3643832.3661879 Citation Details
Chen, Bo and Yan, Zhisheng and Nahrstedt, Klara "Context-aware image compression optimization for visual analytics offloading" MMSys '22: Proceedings of the 13th ACM Multimedia Systems Conference , 2022 https://doi.org/10.1145/3524273.3528178 Citation Details
Chen, Bo and Yan, Zhisheng and Nahrstedt, Klara "Context-aware Optimization for Bandwidth-Efficient Image Analytics Offloading" ACM Transactions on Multimedia Computing, Communications, and Applications , 2023 https://doi.org/10.1145/3638768 Citation Details
Chen, Bo and Yan, Zhisheng and Zhang, Yinjie and Yang, Zhe and Nahrstedt Klara "LiFteR: Unleash Learned Codecs in Video Streaming with Loose Frame Referencing" , 2024 Citation Details
Li, Jiaxi and Liao, Jingwei and Chen, Bo and Nguyen, Anh and Tiwari, Aditi and Zhou, Qian and Yan, Zhisheng and Nahrstedt, Klara "ST-360: SpatialTemporal Filtering-Based Low-Latency 360-Degree Video Analytics Framework" ACM Transactions on Multimedia Computing, Communications, and Applications , 2024 https://doi.org/10.1145/3694685 Citation Details
Murad, Taslim and Nguyen, Anh and Yan, Zhisheng "DAO: Dynamic Adaptive Offloading for Video Analytics" MM '22: Proceedings of the 30th ACM International Conference on Multimedia , 2022 https://doi.org/10.1145/3503161.3548249 Citation Details
Nguyen, Anh and Yan, Zhisheng "Enhancing 360 Video Streaming through Salient Content in Head-Mounted Displays" Sensors , v.23 , 2023 https://doi.org/10.3390/s23084016 Citation Details

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page