Award Abstract # 1808159
Collaborative Research: Multimodal Sensing and Analytics at Scale: Algorithms and Applications

NSF Org: ECCS
Division of Electrical, Communications and Cyber Systems
Recipient: OREGON STATE UNIVERSITY
Initial Amendment Date: August 21, 2018
Latest Amendment Date: April 27, 2020
Award Number: 1808159
Award Instrument: Standard Grant
Program Manager: Huaiyu Dai
hdai@nsf.gov
 (703)292-4568
ECCS
 Division of Electrical, Communications and Cyber Systems
ENG
 Directorate for Engineering
Start Date: September 1, 2018
End Date: August 31, 2022 (Estimated)
Total Intended Award Amount: $249,991.00
Total Awarded Amount to Date: $265,991.00
Funds Obligated to Date: FY 2018 = $249,991.00
FY 2020 = $16,000.00
History of Investigator:
  • Xiao Fu (Principal Investigator)
    xiao.fu@oregonstate.edu
Recipient Sponsored Research Office: Oregon State University
1500 SW JEFFERSON AVE
CORVALLIS
OR  US  97331-8655
(541)737-4933
Sponsor Congressional District: 04
Primary Place of Performance: Oregon State University
OR  US  97331-8507
Primary Place of Performance
Congressional District:
04
Unique Entity Identifier (UEI): MZ4DYXE1SL98
Parent UEI:
NSF Program(s): CCSS-Comms Circuits & Sens Sys
Primary Program Source: 01001819DB NSF RESEARCH & RELATED ACTIVIT
01002021DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 153E, 9102, 9251
Program Element Code(s): 756400
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.041

ABSTRACT

Finding highly correlated latent factors in multimodal signals and data: Scalable algorithms and applications in sensing, imaging, and language processing

Abstract: Multimodal signals and data arise naturally in many walks of science and engineering, and our digital society presents ever-increasing opportunities to collect and extract useful information from such data. For example, brain magnetic resonance imaging and electro-encephalography are two modes of sensing brain activity that can offer different "views" of the same set of patients (entities). Co-occurrence frequencies of a given set of words in different languages is another example. Crime, poverty, welfare, income, tax, school, unemployment, and other types of social data offer different views of a given set of municipalities. Integrating multiple views to extract meaningful common information is of great interest, and finds a vast amount of timely applications -- in brain imaging, machine translation, landscape change detection in remote sensing, and social science research, to name a few. However, existing multiview analytics tools -- notably (generalized) canonical correlation analysis [(G)CCA] -- are struggling to keep pace with the size of today's datasets, and the problem is only getting worse. Furthermore, the complex structure and dynamic nature of some of the underlying phenomena are not accounted for in classical GCCA. This project will provide much needed scalable and flexible computational tools for GCCA-based multimodal sensing and analytics, thereby benefiting a large variety of scientific and engineering applications. It will produce a framework allowing for plug-and-play incorporation of application-specific prior information, and distributed implementation. Beyond linear and batch GCCA, nonlinear GCCA and streaming GCCA will be considered. These are appealing and timely for many applications, but associated computational tools are sorely missing.

In terms of theory and methods, many key aspects of GCCA (such as convergence properties, distributed implementation, and streaming variants) are still poorly understood. The research will provide a set of high-performance computational tools that are backed by advanced optimization theory and rigorous convergence guarantees. The research will evolve along the following synergistic thrusts: 1) scalable and stochastic GCCA algorithms; 2) distributed, streaming and nonlinear GCCA algorithms; and 3) validation, using a series of timely and important applications in remote sensing, brain imaging, natural language processing, and sensor array processing. Devising scalable, flexible, streaming, and nonlinear GCCA algorithms is very well-motivated for modern sensing and analytics problems which involve rapidly increasing amounts of data with unknown underlying dynamics. Using GCCA for large-scale dynamic and complex data also poses very challenging and exciting modeling and optimization problems.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Huang, Lingyi and Deng, Chunhua and Ibrahim, Shahana and Fu, Xiao and Yuan, Bo "VLSI Hardware Architecture of Stochastic Low-rank Tensor Decomposition" Asilomar 2021 , 2021 https://doi.org/10.1109/IEEECONF53345.2021.9723182 Citation Details
Lyu, Qi and Fu, Xiao "Finite-Sample Analysis of Deep CCA-Based Unsupervised Post-Nonlinear Multimodal Learning" IEEE Transactions on Neural Networks and Learning Systems , 2022 https://doi.org/10.1109/TNNLS.2022.3160407 Citation Details
Pu, Wenqiang and Ibrahim, Shahana and Fu, Xiao and Hong, Mingyi "Stochastic Mirror Descent for Low-Rank Tensor Decomposition Under Non-Euclidean Losses" IEEE Transactions on Signal Processing , v.70 , 2022 https://doi.org/10.1109/TSP.2022.3163896 Citation Details
Shrestha, Sagar and Fu, Xiao "Communication-Efficient Distributed MAX-VAR Generalized CCA via Error Feedback-Assisted Quantization" IEEE ICASSP 2022 , 2022 https://doi.org/10.1109/ICASSP43922.2022.9746607 Citation Details

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

Multimodal signals and data are ubiquitously present in the real world. A modality can be regarded as a "view" of a certain entity or event (e.g., picture and audio of a cat are two modalities/views of the cat). It is widely believed that common information from multimodalities represents the essential part of the data entity/event. Effectively and efficiently learning or extracting such common information from real world signals/data has been a long-existing aspiration of the signal processing community at large. Multimodal sensing and analysis is concerned with this task, which finds a vast amount of timely applications -- in brain imaging, machine translation, remote sensing, and social science research, to name a few.  This project developed a series of algorithmic and analytical frameworks to solve challenging problems arising in multimodal sensing and analysis. First, a series of efficient, lightweight, flexible, and privacy-preserving algorithms were developed to address the scalability challenge---which had become a key barrier for processing and analyzing the massive amount of multimodal data in modern days. Second, this project developed an in-depth understanding of a number of multimodal sensing and analysis paradigms, which substantially enriched the analytical toolbox and solidified the theoretical foundations of the pertinent domains. Third, this project developed specialized and principled methodologies for timely multimodal sensing problems, e.g., hyperspectral imaging/remote sensing, wireless communications, brain signal processing, and language data analysis. 

The outcomes of this project already showed positive impacts on other disciplines. The computational principles developed in this project were used in various domains in science and engineering, e.g., medical imaging. The theoretical results were leveraged to analyze long-lingering challenges in machine learning (e.g., to establish finite-sample identifiability of independent component analysis). This project involved 4 REU students, among whom 3 were from historically underrepresented groups in computing. A Ph.D. student was supported and graduated under this project. These showed positive contributions to building a diverse and competitive workforce in engineering. In addition, this project enriched educational resources in multiple ways.  It generated a tutorial article published in IEEE Signal Processing Magazine. The outcomes were integrated into two graduate courses at Oregon State University.


 

 


Last Modified: 12/14/2022
Modified by: Xiao Fu

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page