Traditionally, compression schemes are evaluated using rate-distortion 
trade-offs; this project is interested in rate-accuracy trade-offs, 
where accuracy captures the effect that quantization may have on a 
specific machine learning task. There is particular interest in 
information-theoretic lower bounds and trade-offs, and in explicit 
compression for the following two questions: (1) How to compress for 
model training, when we need to use distributed communication 
constrained nodes to learn a model, fast and efficiently; and (2) How to 
compress for communication during inference. The project will derive 
bounds and algorithms for distributed compression of features coming 
from composite distributions that will be used for a machine learning 
task, such as classification. The goals of this project is to create new 
foundational theory and algorithms which will advance the state of the 
art, and build new connections between the areas of data compression and 
distributed machine learning. If successful, the project will not only 
build new theory, but also could have impact in practical applications 
of learning over communication constrained networks. The project was 
arranged in three thrusts, with Thrust 1 focused on compression for 
distributed model training, Thrust 2 exploring compression for inference 
time  and Thrust 3 that combined both training and inference time 
compression. We believe that we had significant results in the project 
towards these goals.

One measure of academic impact of the project is the number of 
publications during the duration of the project; 45 publications 
resulted from this project (that acknowledged support of this grant) in 
top tier conference and journals.  Papers acknowledging support from 
this project have garnered nearly 500 citations according to Google 
Scholar. It also supported (in part) the PhD work of 7 graduate 
students, one postdoctoral researchers and several undergraduate (REU) 
students. The four graduated PhD students have gone on to careers in 
industrial research labs. Another measure of impact is the recognition 
of the papers and PIs through this project. For example, one of papers 
supported (in part) by this project was recognized through  the 
prestigious 2021 ACM Conference on Computer and Communications Security 
(CCS) best paper award. One of the students supported by this award 
received the UCLA PhD dissertation award. It also resulted in faculty 
research awards on related topics for the PIs from Amazon and Meta. The 
research in this project also fostered a collaboration with industrial 
research labs (Google) through joint publications. The work was also 
disseminated through several invited lectures in various industrial and 
academics settings by the PIs, and through organization of workshops and 
conferences.

This project developed several fundamental ideas compression for 
distributed learning systems, which were discovered and published during 
the course of this project. The details are in the publications and were 
also summarized in annual reports of the project. Among the several 
works (including over 25 journal and journal-equivalent conference 
papers) we would like to highlight the following: (i) Gradient 
compression for decentralized networks: we proposed new algorithms that 
used error-compensation, event-triggering and sparsification to enable 
schemes whose convergence was equivalent to vanilla stochastic 
optimization with orders of magnitude reduction in communications; this 
work was published in several communities including Control (Trans. Aut. 
Control, Automatica, CDC), information theory (JSAIT, ISIT) etc. (ii) 
Compression for contextual multi-arm bandits: In this line of work we 
initated new compression methods for multi-arm bandits and also showed 
fundamental connections between contextual MAB and linear bandit 
problems. These works were published both in learning venues (NeurIPS, 
AISTATS, COLT) as well as information theory venues (JSAIT, ISIT)  (iii) 
Compression for interactive distributed inference: we developed new 
schemes and information theoretic bounds by introducing interaction in 
distributed inference, demonstrating a new way to break dependencies in 
observations (iv) Information theory and algorithms for personalized 
learning: we developed a new statistical framework for personalized 
learning and through that developed both information theoretic bounds as 
well as algorithms. These were published in learning venues (e.g., 
NeurIPS, ICLR) as well as information theory venues (ISIT). These are 
only a few of the many results from the project, which also included a 
review paper on compression for distributed learning and co-editing a 
special issue on the topic in IEEE Journal of Selected Areas in 
Communications. The project also involved out reach and training of 
undergraduate students, including several female students.

Last Modified: 01/19/2025
Modified by: Christina Fragouli