We developed Ula! - an integrated Deep Neural Network (DNN) acceleration framework with advanced learning capability. Ula! consists of three vertically integrated components at the levels of algorithm, hardware, and application as: 1) Reinforcement/Supervised based enhanced unsupervised learning; 2) Novel DNN computing paradigm; 3) Technology applications and realizations.

Generative Adversarial Networks (GANs) have recently drawn tremendous attention as a powerful tool to improve the performance of unsupervised or semi-supervised DNN models. While GANs deliver state-of-the-art performance on these AI tasks, it comes at the cost of high computational complexity. We proposed ReGAN - a novel ReRAM-based Process-In-Memory (PIM) accelerator that can efficiently reduce off-chip memory accesses. Two techniques, namely, Spatial Parallelism and Computation Sharing are particularly proposed to further enhance training efficiency of GANs.

Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. To alleviate the computational cost of RNN, we proposed to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. In the training process, removing a component of Intrinsic Sparse Structures (ISS) in the LSTMs simultaneously decreases the sizes of all basic structures by one and thereby always maintain the dimension consistency.

Process-in-memory (PIM) architecture such as Hybrid Memory Cube (HMC) has been used to improve the data locality for efficient DNN executions. However, it is still hard to efficiently deploy largescale matrix computation in DNN on HMC because of its coarse-grained packet protocol. We proposed NeuralHMC, the first HMCbased accelerator tailored for efficient DNN executions.

Existing nonvolatile memory-based machine learning accelerators could not support the computational needs required by GAN training. Specifically, the generator utilizes a new operator, called transposed convolution, which introduces significant resource underutilization when executed on conventional neural network accelerators as it inserts massive zeros in its input before a convolution operation. We proposed ZARA - A Novel Zero-free Dataflow Accelerator for Generative Adversarial Networks in 3D ReRAM.

Stochastic Gradient Descent (SGD) is a popular training method of DNNs because of its efficiency. However, large-batch SGD tends to converge to sharp minima in the DNNs. We propose the SmoothOut framework to smooth out sharp minima in DNNs and thereby improve generalization of the DNNs. In particular, SmoothOut perturbs multiple copies of the DNN by noise injection and averages these copies. Our experimental results showed that SmoothOut improves generalization in both small-batch and large-batch training on the top of state-of-the-art solutions.

Previous works on Convolutional Neural Network (CNN) acceleration utilize low-rank approximation of the original convolution layers to reduce computation cost. However, these methods are very difficult to conduct upon sparse models, which limits execution speedup since redundancies within the CNN model are not fully exploited. We argue that kernel granularity decomposition can be conducted with low-rank assumption while exploiting the redundancy within the remaining compact coefficients. Based on this observation, we propose PENNI, a CNN model compression framework that is able to achieve model compactness and hardware efficiency simultaneously.

Depth is a key component of DNNs. However, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. Our experiments showed that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets.

In modern GPUs, on-chip memory capacity keeps increasing to support thousands of chip-resident threads. The on-chip memory capacity of GPUs, however, is highly constrained by the large memory cell area and high static power consumption of conventional SRAM implementation. We propose to utilize the emerging multi-level cell (MLC) spin-transfer torque RAM (STT-RAM) technology to implement register files and shared memory in GPUs.

Federated learning (FL) is a popular distributed learning framework that trains a global model through iterative communications between a central server and edge devices. We empirically show that under extremely strong poisoning attacks, the existing defensive methods fail to guarantee the robustness of FL. More importantly, we observed that as long as the global model is polluted, the impact of attacks on the global model will remain in subsequent rounds even if there are no subsequent attacks. We proposed a client-based defense, named White Blood Cell for Federated Learning (FL-WBC), which can mitigate model poisoning attacks that have already polluted the global model.

This 5-year research project supported 7 graduate students and publications of 20 conference papers and 6 journal papers in total.

Last Modified: 11/23/2022
Modified by: Yiran Chen