Intellectual Merit:

Accuracy of traditional numerical algorithms for solving a linear system or computing eigenvalues of a matrix is limited by the condition number of the matrix. In this project, we have developed an accurate preconditioning algorithm together with an error analysis to solve an ill-conditioned linear algebra problem with an accuracy independent of the condition number, provided there is an accurate inversion algorithm for the preconditioner. The algorithm allows us to solve some ill-conditioned linear systems to the accuracy of machine precision regardless of the condition number. The accurate solution of linear systems are used to compute eigenvalues to high relative accuracy. Numerical tests have confirmed the stability of the method developed. As part of the analysis, we have introduced a new stability concept, called inverse-equivalent accuracy, that is fundamental in precisely characterizing the stability gained.

We have studied applications of preconditioning to neural network optimization by developing a novel preconditioning method, called batch normalization preconditioning, to accelerate convergence of neural network training. Our algorithm is rooted in a very effective neural network training method called Batch Normalization (BN), but is derived from a well-established theoretical framework of preconditioning. We have developed a theory to show the batch normalization preconditioning reduces the condition number of Hessian matrix. Our theory is also used to derive a BN method for Convolutional Neural Networks. Extensive experiments have been carried out that demonstrate that our method is highly competitive to BN and, in particular, works with small mini-batches whereas BN does not. We have also developed a new optimization algorithm with a convergence theory that combines the classical nonlinear conjugate gradient with the momentum methods, called adaptive momentum method. Our method has the convergence characteristic of the nonlinear conjugate gradient without the need for line search at each step, which is important in large scale optimization problems such as neural networks. We have applied the adaptive momentum method to convolutional neural network training for the image classification problems, which shows significant improvements over the classical momentum method as well as the Nesterov momentum method.

We have studied stable constructions of orthogonal or unitary matrices in recurrent neural networks (RNNs). We have developed two new methods to construct a recurrent matrix that is orthogonal/unitary to mitigate the exploding/vanishing gradient problem. One method is for real orthogonal matrix and the other is for complex unitary matrix. Both of our methods have outperformed the standard RNNs and their variants on a variety of benchmark testing problems. We have further extended the methods to allow construction of a recurrent matrix with a 2x2 block upper triangular matrix with the diagonal blocks consisting of an orthogonal matrix and a matrix with the spectral radius less than 1. This remedies some disadvantages of an orthogonal recurrent matrix and has led to much improved performance. Our methods have produced state-of-the-art results in several benchmark datasets.

We have proposed a new optimization algorithm for generative adversarial networks (GANs) by introducing an adaptive weighting scheme to calibrate simultaneous optimizations of two losses in GAN's discriminator. The methods have produced some very competitive results, including state-of-the-arts in inception scores for several benchmark datasets.

We have derived conditions on a convolution kernel that can generate or preserve symmetric structures in features. We have applied our method to the sequential recommendation problem and to the RNA secondary structure inference problem. Our experimental results demonstrate that our approaches can improve inference results while using equal or smaller numbers of machine parameters.

Broader Impact:

We have applied the accurate preconconditioning method to compute a few smallest eigenvalues of a finite difference discretization of the biharmonic operator with an accuracy in the order of machine precision. This was not possible without using higher precision by existing methods. The biharmonic eigenvalue problems arise in many applications.

We have collaborated with colleagues in mathematical biology and bioinformatics to study two applications of RNNs. First, we have successfully derived an LSTM method (a variation of RNN) for the state inference of RNA secondary structure that significantly outperforms traditional methods in classification accuracy. Second, we have derived an LSTM implementation for automatic detections of epileptic seizures through analysis of the electroencephalography (EEG) signals and our method has better performance compared with the current state-of-the-art methods and is more robust across patients. Finally, four computer codes derived from this project have been freely distributed at the open source platform GitHub.

Last Modified: 10/01/2020
Modified by: Qiang Ye