TALK TITLE: ACCELERATING DEEP LEARNING WITH HPC TECHNIQUES FOR DENSE LINEAR ALGEBRA
Deep neural networks are making groundbreaking advancements in image/speech recognition, translation, and playing Go. In each area, these neural networks now achieve super-human capabilities at very low cost. Training these deep neural networks is an optimization problem, which has conventionally been solved by very simple stochastic gradient descent methods. Recent work has shown that using higher order methods such as natural gradient descent can give orders of magnitude better convergence. Also, these higher order methods are less sensitive to the increase in batch size, which means they are suitable for executing on massively parallel computers. However, the extra calculation required for the dense Fisher information matrix increases the time per iteration. We present various fast approximation techniques for the factorization of the Fisher information matrix, which could potentially accelerate current deep learning by orders of magnitude on massively parallel supercomputers.
Rio Yokota is currently an Associate Professor at the Tokyo Institute of Technology in Japan. He was a research scientist at the Extreme Computing Research Center at the King Abdullah University of Science and Technology (KAUST) for three and a half years before moving to Japan. He continues his collaboration with Professor David Keyes and Dr. Hatem Ltaief on hierarchical low-rank approximation methods as part of the Intel Parallel Computing Center project. He was the organizer of the first HPC Saudi conference in 2012. He was the recipient of the Gordon Bell prize in 2009.