October 22, 2019

12:00 pm / 1:30 pm


Hodson Hall, Room 310

Jason Lee ? ?Beyond Linearization in Neural Networks? @ Hodson310
Oct 22 @ 12:00 pm ? 1:00 pm
Lunch 1:00 pm – 1:30 pm
?Beyond Linearization in Neural Networks?

Abstract: Deep Learning has had phenomenal empirical successes in many domains including computer vision, natural language processing, and speech recognition. To consolidate and boost the empirical success, we need to develop a more systematic and deeper understanding of the elusive principles of deep learning.

In this talk, I will provide analysis of several elements of deep learning including non-convex optimization, overparametrization, and generalization error. Recent theoretical work has established connections between neural networks and linearized models governeed by Neural Tangent Kernels (NTK). Such theory leads to concrete convergence and generalization results, yet the empirical performanceof neural networks are observed to exceed their linearized models.

Towards closing this gap, we investigate the training of overparametrized neural networks that are more global than the NTK regime. We show that by utilizing more of the parameter space, SGD can learn with lower sample complexity than NTK under mild distributional assumptions.

Bio: Jason Lee is an assistant professor in Electrical Engineering at Princeton University. Prior to that, he was in the Data Science and Operations department at the University of Southern California and a postdoctoral researcher at UC Berkeleyworking with Michael I. Jordan. Jason received his PhD at Stanford University advised by Trevor Hastie and Jonathan Taylor. His research interests are in the theory of machine learning, optimization, and statistics. Lately, he has worked on the foundations of deep learning, non-convex optimization algorithm, and reinforcement learning. He has received a Sloan Research Fellowship in 2019 and NIPS Best Student Paper Award for his work.