research
∙
09/04/2023
Les Houches Lectures on Deep Learning at Large Infinite Width
These lectures, presented at the 2022 Les Houches Summer School on Stati...
share
research
∙
07/12/2023
Quantitative CLTs in Deep Neural Networks
We study the distribution of a fully connected neural network with rando...
research
∙
06/20/2023
Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations
This article derives and validates three principles for initialization a...
research
∙
05/13/2023
Depth Dependence of μP Learning Rates in ReLU MLPs
In this short note we consider random fully connected ReLU networks of w...
research
∙
12/29/2022
Bayesian Interpolation with Deep Linear Networks
This article concerns Bayesian inference using deep linear networks with...
research
∙
12/14/2022
Maximal Initial Learning Rates in Deep ReLU Networks
Training a neural network requires choosing a suitable learning rate, in...
research
∙
05/11/2022
Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis
Advanced deep neural networks (DNNs), designed by either human or AutoML...
research
∙
04/03/2022
Correlation Functions in Random Fully Connected Neural Networks at Finite Width
This article considers fully connected neural networks with Gaussian ran...
research
∙
09/27/2021
Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions
We prove a precise geometric description of all one layer ReLU networks ...
research
∙
07/04/2021
Random Neural Networks in the Infinite Width Limit as Gaussian Processes
This article gives a new proof that fully connected neural networks with...
research
∙
06/18/2021
The Principles of Deep Learning Theory
This book develops an effective theory approach to understanding deep ne...
research
∙
02/21/2021
Deep ReLU Networks Preserve Expected Length
Assessing the complexity of functions computed by a neural network helps...
research
∙
12/28/2020
Neural Network Approximation
Neural Networks (NNs) are the method of choice for building learning alg...
research
∙
10/21/2020
Data augmentation as stochastic optimization
We present a theoretical framework recasting data augmentation as stocha...
research
∙
09/13/2019
Finite Depth and Width Corrections to the Neural Tangent Kernel
We prove the precise scaling, at finite depth and width, for the mean an...
research
∙
06/03/2019
Deep ReLU Networks Have Surprisingly Few Activation Patterns
The success of deep networks has been attributed in part to their expres...
research
∙
01/25/2019
Complexity of Linear Regions in Deep Networks
It is well-known that the expressivity of a neural network depends on it...
research
∙
12/14/2018
Products of Many Large Random Matrices and Gradients in Deep Neural Networks
We study products of random matrices in the regime where the number of t...
research
∙
03/05/2018
How to Start Training: The Effect of Initialization and Architecture
We investigate the effects of initialization and architecture on the sta...
research
∙
01/11/2018
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
We give a rigorous analysis of the statistical behavior of gradients in ...
research
∙
10/31/2017
Approximating Continuous Functions by ReLU Nets of Minimal Width
This article concerns the expressive power of depth in deep feed-forward...
research
∙
08/09/2017