research
∙
09/01/2023
Mechanism of feature learning in convolutional neural networks
Understanding the mechanism of how convolutional neural networks learn f...
share
research
∙
06/07/2023
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
In this paper, we first present an explanation regarding the common occu...
research
∙
06/05/2023
Aiming towards the minimizers: fast convergence of SGD for overparametrized problems
Modern machine learning paradigms, such as deep learning, occur in or cl...
research
∙
06/05/2023
On Emergence of Clean-Priority Learning in Early Stopped Neural Networks
When random label noise is added to a training dataset, the prediction e...
research
∙
02/08/2023
Cut your Losses with Squentropy
Nearly all practical neural models for classification are trained using ...
research
∙
02/06/2023
Toward Large Kernel Models
Recent studies indicate that kernel machines can often perform similarly...
research
∙
12/28/2022
Feature learning in neural networks and kernel machines that recursively learn features
Neural networks have achieved impressive results on many technological a...
research
∙
09/29/2022
Restricted Strong Convexity of Deep Learning Models with Smooth Activations
We consider the problem of optimization of deep learning models with smo...
research
∙
07/23/2022
A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
In this work we establish an algorithm and distribution independent non-...
research
∙
07/14/2022
Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
The practical success of overparameterized neural networks has motivated...
research
∙
06/30/2022
A note on Linear Bottleneck networks and their Transition to Multilinearity
Randomly initialized wide neural networks transition to linear functions...
research
∙
05/26/2022
Kernel Ridgeless Regression is Inconsistent for Low Dimensions
We show that kernel interpolation for a large class of shift-invariant k...
research
∙
05/24/2022
Quadratic models for understanding neural network dynamics
In this work, we propose using a quadratic model as a tool for understan...
research
∙
05/24/2022
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
In this paper we show that feedforward neural networks corresponding to ...
research
∙
04/29/2022
Wide and Deep Neural Networks Achieve Optimality for Classification
While neural networks are used for classification tasks across domains, ...
research
∙
03/10/2022
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Wide neural networks with linear output layer have been shown to be near...
research
∙
02/17/2022
Limitations of Neural Collapse for Understanding Generalization in Deep Learning
The recent work of Papyan, Han, Donoho (2020) presented an intriguin...
research
∙
02/14/2022
Benign Overfitting in Two-layer Convolutional Neural Networks
Modern neural networks often have great expressive power and can be trai...
research
∙
12/30/2021
Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size
Establishing a fast rate of convergence for optimization methods is cruc...
research
∙
07/31/2021
Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
Matrix completion problems arise in many applications including recommen...
research
∙
05/29/2021
Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
In the past decade the mathematical theory of machine learning has lagge...
research
∙
04/28/2021
Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
Modern machine learning systems such as deep neural networks are often h...
research
∙
10/02/2020
On the linearity of large non-linear models: when and why the tangent kernel is constant
The goal of this work is to shed light on the remarkable phenomenon of t...
research
∙
09/18/2020
Linear Convergence and Implicit Regularization of Generalized Mirror Descent with Time-Dependent Mirrors
The following questions are fundamental to understanding the properties ...
research
∙
08/03/2020
Multiple Descent: Design Your Own Generalization Curve
This paper explores the generalization loss of linear regression in vari...
research
∙
06/12/2020
Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
Modern neural architectures for classification tasks are trained using t...
research
∙
05/16/2020
Classification vs regression in overparameterized regimes: Does the loss function matter?
We compare classification and regression tasks in the overparameterized ...
research
∙
02/29/2020
Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning
The success of deep learning is due, to a great extent, to the remarkabl...
research
∙
09/26/2019
Overparameterized Neural Networks Can Implement Associative Memory
Identifying computational mechanisms for memorization and retrieval is a...
research
∙
03/18/2019
Two models of double descent for weak features
The "double descent" risk curve was recently proposed to qualitatively d...
research
∙
12/28/2018
Reconciling modern machine learning and the bias-variance trade-off
The question of generalization in machine learning---how algorithms are ...
research
∙
11/06/2018
On exponential convergence of SGD in non-convex over-parametrized learning
Large over-parametrized models learned via stochastic gradient descent (...
research
∙
11/06/2018
Kernel Machines Beat Deep Neural Networks on Mask-based Single-channel Speech Enhancement
We apply a fast kernel method for mask-based single-channel speech enhan...
research
∙
10/31/2018
MaSS: an Accelerated Stochastic Method for Over-parametrized Learning
In this paper we introduce MaSS (Momentum-added Stochastic Solver), an a...
research
∙
10/16/2018
Downsampling leads to Image Memorization in Convolutional Autoencoders
Memorization of data in deep neural networks has become a subject of sig...
research
∙
06/25/2018
Does data interpolation contradict statistical optimality?
We show that learning methods interpolating the training data can achiev...
research
∙
06/15/2018
Learning kernels that adapt to GPU
In recent years machine learning methods that nearly interpolate the dat...
research
∙
06/13/2018
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
Many modern machine learning models are trained to achieve zero or near-...
research
∙
02/28/2018
Parametrized Accelerated Methods Free of Condition Number
Analyses of accelerated (momentum-based) gradient descent usually assume...
research
∙
02/12/2018
Fast Interactive Image Retrieval using large-scale unlabeled data
An interactive image retrieval system learns which images in the databas...
research
∙
02/05/2018
To understand deep learning we need to understand kernel learning
Generalization performance of classifiers in deep learning has recently ...
research
∙
01/10/2018
Approximation beats concentration? An approximation view on inference with smooth radial kernels
Positive definite kernels and their associated Reproducing Kernel Hilber...
research
∙
12/18/2017
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
Stochastic Gradient Descent (SGD) with small mini-batch is a key compone...
research
∙
06/20/2017
Unperturbed: spectral analysis beyond Davis-Kahan
Classical matrix perturbation results, such as Weyl's theorem for eigenv...
research
∙
03/30/2017
Diving into the shallows: a computational perspective on large-scale shallow learning
In this paper we first identify a basic limitation in gradient descent-b...
research
∙
07/06/2016
Graphons, mergeons, and so on!
In this work we develop a theory of hierarchical clustering for graphs. ...
research
∙
06/21/2015
Beyond Hartigan Consistency: Merge Distortion Metric for Hierarchical Clustering
Hierarchical clustering is a popular method for analyzing data which ass...
research
∙
02/27/2015
Probabilistic Zero-shot Classification with Semantic Rankings
In this paper we propose a non-metric ranking-based representation of se...
research
∙
02/13/2015
A Pseudo-Euclidean Iteration for Optimal Recovery in Noisy ICA
Independent Component Analysis (ICA) is a popular model for blind signal...
research
∙
03/04/2014