research
∙
06/02/2023
MLP-Mixer as a Wide and Sparse MLP
Multi-layer perceptron (MLP) is a fundamental component of deep learning...
share
research
∙
12/09/2022
Attention in a family of Boltzmann machines emerging from modern Hopfield networks
Hopfield networks and Boltzmann machines (BMs) are fundamental energy-ba...
research
∙
10/06/2022
Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias
Gradient regularization (GR) is a method that penalizes the gradient nor...
research
∙
02/10/2022
Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel
A biological neural network in the cortex forms a neural field. Neurons ...
research
∙
12/03/2021
Learning Curves for Sequential Training of Neural Networks: Self-Knowledge Transfer and Forgetting
Sequential training from task to task is becoming one of the major objec...
research
∙
10/29/2020
Self-paced Data Augmentation for Training Neural Networks
Data augmentation is widely used for machine learning; however, an effec...
research
∙
10/02/2020
Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks
Natural Gradient Descent (NGD) helps to accelerate the convergence of gr...
research
∙
06/14/2020
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
The Fisher information matrix (FIM) is fundamental for understanding the...
research
∙
10/14/2019
Pathological spectra of the Fisher information metric and its variants in deep neural networks
The Fisher information matrix (FIM) plays an essential role in statistic...
research
∙
06/07/2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
Normalization methods play an important role in enhancing the performanc...
research
∙
08/22/2018
Fisher Information and Natural Gradient Learning of Random Deep Networks
A deep neural network is a hierarchical nonlinear model transforming inp...
research
∙
08/22/2018
Statistical Neurodynamics of Deep Networks: Geometry of Signal Spaces
Statistical neurodynamics studies macroscopic behaviors of randomly conn...
research
∙
06/04/2018
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
This study analyzes the Fisher information matrix (FIM) by applying mean...
research
∙
12/12/2017