research
∙
07/27/2023
The Marginal Value of Momentum for Small Learning Rate SGD
Momentum is known to accelerate the convergence of gradient descent in s...
share
research
∙
07/03/2023
Trainable Transformer in Transformer
Recent works attribute the capability of in-context learning (ICL) in la...
research
∙
05/27/2023
Fine-Tuning Language Models with Just Forward Passes
Fine-tuning language models (LMs) has yielded success on diverse downstr...
research
∙
10/11/2022
A Kernel-Based View of Language Model Fine-Tuning
It has become standard to solve NLP tasks by fine-tuning pre-trained lan...
research
∙
05/20/2022
On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
Approximating Stochastic Gradient Descent (SGD) as a Stochastic Differen...
research
∙
02/24/2021
On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
It is generally recognized that finite learning rate (LR), in contrast t...
research
∙
10/07/2020