research
∙
09/02/2023
RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model
Text-to-image generation (TTI) refers to the usage of models that could ...
share
research
∙
08/02/2023
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
ChatGPT-like models have revolutionized various applications in artifici...
research
∙
07/19/2023
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
In the complex domain of large language models (LLMs), striking a balanc...
research
∙
03/15/2023
A Comprehensive Study on Post-Training Quantization for Large Language Models
Post-training quantization () had been recently shown as a compromising ...
research
∙
01/27/2023
Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
Improving the deployment efficiency of transformer-based language models...
research
∙
12/07/2022
DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
Recent advances on deep learning models come at the price of formidable ...
research
∙
11/17/2022
Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers
Large-scale transformer models have become the de-facto architectures fo...
research
∙
06/04/2022
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
How to efficiently serve ever-larger trained natural language models in ...
research
∙
06/04/2022
Extreme Compression for Pre-trained Transformers Made Simple and Efficient
Extreme compression, particularly ultra-low bit precision (binary/ternar...
research
∙
10/14/2021
Adaptive Differentially Private Empirical Risk Minimization
We propose an adaptive (stochastic) gradient perturbation method for dif...
research
∙
09/17/2021
AdaLoss: A computationally-efficient and provably convergent adaptive gradient method
We propose a computationally-friendly adaptive learning rate schedule, "...
research
∙
04/15/2021
Hierarchical Learning for Generation with Long Source Sequences
One of the challenges for current sequence to sequence (seq2seq) models ...
research
∙
12/05/2020
When Do Curricula Work?
Inspired by human learning, researchers have proposed ordering examples ...
research
∙
01/10/2020
Choosing the Sample with Lowest Loss makes SGD Robust
The presence of outliers can potentially significantly skew the paramete...
research
∙
11/18/2019
Implicit Regularization of Normalization Methods
Normalization methods such as batch normalization are commonly used in o...
research
∙
08/28/2019
Linear Convergence of Adaptive Stochastic Gradient Descent
We prove that the norm version of the adaptive stochastic gradient metho...
research
∙
02/19/2019
Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
Adaptive gradient methods like AdaGrad are widely used in optimizing neu...
research
∙
06/05/2018
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization
Adaptive gradient methods such as AdaGrad and its variants update the st...
research
∙
03/07/2018