research
∙
05/18/2023
LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretra...
share
research
∙
04/19/2023
A Theory on Adam Instability in Large-Scale Machine Learning
We present a theory for the previously unexplained divergent behavior no...
research
∙
04/04/2023
Effective Theory of Transformers at Initialization
We perform an effective-theory analysis of forward-backward signal propa...
research
∙
01/10/2023
Scaling Laws for Generative Mixed-Modal Language Models
Generative language models define distributions over sequences of tokens...
research
∙
05/02/2022
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands...
research
∙
12/13/2019
Long-Term Planning and Situational Awareness in OpenAI Five
Understanding how knowledge about the world is represented within model-...
research
∙
12/13/2019
Neural Network Surgery with Sets
The cost to train machine learning models has been increasing exponentia...
research
∙
12/13/2019