research
∙
05/21/2023
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in impr...
share
research
∙
01/27/2023
Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation
Language models have steadily increased in size over the past few years....
research
∙
10/12/2022
Foundation Transformers
A big convergence of model architectures across language, vision, speech...
research
∙
04/20/2022
On the Representation Collapse of Sparse Mixture of Experts
Sparse mixture of experts provides larger model capacity while requiring...
research
∙
04/13/2022
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding l...
research
∙
04/07/2022
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
We present a new framework AMOS that pretrains text encoders with an Adv...
research
∙
06/30/2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual languag...
research
∙
06/04/2021
Language Scaling for Universal Suggested Replies Model
We consider the problem of scaling automated suggested replies for Outlo...
research
∙
02/16/2021
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
We present COCO-LM, a new self-supervised learning framework that pretra...
research
∙
06/05/2018
Querying Complex Networks in Vector Space
Learning vector embeddings of complex networks is a powerful approach us...
research
∙
09/07/2017