research
∙
08/28/2023
MEMORY-VQ: Compression for Tractable Internet-Scale Memory
Retrieval augmentation is a powerful but expensive method to make langua...
share
research
∙
06/17/2023
GLIMMER: generalized late-interaction memory reranker
Memory-augmentation is a powerful approach for efficiently incorporating...
research
∙
05/22/2023
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Multi-query attention (MQA), which only uses a single key-value head, dr...
research
∙
05/18/2023
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences
We present our work on developing a multilingual, efficient text-to-text...
research
∙
05/09/2023
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset
Webpages have been a rich resource for language and vision-language task...
research
∙
05/05/2023
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Webpages have been a rich, scalable resource for vision-language and lan...
research
∙
04/11/2023
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
We propose Conditional Adapter (CoDA), a parameter-efficient transfer le...
research
∙
03/17/2023
CoLT5: Faster Long-Range Transformers with Conditional Computation
Many natural language processing tasks benefit from long inputs, but pro...
research
∙
01/25/2023
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute
Retrieval-augmented language models such as Fusion-in-Decoder are powerf...
research
∙
12/15/2022
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model...
research
∙
12/09/2022
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Training large, deep neural networks to convergence can be prohibitively...
research
∙
09/29/2022
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
A common recent approach to semantic parsing augments sequence-to-sequen...
research
∙
05/24/2022
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT
We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with ...
research
∙
03/28/2022
LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models
Machine learning models such as Transformers or LSTMs struggle with task...
research
∙
03/16/2022
FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction
Sequence modeling has demonstrated state-of-the-art performance on natur...
research
∙
12/15/2021
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Recent work has shown that either (1) increasing the input length or (2)...
research
∙
10/08/2021
Iterative Decoding for Compositional Generalization in Transformers
Deep learning models do well at generalizing to in-distribution data but...
research
∙
09/02/2021
ShopTalk: A System for Conversational Faceted Search
We present ShopTalk, a multi-turn conversational faceted search system f...
research
∙
08/09/2021
Making Transformers Solve Compositional Tasks
Several studies have reported the inability of Transformer models to gen...
research
∙
06/19/2021
Improving Compositional Generalization in Classification Tasks via Structure Annotations
Compositional generalization is the ability to generalize systematically...
research
∙
05/10/2021
ReadTwice: Reading Very Large Documents with Memories
Knowledge-intensive tasks such as question answering often require assim...
research
∙
05/09/2021
FNet: Mixing Tokens with Fourier Transforms
We show that Transformer encoder architectures can be massively sped up,...
research
∙
12/21/2020
RealFormer: Transformer Likes Residual Attention
Transformer is the backbone of modern NLP models. In this paper, we prop...
research
∙
07/28/2020
Big Bird: Transformers for Longer Sequences
Transformers-based models, such as BERT, have been one of the most succe...
research
∙
04/17/2020