research
∙
10/19/2022
A Unified View of Masked Image Modeling
Masked image modeling has demonstrated great potential to eliminate the ...
share
research
∙
08/22/2022
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is eme...
research
∙
08/12/2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Masked image modeling (MIM) has demonstrated impressive results in self-...
research
∙
06/02/2022
VL-BEiT: Generative Vision-Language Pretraining
We introduce a vision-language foundation model called VL-BEiT, which is...
research
∙
06/01/2022
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
As more and more pre-trained language models adopt on-cloud deployment, ...
research
∙
02/07/2022
Corrupted Image Modeling for Self-Supervised Visual Pre-Training
We introduce Corrupted Image Modeling (CIM) for self-supervised visual p...
research
∙
11/03/2021
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
We present a unified Vision-Language pretrained Model (VLMo) that jointl...
research
∙
10/26/2021
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved signi...
research
∙
06/25/2021
Learning to Sample Replacements for ELECTRA Pre-Training
ELECTRA pretrains a discriminator to detect replaced tokens, where the r...
research
∙
06/15/2021
BEiT: BERT Pre-Training of Image Transformers
We introduce a self-supervised vision representation model BEiT, which s...
research
∙
06/07/2021
Attention Temperature Matters in Abstractive Summarization Distillation
Recent progress of abstractive text summarization largely relies on larg...
research
∙
12/31/2020
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2...
research
∙
02/28/2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding a...
research
∙
02/25/2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its va...
research
∙
09/12/2018
Neural Melody Composition from Lyrics
In this paper, we study a novel task that learns to compose music from n...
research
∙
04/06/2017