research
∙
09/18/2023
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
This paper presents a comprehensive survey of the taxonomy and evolution...
share
research
∙
07/27/2023
Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models
In this paper, we study the denoising diffusion probabilistic model (DDP...
research
∙
06/30/2023
DisCo: Disentangled Control for Referring Human Dance Generation in Real World
Generative AI has made significant strides in computer vision, particula...
research
∙
06/26/2023
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Despite the promising progress in multi-modal tasks, current large multi...
research
∙
06/07/2023
MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos
Multimodal summarization with multimodal output (MSMO) has emerged as a ...
research
∙
04/28/2023
An Empirical Study of Multimodal Model Merging
Model merging (e.g., via interpolation or task arithmetic) fuses multipl...
research
∙
04/13/2023
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Spatial control is a core capability in controllable image generation. A...
research
∙
04/12/2023
Adaptive Human Matting for Dynamic Videos
The most recent efforts in video matting have focused on eliminating tri...
research
∙
03/25/2023
Equivariant Similarity for Vision-Language Foundation Models
This study explores the concept of equivariance in vision-language found...
research
∙
03/20/2023
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a po...
research
∙
02/21/2023
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
3D photography renders a static image into a video with appealing 3D vis...
research
∙
12/21/2022
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixe...
research
∙
10/17/2022
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
This paper surveys vision-language pre-training (VLP) methods for multim...
research
∙
09/04/2022
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Masked visual modeling (MVM) has been recently proven effective for visu...
research
∙
06/15/2022
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable att...
research
∙
06/14/2022
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Unified vision-language frameworks have greatly advanced in recent years...
research
∙
05/27/2022
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transforme...
research
∙
05/03/2022
Cross-modal Representation Learning for Zero-shot Action Recognition
We present a cross-modal Transformer-based framework, which jointly enco...
research
∙
12/08/2021
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
We initiate the first empirical study on the use of MLP architectures fo...
research
∙
11/25/2021
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
The canonical approach to video captioning dictates a caption generation...
research
∙
11/24/2021
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
A great challenge in video-language (VidL) modeling lies in the disconne...
research
∙
06/08/2021
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dat...
research
∙
06/01/2021
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
With large-scale pre-training, the past two years have witnessed signifi...
research
∙
04/23/2021
Playing Lottery Tickets with Vision and Language
Large-scale transformer-based pre-training has recently revolutionized v...
research
∙
04/01/2021
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Vision-and-language pre-training has achieved impressive success in lear...
research
∙
03/16/2021
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Multimodal pre-training has propelled great advancement in vision-and-la...
research
∙
02/11/2021
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
The canonical approach to video-and-language learning (e.g., video quest...
research
∙
12/15/2020
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
Large-scale pre-trained multimodal transformers, such as ViLBERT and UNI...
research
∙
06/26/2020
Graph Optimal Transport for Cross-Domain Alignment
Cross-domain alignment between two sets of entities (e.g., objects in an...
research
∙
06/11/2020
Large-Scale Adversarial Training for Vision-and-Language Representation Learning
We present VILLA, the first known effort on large-scale adversarial trai...
research
∙
05/01/2020
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
We present HERO, a Hierarchical EncodeR for Omni-representation learning...
research
∙
10/08/2019
Meta Module Network for Compositional Visual Reasoning
There are two main lines of research on visual reasoning: neural module ...
research
∙
09/25/2019
UNITER: Learning UNiversal Image-TExt Representations
Joint image-text embedding is the bedrock for most Vision-and-Language (...
research
∙
03/29/2019
Relation-aware Graph Attention Network for Visual Question Answering
In order to answer semantically-complicated questions about an image, a ...
research
∙
02/01/2019
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
This paper presents Recurrent Dual Attention Network (ReDAN) for visual ...
research
∙
05/05/2017