research
∙
08/15/2023
A^2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
We study the task of zero-shot vision-and-language navigation (ZS-VLN), ...
share
research
∙
07/24/2023
3D-LLM: Injecting the 3D World into Large Language Models
Large language models (LLMs) and Vision-Language Models (VLMs) have been...
research
∙
07/22/2023
Learning Vision-and-Language Navigation from YouTube Videos
Vision-and-language navigation (VLN) requires an embodied agent to navig...
research
∙
07/20/2023
Reparameterized Policy Learning for Multimodal Trajectory Optimization
We investigate the challenge of parametrizing policies for reinforcement...
research
∙
07/05/2023
Building Cooperative Embodied Agents Modularly with Large Language Models
Large Language Models (LLMs) have demonstrated impressive planning abili...
research
∙
06/29/2023
An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training
We present a model that can perform multiple vision tasks and can be ada...
research
∙
06/27/2023
Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
General physical scene understanding requires more than simply localizin...
research
∙
06/07/2023
ModuleFormer: Learning Modular Large Language Models From Uncurated Data
Large Language Models (LLMs) have achieved remarkable results. But exist...
research
∙
05/04/2023
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Recent AI-assistant agents, such as ChatGPT, predominantly rely on super...
research
∙
04/27/2023
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
We propose a hybrid neural network (NN) and PDE approach for learning ge...
research
∙
04/19/2023
EC^2: Emergent Communication for Embodied Control
Embodied control requires agents to leverage multi-modal pre-training to...
research
∙
04/17/2023
Hyper-Decision Transformer for Efficient Online Policy Adaptation
Decision Transformers (DT) have demonstrated strong performances in offl...
research
∙
04/07/2023
Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
Humans, even at a very early age, can learn visual concepts and understa...
research
∙
04/06/2023
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
Humans possess a versatile mechanism for extracting structured represent...
research
∙
03/29/2023
Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
Modeling sounds emitted from physical object interactions is critical fo...
research
∙
03/27/2023
DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
In this work, we aim to learn dexterous manipulation of deformable objec...
research
∙
03/20/2023
3D Concept Learning and Reasoning from Multi-View Images
Humans are able to accurately reason in 3D by gathering multi-view obser...
research
∙
03/09/2023
PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Existing approaches to system identification (estimating the physical pa...
research
∙
03/09/2023
Planning with Large Language Models for Code Generation
Existing large language model-based code generation pipelines typically ...
research
∙
12/15/2022
Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners
Optimization in multi-task learning (MTL) is more challenging than singl...
research
∙
11/21/2022
CLAWSAT: Towards Both Robust and Accurate Code Models
We integrate contrastive learning (CL) with adversarial learning to co-o...
research
∙
10/27/2022
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Effective planning of long-horizon deformable object manipulation requir...
research
∙
10/15/2022
Revisiting the Roles of "Text" in Text Games
Text games present opportunities for natural language understanding (NLU...
research
∙
10/14/2022
Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
We address a practical yet challenging problem of training robot agents ...
research
∙
10/14/2022
Learning Active Camera for Multi-Object Navigation
Getting robots to navigate to multiple objects autonomously is essential...
research
∙
10/12/2022
M^3Video: Masked Motion Modeling for Self-Supervised Video Representation Learning
We study self-supervised video representation learning that seeks to lea...
research
∙
09/17/2022
MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning
Decentralized learning has shown great promise for cooperative multi-age...
research
∙
09/01/2022
Gait Recognition in the Wild with Multi-hop Temporal Switch
Existing studies for gait recognition are dominated by in-the-lab scenar...
research
∙
07/22/2022
Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation
This paper studies a new, practical but challenging problem, called Clas...
research
∙
07/13/2022
3D Concept Grounding on Neural Fields
In this paper, we address the challenging problem of 3D concept groundin...
research
∙
07/07/2022
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
The way an object looks and sounds provide complementary reflections of ...
research
∙
07/05/2022
Weakly Supervised Grounding for VQA in Vision-Language Transformers
Transformers for visual-language representation learning have been getti...
research
∙
06/30/2022
On-Device Training Under 256KB Memory
On-device training enables the model to adapt to new data collected from...
research
∙
06/27/2022
Prompting Decision Transformer for Few-Shot Policy Generalization
Humans can leverage prior experience and learn novel tasks from a handfu...
research
∙
06/03/2022
SNAKE: Shape-aware Neural 3D Keypoint Field
Detecting 3D keypoints from point clouds is important for shape reconstr...
research
∙
05/29/2022
EfficientViT: Enhanced Linear Attention for High-Resolution Low-Computation Visual Recognition
Vision Transformer (ViT) has achieved remarkable performance in many vis...
research
∙
05/11/2022
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
This work considers identifying parameters characterizing a physical sys...
research
∙
05/05/2022
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Differentiable physics has recently been shown as a powerful tool for so...
research
∙
05/05/2022
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
This paper studies the problem of fixing malfunctional 3D objects. While...
research
∙
05/02/2022
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
Objects' motions in nature are governed by complex interactions and thei...
research
∙
04/04/2022
Learning Neural Acoustic Fields
Our environment is filled with rich and dynamic acoustic information. Wh...
research
∙
03/31/2022
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
We consider the problem of sequential robotic manipulation of deformable...
research
∙
03/30/2022
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
We present a meta-learning framework for learning new visual concepts qu...
research
∙
03/13/2022
AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation
Training a generalizable 3D part segmentation network is quite challengi...
research
∙
12/09/2021
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse vis...
research
∙
12/01/2021
Graph Convolutional Module for Temporal Action Localization in Videos
Temporal action localization has long been researched in computer vision...
research
∙
11/01/2021
When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?
Contrastive learning (CL) can learn generalizable feature representation...
research
∙
10/28/2021
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
In this work, we propose a unified framework, called Visual Reasoning wi...
research
∙
10/28/2021
MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
Tiny deep learning on microcontroller units (MCUs) is challenging due to...
research
∙
10/17/2021