research
∙
02/03/2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
We propose a new class of linear Transformers called FourierLearner-Tran...
share
research
∙
02/02/2023
Efficient Graph Field Integrators Meet Point Clouds
We present two new classes of algorithms for efficient field integration...
research
∙
02/01/2023
FAVOR#: Sharp Attention Kernel Approximations via New Classes of Positive Random Features
The problem of efficient approximation of a linear operator induced by t...
research
∙
01/31/2023
Simplex Random Features
We present Simplex Random Features (SimRFs), a new random feature (RF) m...
research
∙
05/30/2022
Chefs' Random Tables: Non-Trigonometric Random Features
We introduce chefs' random tables (CRTs), a new class of non-trigonometr...
research
∙
11/25/2021
PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Can we train a single transformer model capable of processing multiple m...
research
∙
10/08/2021
Hybrid Random Features
We propose a new class of random feature methods for linearizing softmax...
research
∙
06/07/2021
On the Expressive Power of Self-Attention Matrices
Transformer networks are able to capture patterns in data coming from ma...
research
∙
06/04/2021
Debiasing a First-order Heuristic for Approximate Bi-level Optimization
Approximate bi-level optimization (ABLO) consists of (outer-level) optim...
research
∙
02/08/2021
Unlocking Pixels for Reinforcement Learning via Implicit Attention
There has recently been significant interest in training reinforcement l...
research
∙
12/21/2020
Sub-Linear Memory: How to Make Performers SLiM
The Transformer architecture has revolutionized deep learning on sequent...
research
∙
09/30/2020
Rethinking Attention with Performers
We introduce Performers, Transformer architectures which can estimate re...
research
∙
06/19/2020
An Ode to an ODE
We present a new paradigm for Neural ODE algorithms, calledODEtoODE, whe...
research
∙
06/05/2020
UFO-BLO: Unbiased First-Order Bilevel Optimization
Bilevel optimization (BLO) is a popular approach with many applications ...
research
∙
06/05/2020
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
Transformer models have achieved state-of-the-art results across a diver...
research
∙
04/18/2020
CWY Parametrization for Scalable Learning of Orthogonal and Stiefel Matrices
In this paper we propose a new approach for optimization over orthogonal...
research
∙
03/30/2020
Stochastic Flows and Geometric Optimization on the Orthogonal Group
We present a new class of stochastic, geometrically-driven optimization ...
research
∙
10/22/2019
Tractable Minor-free Generalization of Planar Zero-field Ising Models
We present a new family of zero-field Ising models over N binary variabl...
research
∙
06/14/2019
A New Family of Tractable Ising Models
We present a new family of zero-field Ising models over N binary variabl...
research
∙
12/22/2018