research
∙
07/17/2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Scaling Transformers to longer sequence lengths has been a major problem...
share
research
∙
05/09/2023
StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the r...
research
∙
03/16/2023
Effectively Modeling Time Series with Simple Discrete State Spaces
Time series modeling is a well-established problem, which often requires...
research
∙
02/21/2023
Hyena Hierarchy: Towards Larger Convolutional Language Models
Recent advances in deep learning have relied heavily on the use of large...
research
∙
02/13/2023
Simple Hardware-Efficient Long Convolutions for Sequence Modeling
State space models (SSMs) have high performance on long sequence modelin...
research
∙
12/28/2022
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
State space models (SSMs) have demonstrated state-of-the-art sequence mo...
research
∙
11/26/2022
Transform Once: Efficient Operator Learning in Frequency Domain
Spectral analysis provides one of the most effective paradigms for infor...
research
∙
10/12/2022
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
Visual data such as images and videos are typically modeled as discretiz...
research
∙
09/28/2022
ButterflyFlow: Building Invertible Layers with Butterfly Matrices
Normalizing flows model complex probability distributions using maps obt...
research
∙
06/02/2022
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Communication compression is a crucial technique for modern distributed ...
research
∙
06/02/2022
Decentralized Training of Foundation Models in Heterogeneous Environments
Training foundation models, such as GPT-3 and PaLM, can be extremely exp...
research
∙
05/27/2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Transformers are slow and memory-hungry on long sequences, since the tim...
research
∙
11/30/2021
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
Overparameterized neural networks generalize well but are expensive to t...
research
∙
10/28/2021
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
Recent advances in efficient Transformers have exploited either the spar...
research
∙
10/26/2021
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
Recurrent neural networks (RNNs), temporal convolutions, and neural diff...
research
∙
04/20/2021
Knowledge Distillation as Semiparametric Inference
A popular approach to model compression is to train an inexpensive stude...
research
∙
03/29/2021
Rethinking Neural Operations for Diverse Tasks
An important goal of neural architecture search (NAS) is to automate-awa...
research
∙
12/29/2020
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
Modern neural network architectures use structured linear transformation...
research
∙
08/17/2020
HiPPO: Recurrent Memory with Optimal Polynomial Projections
A central problem in learning from sequential data is representing cumul...
research
∙
11/26/2019
Approximating the Permanent by Sampling from Adaptive Partitions
Computing the permanent of a non-negative matrix is a core problem with ...
research
∙
09/03/2019
On the Downstream Performance of Compressed Word Embeddings
Compressing word embeddings is important for deploying NLP models in mem...
research
∙
03/14/2019
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations
Fast linear transforms are ubiquitous in machine learning, including the...
research
∙
10/31/2018
Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximation
We investigate how to train kernel approximation methods that generalize...
research
∙
10/04/2018
Learning Compressed Transforms with Low Displacement Rank
The low displacement rank (LDR) framework for structured matrices repres...
research
∙
03/16/2018