research
∙
07/06/2023
When Does Confidence-Based Cascade Deferral Suffice?
Cascades are a classical strategy to enable inference cost to vary adapt...
share
research
∙
06/06/2023
On the Role of Attention in Prompt-tuning
Prompt-tuning is an emerging strategy to adapt large language models (LL...
research
∙
02/03/2023
ResMem: Learn what you can and memorize the rest
The impressive generalization performance of modern neural networks is a...
research
∙
01/28/2023
Supervision Complexity and its Role in Knowledge Distillation
Despite the popularity and efficacy of knowledge distillation, there is ...
research
∙
01/27/2023
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Large neural models (such as Transformers) achieve state-of-the-art perf...
research
∙
11/09/2022
Large Language Models with Controllable Working Memory
Large language models (LLMs) have led to a series of breakthroughs in na...
research
∙
10/12/2022
Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers
This paper studies the curious phenomenon for machine learning models wi...
research
∙
10/06/2022
Generalization Properties of Retrieval-based Models
Many modern high-performing machine learning models such as GPT-3 primar...
research
∙
10/05/2022
A Fourier Approach to Mixture Learning
We revisit the problem of learning mixtures of spherical Gaussians. Give...
research
∙
04/27/2022
ELM: Embedding and Logit Margins for Long-Tail Learning
Long-tail learning is the problem of learning under skewed label distrib...
research
∙
01/28/2022
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
In classical federated learning, the clients contribute to the overall t...
research
∙
10/19/2021
When in Doubt, Summon the Titans: Efficient Inference with Large Models
Scaling neural networks to "large" sizes, with billions of parameters, h...
research
∙
05/12/2021
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Negative sampling schemes enable efficient training given a large number...
research
∙
02/13/2021
Distilling Double Descent
Distillation is the technique of training a "student" model based on exa...
research
∙
02/05/2021
On the Reproducibility of Neural Network Predictions
Standard training techniques for neural networks involve multiple source...
research
∙
12/01/2020
Modifying Memories in Transformer Models
Large Transformer models have achieved impressive performance in many na...
research
∙
06/08/2020
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
Transformer networks use pairwise attention to compute contextual embedd...
research
∙
05/21/2020
Why distillation helps: a statistical perspective
Knowledge distillation is a technique for improving the performance of a...
research
∙
04/23/2020
Doubly-stochastic mining for heterogeneous retrieval
Modern retrieval problems are characterised by training sets with potent...
research
∙
04/21/2020
Federated Learning with Only Positive Labels
We consider learning a multi-class classification model in the federated...
research
∙
04/11/2020
Robust Large-Margin Learning in Hyperbolic Space
Recently, there has been a surge of interest in representation learning ...
research
∙
02/20/2020
Reliable Distributed Clustering with Redundant Data Assignment
In this paper, we present distributed generalized clustering algorithms ...
research
∙
02/17/2020
Low-Rank Bottleneck in Multi-head Attention Models
Attention based Transformer architecture has enabled significant advance...
research
∙
01/27/2020
Achieving Multi-Port Memory Performance on Single-Port Memory with Coding Techniques
Many performance critical systems today must rely on performance enhance...
research
∙
12/20/2019
Are Transformers universal approximators of sequence-to-sequence functions?
Despite the widespread adoption of Transformer models for NLP tasks, the...
research
∙
07/24/2019
Sampled Softmax with Random Fourier Features
The computational cost of training with softmax cross entropy loss grows...
research
∙
07/18/2018
The Generalized Lasso for Sub-gaussian Measurements with Dithered Quantization
In the problem of structured signal recovery from high-dimensional linea...
research
∙
05/22/2018
Robust Gradient Descent via Moment Encoding with LDPC Codes
This paper considers the problem of implementing large-scale gradient de...
research
∙
03/12/2018
Representation Learning and Recovery in the ReLU Model
Rectified linear units, or ReLUs, have become the preferred activation f...
research
∙
12/11/2017
The PhaseLift for Non-quadratic Gaussian Measurements
We study the problem of recovering a structured signal x_0 from high-dim...
research
∙
09/24/2017