research
∙
05/22/2023
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, Toxicity
Pretraining is the preliminary and fundamental step in developing capabl...
share
research
∙
04/18/2023
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining
Pretrained multilingual large language models have typically used heuris...
research
∙
01/31/2023
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
We study the design decisions of publicly available instruction tuning m...
research
∙
01/30/2023
SingSong: Generating musical accompaniments from singing
We present SingSong, a system that generates instrumental music to accom...
research
∙
01/26/2023
MusicLM: Generating Music From Text
We introduce MusicLM, a model generating high-fidelity music from text d...
research
∙
12/20/2022
Character-Aware Models Improve Visual Text Rendering
Current image generation models struggle to reliably produce well-formed...
research
∙
11/17/2022
VeLO: Training Versatile Learned Optimizers by Scaling Up
While deep learning models have replaced hand-designed features across m...
research
∙
11/15/2022
Large Language Models Struggle to Learn Long-Tail Knowledge
The internet contains a wealth of knowledge – from the birthdays of hist...
research
∙
11/03/2022
Crosslingual Generalization through Multitask Finetuning
Multitask prompted finetuning (MTF) has been shown to help large languag...
research
∙
06/11/2022
Multi-instrument Music Synthesis with Spectrogram Diffusion
An ideal music synthesizer should be both interactive and expressive, ge...
research
∙
04/12/2022
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
Large pretrained Transformer language models have been shown to exhibit ...
research
∙
04/05/2022
PaLM: Scaling Language Modeling with Pathways
Large language models have been shown to achieve remarkable performance ...
research
∙
03/31/2022
Scaling Up Models and Data with and
Recent neural network-based language models have benefited greatly from ...
research
∙
05/28/2021
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Most widely-used pre-trained language models operate on sequences of tok...
research
∙
02/23/2021
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transfo...
research
∙
01/01/2021
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020. The competition...
research
∙
12/14/2020
Extracting Training Data from Large Language Models
It has become common to publish large (billion parameter) language model...
research
∙
10/22/2020
mT5: A massively multilingual pre-trained text-to-text transformer
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified ...
research
∙
04/30/2020
WT5?! Training Text-to-Text Models to Explain their Predictions
Neural networks have recently achieved human-level performance on variou...
research
∙
02/10/2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model?
It has recently been observed that neural language models trained on uns...
research
∙
01/14/2020
DDSP: Differentiable Digital Signal Processing
Most generative models of audio directly generate samples in one of two ...
research
∙
10/23/2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich tas...
research
∙
07/14/2019
The Bach Doodle: Approachable music composition with machine learning at scale
To make music composition more approachable, we designed the first AI-po...
research
∙
05/14/2019
Learning to Groove with Inverse Sequence Transformations
We explore models for translating abstract musical ideas (scores, rhythm...
research
∙
03/18/2019
Counterpoint by Convolution
Machine learning models of music typically break up the task of composit...
research
∙
02/23/2019
GANSynth: Adversarial Neural Audio Synthesis
Efficient audio synthesis is an inherently difficult machine learning ta...
research
∙
10/29/2018
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
Generating musical audio directly with neural networks is notoriously di...
research
∙
06/01/2018
Learning a Latent Space of Multitrack Measures
Discovering and exploring the underlying structure of multi-instrumental...
research
∙
03/13/2018
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
The Variational Autoencoder (VAE) has proven to be an effective model fo...
research
∙
11/15/2017
Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models
Deep generative neural networks have proven effective at both conditiona...
research
∙
10/30/2017
Onsets and Frames: Dual-Objective Piano Transcription
We consider the problem of transcribing polyphonic piano music with an e...
research
∙
04/05/2017