research
∙
06/01/2023
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Large language models are commonly trained on a mixture of filtered web ...
share
research
∙
10/27/2022
What Language Model to Train if You Have One Million GPU Hours?
The crystallization of modeling methods around the Transformer architect...
research
∙
10/26/2022
Scaling Laws Beyond Backpropagation
Alternatives to backpropagation have long been studied to better underst...
research
∙
04/12/2022
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
Large pretrained Transformer language models have been shown to exhibit ...
research
∙
10/16/2021
PAGnol: An Extra-Large French Generative Model
Access to large pre-trained models of varied architectures, in many diff...
research
∙
09/24/2021
Is the Number of Trainable Parameters All That Actually Matters?
Recent work has identified simple empirical scaling laws for language mo...
research
∙
07/25/2021
LightOn Optical Processing Unit: Scaling-up AI and HPC with a Non von Neumann co-processor
We introduce LightOn's Optical Processing Unit (OPU), the first photonic...
research
∙
06/07/2021
Photonic Differential Privacy with Direct Feedback Alignment
Optical Processing Units (OPUs) – low-power photonic chips dedicated to ...
research
∙
01/06/2021
Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
We propose a new defense mechanism against adversarial attacks inspired ...
research
∙
12/11/2020
Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
The scaling hypothesis motivates the expansion of models past trillions ...
research
∙
06/23/2020
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
Despite being the workhorse of deep learning, the backpropagation algori...
research
∙
06/02/2020
Light-in-the-loop: using a photonics co-processor for scalable training of neural networks
As neural networks grow larger and more complex and data-hungry, trainin...
research
∙
06/11/2019