DeepAI AI Chat
Log In Sign Up

SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation

by   Malyaban Bal, et al.
Penn State University

Large language Models (LLMs), though growing exceedingly powerful, comprises of orders of magnitude less neurons and synapses than the human brain. However, it requires significantly more power/energy to operate. In this work, we propose a novel bio-inspired spiking language model (LM) which aims to reduce the computational cost of conventional LMs by drawing motivation from the synaptic information flow in the brain. In this paper, we demonstrate a framework that leverages the average spiking rate of neurons at equilibrium to train a neuromorphic spiking LM using implicit differentiation technique, thereby overcoming the non-differentiability problem of spiking neural network (SNN) based algorithms without using any type of surrogate gradient. The steady-state convergence of the spiking neurons also allows us to design a spiking attention mechanism, which is critical in developing a scalable spiking LM. Moreover, the convergence of average spiking rate of neurons at equilibrium is utilized to develop a novel ANN-SNN knowledge distillation based technique wherein we use a pre-trained BERT model as "teacher" to train our "student" spiking architecture. While the primary architecture proposed in this paper is motivated by BERT, the technique can be potentially extended to different kinds of LLMs. Our work is the first one to demonstrate the performance of an operational spiking LM architecture on multiple different tasks in the GLUE benchmark.


page 1

page 2

page 3

page 4


Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State

Spiking neural networks (SNNs) are brain-inspired models that enable ene...

Training Deep Spiking Neural Networks

Computation using brain-inspired spiking neural networks (SNNs) with neu...

Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation

Spiking neural networks (SNNs) are well known as the brain-inspired mode...

SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks

As the size of large language models continue to scale, so does the comp...

Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning

Spiking neural networks (SNNs) with leaky integrate and fire (LIF) neuro...

Community detection with spiking neural networks for neuromorphic hardware

We present results related to the performance of an algorithm for commun...

Deep Spiking-UNet for Image Processing

U-Net, known for its simple yet efficient architecture, is widely utiliz...