Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained)

#mamba #s4 #ssm

OUTLINE:
0:00 - Introduction
0:45 - Transformers vs RNNs vs S4
6:10 - What are state space models?
12:30 - Selective State Space Models
17:55 - The Mamba architecture
22:20 - The SSM layer and forward propagation
31:15 - Utilizing GPU memory hierarchy
34:05 - Efficient computation via prefix sums / parallel scans
36:01 - Experimental results and comments
38:00 - A brief look at the code


Paper: https://arxiv.org/abs/2312.00752

Abstract:
Foundation models, now powering most of the …

architecture computation explained gpu introduction layer linear mamba memory modeling paper propagation space spaces state transformers