Alexander Kozachinskiy

Indexing: the Beginning and the End

Alexander Kozachinskiy, Vicente Opazo, Felipe Urrutia

We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive. In this primitive, the input consists of $n$ bits and one integer $i$ from $1$…Read more