Sparse Transformers Explained, Part 1
How Sparse Transformers cut the quadratic cost of attention by factorising it into strided and fixed patterns, and what that buys you in context length.
Blog
Explainers and notes from building ML systems. Some pieces live on Medium and open there.