Sparse Transformers Explained, Part 1
How Sparse Transformers cut the quadratic cost of attention by factorising it into strided and fixed patterns, and what that buys you in context length.
Tag
Tag
How Sparse Transformers cut the quadratic cost of attention by factorising it into strided and fixed patterns, and what that buys you in context length.