Fast Transformers is an open-source PyTorch library implementing fast, memory-efficient transformer attention mechanisms. Developed at the Idiap Research Institute, it has 1,784 stars on GitHub and was last updated in 2023. The library collects the institute’s research on making transformers scale to long sequences.
Standard self-attention scales quadratically with sequence length, which makes long documents, audio, and time series painfully expensive. Fast Transformers implements a family of alternatives (including linear attention and other efficient attention variants from the research literature) behind one consistent API. A builder interface lets you assemble a transformer encoder by choosing the attention type, layer counts, and dimensions, so you can swap a softmax-attention model for a linear-attention one and directly compare speed and memory on sequences of thousands of tokens. The implementations focus on the attention bottleneck specifically: by reducing attention from quadratic to linear complexity in sequence length, models can process far longer contexts on the same hardware. Each attention module can also be lifted out and used standalone inside an existing model architecture.
Fast Transformers is free and open source, distributed through GitHub and PyPI. There is no paid tier; you run it on your own hardware.
The library is for deep learning researchers and engineers working with long-sequence transformers who need efficient attention implementations that are ready to drop into PyTorch experiments. For many users it doubles as a readable reference implementation when studying how modern efficient attention methods actually work under the hood.