Performer PyTorch is an open-source PyTorch implementation of Performer, the linear attention-based transformer variant from Google Research that approximates softmax attention with random feature maps. Released under the MIT license, it has gathered about 1,200 stars on GitHub and was last updated in 2022.
Standard transformer attention scales quadratically with sequence length, which makes long documents, audio and genomic sequences expensive. Performer replaces exact attention with the FAVOR+ mechanism (Fast Attention Via positive Orthogonal Random features), which approximates the softmax kernel in linear time and memory. This repository, by prolific open-source re-implementer Phil Wang (lucidrains), turns the paper into a compact, hackable PyTorch package: a Performer encoder and a PerformerLM language model you can configure with depth, heads, feature counts and redraw intervals. It supports causal autoregressive modeling as well as bidirectional encoding, generalized attention kernels, and reversible layers for memory-efficient training. Like many lucidrains projects, it is widely used as a readable reference implementation for experimenting with efficient attention research.
The package mirrors the structure of the original paper closely enough to serve as study material: the attention module exposes the random feature mapping explicitly, so readers can see exactly how the softmax kernel is approximated. It has been a common building block in community experiments with efficient transformers for music, images and long-document modeling, where quadratic attention is the first bottleneck encountered.
Performer PyTorch is free and open source under the MIT license. There is no paid version; you install the package and train models on your own hardware.
Performer PyTorch is for deep learning researchers and engineers who want to experiment with efficient attention, prototype long-sequence transformers, or study a clean implementation of the Performer paper.