Reformer PyTorch is a clean, widely used PyTorch implementation of the Reformer — “the efficient Transformer” from Google’s 2020 paper — by prolific open-source implementer lucidrains. With around 2,200 stars on GitHub, it remains the go-to reference for experimenting with locality-sensitive hashing attention and reversible layers.
Standard transformer attention scales quadratically with sequence length. The Reformer attacks that cost with LSH attention, which hashes similar queries and keys into buckets so attention is computed only within buckets, and with reversible residual layers that avoid storing activations. This implementation packages those ideas into composable PyTorch modules you can train on long sequences on modest hardware.
Beyond the model itself, the repository is a study aid: the code isolates the paper’s key components — LSH attention, shared query-key projections, reversible residual blocks, and chunked feed-forward computation — so each can be read, tested, and reused independently. It belongs to lucidrains’ large family of faithful paper reimplementations, which many researchers use as the starting point when reproducing or extending published architectures.
The package can be installed from PyPI and dropped into existing training loops; examples in the repository show it trained on enwik8 and other long-sequence corpora. Because attention cost grows slowly with sequence length, the Reformer design is often used to teach why naive attention is expensive and how hashing-based approximations trade a little accuracy for large memory savings — lessons that remain relevant to modern long-context architectures.
Reformer PyTorch is free and open source under the MIT license.
ML researchers and students studying efficient transformer architectures, and engineers prototyping long-context models in PyTorch.