Longformer is the official open-source release from the Allen Institute for AI of the long-document transformer, with 2,200+ stars on GitHub — a pretrained model family that reads documents thousands of tokens long instead of truncating them.
Standard transformers like BERT cap out at 512 tokens because full self-attention scales quadratically, which shuts them out of legal contracts, scientific papers, books and clinical notes. Longformer, released by AI2 in 2020, replaces full attention with a pattern that combines local sliding-window attention with a few global-attention tokens, scaling linearly with sequence length. The release includes pretrained encoder models and LongformerEncoderDecoder (LED), the sequence-to-sequence variant for summarization and generation over long inputs — with gradient checkpointing and fp16, LED handles inputs up to 16K tokens on a 48GB GPU. The repository provides the models, example scripts such as long-document summarization, and integration notes for the Hugging Face transformers ecosystem, where Longformer became a standard baseline for long-context research. The code was last updated in 2023 and remains a canonical reference implementation.
Longformer is free and open source under the Apache License 2.0, including the pretrained model checkpoints. You run everything on your own hardware; there is no hosted service or paid tier.
NLP researchers and engineers working with long documents — summarization, long-form QA, legal and scientific text — who need a proven, inspectable long-context transformer baseline.