MEGABYTE-pytorch is a PyTorch implementation of MEGABYTE, the multiscale transformer architecture from the paper “Predicting Million-byte Sequences with Multiscale Transformers”. It models raw bytes directly, without a tokenizer, which opens the door to very long contexts across text, audio and other byte streams. The project is released under the MIT license, has about 657 stars on GitHub, and was last updated in December 2024.
The MEGABYTE design splits a byte sequence into patches. A large global transformer reasons over the patch-level stream, while smaller local transformers predict the bytes inside each patch, so the model can in principle handle sequences of a million bytes. This repository by Phil Wang (lucidrains) follows the paper and then generalizes it further: you can stack more than the paper’s two stages, with multiple local models at different scales. Configuration is explicit and code-first, setting vocabulary size, per-stage dimensions, sequence lengths, depths and attention heads, with flash attention supported. Installation is a single pip command, and the README shows a complete training and sampling example in a few lines.
Like other lucidrains repositories, the code favors clarity over framework magic, which makes it a popular reference implementation to read alongside the paper itself. Researchers often use it as a baseline, or as a starting point for testing their own multiscale and long-context ideas, before scaling experiments up to larger runs.
MEGABYTE-pytorch is free and open source under the MIT license. There is no hosted version; you train and run models on your own hardware.
MEGABYTE-pytorch is for deep learning researchers experimenting with long-context modeling, tokenizer-free architectures and byte-level sequence prediction across different data modalities.