GPT-NeoX is EleutherAI’s framework for training large autoregressive language models at scale on GPUs. It is the codebase behind GPT-NeoX-20B and the Pythia model suite, and it remains a reference implementation for anyone who wants to pretrain a serious transformer without starting from zero. The project has about 7,500 stars on GitHub, is released under the Apache 2.0 license, and was still updated in 2026.
Training a multi-billion-parameter model is as much a systems problem as a modeling one: the work has to be sharded across many GPUs with the right mix of data, tensor and pipeline parallelism. GPT-NeoX builds on Megatron-LM and DeepSpeed to provide exactly that, with configuration files controlling architecture, parallelism strategy, optimizer and data pipeline. It supports rotary embeddings, parallel attention and MLP blocks, Flash Attention and efficient checkpointing, and it has been validated at the tens-of-billions-of-parameters scale in public training runs. Because EleutherAI publishes both the framework and the models trained with it, researchers can trace a result from final weights all the way back to the exact training configuration.
Free. GPT-NeoX is open source under the Apache 2.0 license; the real cost of using it is the GPU cluster you train on.
Research labs and ML engineers who pretrain or extensively fine-tune large language models and need a battle-tested, fully open training stack they can inspect and modify.