GPT2 (ConnorJL)

This GPT-2 implementation by ConnorJL is a clean, hackable reimplementation of OpenAI’s GPT-2 training code with one notable addition: support for training on TPUs as well as GPUs. The open-source project has around 1,400 stars on GitHub and is a long-standing favorite for learning how GPT-2 actually trains.

Overview

OpenAI’s original GPT-2 release focused on inference and sampling; training the model from scratch, especially at the larger sizes, required community work. This repository implements the full training loop, letting you pretrain GPT-2-style models on your own text corpora and then generate from them with the companion sampling code. TPU support via TensorFlow’s distribution machinery makes it practical to train on Google Cloud TPUs and Colab-class hardware, which is how many students and independent researchers first reproduced GPT-2 results. The codebase stays close to the original architecture — the same transformer blocks, tokenization, and checkpoint conventions — which makes it useful as readable reference code for understanding pre-LLM-era training in detail. The repository has been stable since 2022 and is feature-complete for its purpose: an educational and experimental training implementation rather than a modern fine-tuning framework. It is MIT-licensed.

Key Features

• Full GPT-2 training implementation, not just inference
• TPU support for affordable large-scale pretraining experiments
• Pretrain on your own text corpora, then sample from your checkpoints
• Faithful to OpenAI’s original architecture and checkpoint conventions
• Readable TensorFlow code suited to learning transformer training internals
• MIT-licensed and widely forked as a starting point

💳 Pricing

Free. The implementation is fully open source under the MIT license; you pay only for any compute you train on.

Who Is It For

Students, researchers, and tinkerers who want to train GPT-2 from scratch — including on TPUs — and understand exactly how the training loop works.