GPT2-ML is an open-source project that extends OpenAI’s GPT-2 language model to multiple languages, including a 1.5-billion-parameter pretrained Chinese model. With about 1,700 stars on GitHub, it pairs simplified GPT-2 training scripts with ready-to-use pretrained weights and Colab demos, with the repository last updated in 2023.
The project provides simplified GPT-2 training scripts based on Grover that support TPUs, along with a ported BERT tokenizer that is compatible with multilingual corpora. Its headline release is a 1.5B-parameter Chinese GPT-2 trained on tens of gigabytes of text over hundreds of thousands of steps, using Google Cloud TPUs provided through the TensorFlow Research Cloud program. Batteries-included Colab notebooks let you try text generation without any local setup, and the weights are distributed through GitHub releases. While newer models have since surpassed GPT-2, the repository remains a compact, readable example of how to train and release a large language model end to end.
The repository documents its training configuration and data pipeline in both English and Chinese, which made it a popular starting point for Chinese-language GPT-2 experiments and fine-tuning projects when large open Chinese models were still rare.
GPT2-ML is free and open source under the Apache-2.0 license. The pretrained models can be downloaded at no cost; you only provide the compute to fine-tune or run them. There is no paid tier, hosted API or commercial support plan attached to the project.
NLP researchers and hobbyists who want a hands-on GPT-2 codebase for languages beyond English, especially Chinese, and students studying how classic transformer language models were trained. It is also useful as a historical baseline when comparing modern open models against their predecessors.