GPT2-Chinese is an open-source project for training GPT-2 language models on Chinese text. It pairs OpenAI’s GPT-2 architecture with a BERT tokenizer suited to Chinese characters, plus training and generation code that made it easy — in the era before giant multilingual models — to build a Chinese text generator for poetry, prose, lyrics and more. The project has earned about 7,600 stars on GitHub.
When GPT-2 appeared, its tokenizer and training data were heavily English-centric, and generating coherent Chinese required rethinking the tokenization. GPT2-Chinese solves this by using BERT’s character-level Chinese tokenizer, then provides the full pipeline: preparing corpora, training models of GPT-2 scale from scratch or from pretrained weights, and generating text with the usual sampling controls. The project shared pretrained models trained on Chinese corpora such as classical and modern poetry, which is why it became a favorite for AI-poetry experiments and university coursework across the Chinese-speaking NLP community. While today’s large models dwarf GPT-2, the repository remains an unusually clear, compact teaching example of how a transformer language model is actually trained end to end — data, tokenizer, training loop and generation — in a language other than English.
GPT2-Chinese is free and open source under the MIT license. Training costs are whatever compute you run it on.
GPT2-Chinese is for students, researchers and hobbyists who want to train and understand a Chinese language model end to end, or experiment with AI-generated Chinese poetry and prose.