MiniMind

MiniMind is an open-source educational project with more than 64,000 stars on GitHub that trains a 64-million-parameter large language model completely from scratch in about two hours.

Overview

Most LLM tutorials fine-tune someone else’s model; MiniMind rebuilds the whole journey at miniature scale. Starting from raw text, it walks through tokenization, pretraining, supervised fine-tuning, and alignment techniques such as RLHF-style preference optimization and knowledge distillation, plus mixture-of-experts variants — all in clean, readable PyTorch code sized so a single modest GPU can finish a run in roughly two hours for a couple of dollars of compute. The resulting tiny model can hold simple conversations, which makes the entire pipeline tangible: you can inspect every stage that normally hides inside industrial training runs. The project has trended 58 times, ships a friendly project page, and remains actively maintained with pushes in 2026 under the Apache License 2.0.

MiniMind’s author built it after finding that existing tutorials either hid the machinery or demanded industrial hardware. By keeping the dataset, model and training scripts small but complete, it exposes details usually glossed over — how a tokenizer is trained, how masking works in SFT data, why alignment changes behavior — in a form learners can rerun, break and fix themselves.

Key Features

• Full pipeline from zero – pretraining, SFT, preference alignment and distillation in one codebase.
• Tiny by design – a 64M-parameter model trainable in about two hours on one GPU.
• Readable PyTorch – minimal abstraction so every mechanism can be studied directly.
• Modern techniques included – MoE variants, RLHF-style alignment and knowledge distillation.
• Working end product – the trained mini model actually chats, proving the pipeline end to end.
• Low cost – designed so students can afford to run every experiment themselves.

💳 Pricing

MiniMind is free and open source under the Apache License 2.0. Your only expense is the small amount of GPU time needed for training runs.

Who Is It For

Students, self-taught ML engineers and curious developers who want to truly understand how an LLM is built — by training one themselves, end to end, instead of only calling APIs.