DataFlow is an open-source data preparation framework from OpenDCAI that uses LLM-based operators to build high-quality training data, published on GitHub with around 8,200 stars. It treats data curation as a pipeline problem: composable operators clean, generate, filter, and evaluate data at scale.
Model quality is capped by data quality, yet preparing LLM training data usually means scattered scripts. DataFlow unifies the work into pipelines of reusable operators โ more than a hundred of them โ covering tasks such as text cleaning, deduplication, quality scoring, synthetic data generation, question-answer construction, and data evaluation across domains like general text, mathematics, and code. Heavy operators run on fast inference backends including vLLM and SGLang, so pipelines can process large corpora efficiently, and a Gradio interface makes it easy to inspect data and adjust pipelines interactively. The project comes out of an academic team (with a SIGMOD 2027 paper describing the system) but is engineered as a practical tool: modular operators, documented pipelines, and active development with updates continuing into late 2026.
DataFlow is free and open source under the Apache-2.0 license. There is no hosted paid tier; you run pipelines on your own machines and pay only for compute and any LLM APIs you plug in.
DataFlow is for LLM training teams and data engineers who build pretraining or fine-tuning corpora and want a systematic, reusable framework for data curation instead of one-off scripts.