DocETL

DocETL is a system from UC Berkeley’s EPIC Lab for running LLM-powered data-processing pipelines over large collections of unstructured documents. You declare pipelines of semantic operators — map, reduce, filter, resolve — in YAML or Python, and DocETL’s optimizer rewrites and schedules them so extraction is both accurate and affordable. The project has about 4,100 stars on GitHub and is actively developed in 2026.

Overview

Traditional ETL assumes structured rows; DocETL treats documents as the input and LLMs as the operators. A map step might extract every medication from clinical notes, a reduce step aggregate them per patient, and a resolve step deduplicate entities that refer to the same thing. What sets DocETL apart is its query optimizer for LLM pipelines: agent-based rewriting splits or merges operations, samples outputs to estimate accuracy, and searches for plan variants that meet quality targets at lower token cost — techniques from database systems applied to semantic processing. It includes an interactive front-end for building and inspecting pipelines, supports any major model provider or local models, and has been used for policy-document analysis, legal corpora, medical records and large-scale research datasets in published studies.

DocETL emerged from Berkeley’s EPIC Data Lab and is described in peer-reviewed database research; its optimizer treats LLM calls the way a database optimizer treats joins — reordering, decomposing and caching them, and even synthesizing new intermediate operations when the user’s plan is too coarse for the model to follow accurately. A growing library of example pipelines covers tasks like extracting provisions from legislation, coding interview transcripts and aggregating product reviews, giving new users working templates rather than a blank page.

Key Features

• Declarative LLM pipelines over unstructured document collections
• Semantic operators: map, reduce, filter, resolve, equi-join and more
• Agent-based optimizer that rewrites plans for accuracy and cost
• YAML or Python pipeline definitions with schema-checked outputs
• Interactive UI to build pipelines and inspect intermediate results
• Works with major hosted models and local open models

💳 Pricing

DocETL is free and open source under the MIT license; your only cost is the LLM API usage your pipelines consume.

Who Is It For

DocETL is for researchers, data engineers and analysts who need to extract structured insight from thousands of documents — legally, medically, scientifically or commercially — with pipelines that stay accurate at scale.