Petastorm is Uber’s open-source data library for training and evaluating deep learning models directly from Apache Parquet datasets, on a single machine or across a cluster. It bridges the gap between big-data storage formats and ML frameworks, streaming rows from Parquet files into TensorFlow, PyTorch and PySpark pipelines without custom conversion code. It is an open-source project on GitHub with around 1,900 stars, still updated in 2026.
Large training sets usually live in Parquet on data lakes, while training frameworks expect their own input pipelines. Petastorm provides a unified reader that handles sharding, shuffling, partitioning and predicate pushdown, and materializes records as tensors or dictionaries ready for model code. It supports datasets stored on local disks, HDFS and cloud object stores, decodes complex nested fields, and can cache decoded data for repeated epochs. Born inside Uber’s autonomous vehicle and Michelangelo ML platform work, it is battle-tested at significant scale.
Free. Petastorm is open source under the Apache-2.0 license.
Petastorm offers two reading styles: a row-wise reader that feeds frameworks through a Python iterator, and a batch reader optimized for Spark-driven preprocessing. Datasets carry an embedded schema (Unischema) describing field types and encodings, so image bytes, arrays and scalars survive the trip from Parquet to tensor without per-project glue code. On the write side, utilities help Spark jobs materialize training datasets with the metadata Petastorm readers expect, closing the loop between data engineering and model training.
ML platform and data engineers whose training data already lives in Parquet, and teams that want one reader layer across experimentation and distributed training jobs.