Spark Deep Learning (Deep Learning Pipelines) is Databricks’ open-source library for running deep learning on Apache Spark. It set out to make distributed model training and large-scale inference look like ordinary Spark jobs: train with Horovod across a cluster, or apply a deep learning model to a DataFrame of images and tensors with a Spark transformer. The repository has around 2,000 GitHub stars; development has wound down (last code push in 2023) and the repo today mainly hosts the HorovodRunner component and API documentation, with the functionality largely absorbed into Databricks Runtime for Machine Learning.
The library’s two historical pillars were distributed training via HorovodRunner, which distributes a Horovod training function across Spark executors, and batch inference, which wrapped TensorFlow and Keras models as Spark ML transformers for scoring large datasets. It reflects the first generation of Spark-plus-deep-learning tooling that let data engineers stay inside the Spark ecosystem instead of exporting data to a separate training cluster.
Free. The library is open source under the Apache-2.0 license; running it requires your own Spark cluster (or a Databricks workspace, which is a separate commercial product).
Data engineers maintaining legacy Spark-based ML pipelines, and teams studying how distributed training and batch inference were integrated into the Spark ecosystem.
Deep Learning Pipelines arrived when Spark users wanted to apply neural networks to data already living in Spark clusters, and its TensorFlow-as-transformer pattern influenced how later systems approached batch inference on DataFrames. While new projects generally choose modern successors, many production clusters still run pipelines built on this library, and its HorovodRunner remains a documented integration point in Databricks environments.