Hivemind

Hivemind is an open-source PyTorch library for decentralized deep learning: training large models across the internet on hardware volunteered by thousands of people. It has around 2,500 GitHub stars and comes from the team behind Petals.

Overview

Instead of a datacenter, Hivemind organizes participants through a distributed hash table (Kademlia DHT) so peers can find each other, split work and average gradients without any central server. Its design goal is to make training behave like torrenting: many unreliable peers contributing small pieces add up to a system that is robust as a whole. It is fault-tolerant by design — nodes can join, leave or fail mid-training and the run adapts — and it handles the realities of training over home internet connections, including compression and asynchronous updates. The library integrates with PyTorch training loops and supports techniques such as mixture-of-experts layers. Documentation and examples show how to launch a training swarm from a handful of machines. This is the machinery that made collaborative, internet-scale training runs practical, and it is still maintained, with updates in 2026.

Key Features

• Train PyTorch models across untrusted, volunteer hardware
• Kademlia DHT for serverless peer discovery and coordination
• Fault tolerant: training survives nodes dropping out
• Gradient compression for slow, real-world connections
• Mixture-of-experts and distributed optimizer support
• Python API that plugs into standard PyTorch optimizers

💳 Pricing

Hivemind is free and open source under the MIT license. There are no fees; participants contribute their own compute. The library is a research-grade codebase you embed in your own training scripts.

Who Is It For

ML researchers and open-source collectives who want to train models collaboratively without renting a datacenter, and anyone studying decentralized or federated learning systems.