Distributed ML Training Framework
Built a distributed training framework that orchestrates heterogeneous consumer laptops over a LAN, enabling synchronized PyTorch Distributed Data Parallel (DDP) training with automated node management, scheduling, and telemetry.
Related Projects
SYSTEM ARCHITECTURE
Heterogeneous consumer nodes coordinated through the cluster-management layer.
Problem
Training large models on a single machine is not practical, and off-the-shelf distributed training setups often assume homogeneous, centrally managed infrastructure. The challenge was to make distributed PyTorch training work reliably across mixed consumer hardware on a local network.
Architecture
Browser-driven cluster manager coordinating a SQLite-backed registry, a job scheduler, and node agents that register workers, stream telemetry, and launch PyTorch DDP workers across CPU and GPU nodes.
Challenges
- Designed node discovery across heterogeneous hardware
- Managed DDP initialization and rendezvous
- Implemented a scheduler for available compute nodes
- Added a persistent cluster registry using SQLite
- Implemented worker heartbeat and telemetry streaming
- Added CPU fallback using the Gloo backend
Benchmarks
Implemented multi-node distributed training with automatic worker registration, gRPC-based control messages, job scheduling, hardware telemetry, CPU and GPU node support, and fault-aware node monitoring.
Lessons Learned
- Overlap of communication and computation matters more than raw node count
- A small persistent registry can make a distributed system much easier to debug
- Telemetry and heartbeat checks are essential in heterogeneous clusters