Back to Projects
Weather Prediction at Scale
Built a city-specific weather prediction pipeline using gradient boosting models trained on approximately 10 years of historical weather observations across thousands of cities.
XGBoostPythonPandasTime SeriesML Pipelines90GB Data
Related Projects
SYSTEM ARCHITECTURE
City-specific XGBoost models are exposed through an API-oriented serving layer.
Problem
Weather forecasting at a city level needs a scalable modeling approach that can handle many locations while keeping per-city signal strong enough to be useful.
Architecture
Data pipeline that trains one XGBoost model per city using long-range historical observations and city-specific feature sets.
Challenges
- Managed a dataset of roughly 90 GB
- Modeled 4,300+ cities independently
- Built a training workflow that scales across many per-city models
- Balanced locality-specific patterns with global feature reuse
- Handled missing observations and temporal consistency
Benchmarks
Approximately 10 years of history, 90 GB of data, and 4,300+ modeled cities with one XGBoost model per city.
Lessons Learned
- Per-city models can outperform a single global model when local patterns matter
- Data organization matters as much as model choice at this scale
- Pipeline simplicity helps when training thousands of related models
- Automated retraining is essential as new data arrives