Data lake-aware checkpointing: Enabling resilient large-scale model training through precise data consumption tracking

Data Lake-Aware Checkpointing addresses a critical gap in large-scale model training resilience by incorporating reader state as a first-class citizen in training checkpoints. While traditional frameworks save only model and optimizer states, they neglect data reader progress, leading to overlapping or missed data reads when resuming training from massive data lakes. This article proposes a system that tracks consumed Parquet files and row group offsets across distributed fleets, and includes that information as part of the checkpoint, creating comprehensive checkpoints that enable precise recovery without data loss or duplication. It integrates seamlessly with existing training pipelines and distributed storage systems, establishing the foundation for truly epoch-less, streaming-style training on vast data repositories. By treating data consumption state with the same importance as model parameters, we significantly enhance fault tolerance and training reliability for large language models at scale.