LLM-OFA: On-the-Fly Adaptation of Large Language Models to Address Temporal Drift Across Two Decades of News

We investigate the problem of on-the-fly adaptation (OFA) with online feedback for large language models (LLMs) in the context of temporally evolving data. In this setting, each incoming instance-or a small batch- is first processed for inference, and its true label is revealed immediately after prediction, allowing the model to be updated in a sequential, single-pass manner. While pre-trained LLMs achieve state-of-the-art results across NLP tasks, they often struggle to generalize under dynamic distribution shifts-particularly in continuously evolving environments. Despite the importance of this problem, existing research on online adaptation of LLMs remains limited, and there is a lack of large-scale benchmarks for evaluating such methods. To address these gaps, we introduce 1M-News, a large-scale benchmark of one million New York Times headlines spanning two decades, and benchmark six state-of-the-art LLMs by fine-tuning them on the first 10 years and applying OFA on the following 10 years. To improve adaptation performance, we develop Adaptimizer, the first optimizer specifically designed for OFA, enabling rapid and stable model updates under temporal distribution shift. Adaptimizer maintains two sets of weights-fast and slow-balancing rapid adaptation with long-term stability and generalization across the stream. Our experiments demonstrate that OFA with Adaptimizer achieves consistent improvements over static baselines. All code and data are publicly available at https://github.com/pouyaghahramanian/LLM-OFA.