Decades of fertilizer and manure applications have led to a buildup of phosphorus (P) in agricultural soils and sediments, commonly referred to as legacy P. Legacy P can provide a long-term source of P to surface waters where it causes eutrophication. Using a suite of numerical models, we investigated the influence of legacy P on water quality in the Yahara Watershed of southern Wisconsin, USA. The suite included Agro-IBIS, a terrestrial ecosystem model; THMB, a hydrologic and nutrient routing model; and the Yahara Water Quality Model which estimates water quality indicators in the Yahara chain of lakes. Using five alternative scenarios of antecedent P storage (legacy P) in soils and channels under historical climate conditions, we simulated outcomes of P yield from the landscape, lake P loading, and three lake water quality indicators. Legacy P had a significant effect on lake loads and water quality. Across the five scenarios for Lake Mendota, the largest and most upstream lake, average P yield (kg ha−1) varied by −41 to +22%, P load (kg y−1) by −35 to +14%, summer total P (TP) concentration (mg l−1) by −25 to +12%, Secchi depth (m) by −7 to +3%, and the probability of hypereutrophy by −67 to +34%, relative to baseline conditions. The minimum storage scenario showed that a 35% reduction in present-day loads to Lake Mendota corresponded with a 25% reduction in summer TP and smaller reductions in the downstream lakes. Water quality was more vulnerable to heavy rainfall events at higher amounts of P storage and less so at lower amounts. Increases in heavy precipitation are expected with climate change; therefore, water quality could be protected by decreasing P reserves.