Waterford suffered greatly during the Civil War. In some ways it never recovered from the physical, economic, and psychological blows. Because the large Quaker population remained loyal to the Union—and steadfastly pacifist—it endured repeated Confederate harassment and depredation. Many Quakers fled to the Northern states. Samuel Cornelius Means, a Quaker who owned the mill and a house on Bond Street during the war, was finally provoked by Rebel confiscation of his horses and supplies to raise a cavalry unit to fight for the Union. His Loudoun Rangers were the only organized Union cavalry unit from Virginia. But even Union forces took their toll on this largely loyal corner of Loudoun County. Both friend and foe were scorched during General Sheridan’s “Burning Raid” in 1864, designed to destroy anything of potential value to the enemy.
After the war many former residents of Waterford stayed in the North or moved to the Midwest—others followed them. Of those who persevered in Waterford, few regained their former prosperity. Commercial development in the village was further weakened in the 1870s, when the railroad pushed west of Leesburg but bypassed Waterford. While local farmers benefited from the improved access to urban markets, the rising tide of cheaper goods from large manufacturing centers rendered much local cottage industry obsolete. New construction in Waterford dried up. While much of America experienced a rise in productivity and industry in this, the Gilded Age, Waterford struggled to survive.