The mill was built in 1874 as a saw mill powered by steam, by John T. Hirst and John R. Smith. The two men soon diversified the operation and purchased a machine to separate seed from chaff.

Orchard grass was among the grains and grasses grown in the region, but was not a major crop. The mill burned in 1882, was rebuilt and burned again in 1904 and rebuilt in 1905. In 1912 electricity was added to the mill.

In 1921 it was sold to William Henry Adams and his son, Contee Lynn Adams. The Adams changed the mill's name to Loudoun Valley Milling. By the late 1930's the mill's main business was the cleaning of grass seed necessary for the expanding dairy business in the county.

During the years of the early 1940s, much of the orchard grass seed was shipped to Europe as packing for shipments of artillery shells. In 1943, Adams decided to concentrate solely on cleaning seed.

He sold Loudoun Valley Milling to Wilkins and Rogers, and moved his business across the street to the Dillon building, also built in 1874.

Adams changed the firm name to Contee Adams Seed Co. The two business agreed not to compete against each other. Adams would not grind grains or sell feed and Wilkins and Rogers would not refine or sell seed. Linehsafts and pulleys still in place give a ring of authenticity to the old mill.

A French buhr sectioned grindstone outside of the restaurant. By 1954, Wilkins and Rogers sold the mill. The mill continued to be worked until 1992.

Now, the old seed mill is known as Magnolias at the Mill, a premier restaurant complex, since 2004, geared to preserving a vital part of the western Loudoun community as a functioning entity.