The limestone mill has ledgers pertaining to its operations that report mill tranactions dating back to 1829. A main structural beam within the mill has the letters "D. H. carved into it along with the the year "1814".

The Virginia Survey of Historic Buildings has suggested the date of 1840, although the limestone structure suggests the earlier dates. The miller's brick house, the Spitler Residence, is situated on a rise opposite the mill along Baylor's Mill Road.

Martin Baylor's original deed for 50 acres showed the land was purchased in 1797 from a German settler named Peter Hanger. The 1999 owner, Gene Spitler, a descendent of J.M. Spitler, who owned this mill at Arbor Hill a century before, believes the mill had some relationship to Hessian soldiers that settled the area after the Revolutionary War. Many of them were stone masons.

Generations of Spitler family members have perpetuated the story of how flour was hauled over the Blue Ridge Mountains to Scottsville in barrels and floated to Richmond down the James River before the railroads were built.