The manor once comprised 52,500 acres or 200 sq. miles of property owned by Frederick Philipse, a Dutch colonial who was Peter Styvesant's chief carpenter. He went on to become one of the wealthiest men in New Amsterdam. In 1672, Philipse purchased the Yonkers' Nepperhan millsite, which soon became the center for his estate established by royal decree in 1693 as "Mannour of Philips Borough". The manor house dates to 1680, and mills were built at the Upper Mills and the Lower Mills at about the same time.

The Upper Mills was built with a manor house, gristmill, warehouse and wharf, plus housing for about 40 families to run the facilities. Much of the labor was from African slaves to build and run the enterprise at the junction of the Pocantico and the Hudson Rivers. A cove on the Hudson at the foot of the Upper Mills dam allowed small ships to load and unload at the mill and warehouse. The Philipse Estate was sold by the State of New York in 1786, after the revolution, and when the last Philipse had died in England, where they had sought refuge during the emotionally charged Revolutionary Period.The Upper Mills remained an intact farm parcel through the mid 1800's, then was owned by a succession of owners until 1937, when through the efforts of the Tarrytown Historical Society to save the Upper Mills Complex resulted in interest and help from a wealthy neighbor, John D. Rockefeller. This push, to save and restore the Manor and its environs, resulted in the formation of the Sleepy Hollow Restorations.

The manor house and other buildings were restored, and a period barn built or moved in, and the gristmill was reconstructed in 1958-60. The property was opened to public use in 1969. Sleepy hollow Restorations also restored and maintain The Van Courtlandt Manor and Washington Irving's Sunnyside residence.

The Haunted Mill, the Old Mill at Tarrytown or Sleepy Hollow. A Relic of the American Revolution. This is an 1870 photo of the derelict original mill. The wheel house and horizontal water wheel have been removed. This images is similar to the classic Currier & Ives print, "Mill-Dam at Sleepy Hollow."

The Old Philipse Manor House and Mill, Sleepy Hollow, New York. A later 19th century photograph of the original mill.

A 1941 post card of the Fitz Water Wheel Company Restoration of Philipse Castle Upper Mill.

A Postcard image of the 1941 Fitz Water Wheel Company Restoration of the Dutch Upper Mill at Sleepy Hollow. Note the wheel house for the horizontal water wheel.