The Hall was reputed to have belonged to the Ayscough (Askew, Ainscough) family in the early part of the 16th century - Fee referring to the Knight's Fee or living from the property.
In the seventeenth century, the Hall passed into the Johnson family.