John of Gaunt was lord of the manor from 1372 and Aylsham became the principal town of the Duchy of Lancaster.
The historic Black Boys Inn in the Market Place is one of Aylsham's oldest surviving buildings, and has been on the site since the 1650s, although the present frontage dates to between 1710 and 1720.