Philipstown was first settled in the westernmost of this parcel in around 1715. In 1754 the Highland Patent, subsequently known as the Philipse Patent, was divided among Philipse heirs into nine parcels. In 1788 Philipstown was established out of the three river lots and part of a fourth inland to the north, becoming one of the three original towns in what is now Putnam County. In 1806 a very small portion north of the Hudson Highlands by the mouth of Fishkill Creek was split off from Philipstown and given to the Town of Fishkill.