According to the United States Census Bureau, the town has a total area of , of which is land and , or 1.54%, is water.
Phillipston is bordered by Petersham to the southwest, Athol to the northwest, Royalston to the north, Templeton to the east, Hubbardston to the southeast, and a small portion of Barre to the south.