Castle Hill is primarily a residential neighborhood geographically located in the South Central section of the borough of the Bronx in New York City.ts boundaries, starting from the North and moving clockwise are: Waterbury and Westchester Avenues to the North, Westchester Creek to the east, the East River to the South, and White Plains Road to the West. Castle Hill Avenue runs from East Tremont Avenue in the north to Castle Hill Park in the south.