Google apps timeline inaccurate location
Do you use your Google account on more than one device? Or do you turn off location and your cell data and connect only via wifi? If you were connected to WiFi on your phone, or another WiFi or ethernet connected device logged in with your Google account, Google may have assumed your location was the location of the IP address you were using (which isn't necessarily where the actual WiFi router/access point is). 4 miles away is about the right distance for say a cable broadband hub and the IP address location may have resolved to somewhere near the hub. 

If you go to http://whatsmyip.org and click on the "more information" button, it will attempt to locate you via your IP address. That's likely analogous to where your weird Google location came from.