Google Latitude History is telling me that I'm living on the next block
You can manually set the location using an iGoogle gadget or through Firefox.  What you are probably seeing when the location is "one block away" is that GPS is not actually turned on on the handset.  The phone is probably using positioning that is determined used cell phone tower location, so it will get you in roughly the correct place, but not the exact location you would expect if GPS was turned on.