Lat/long gets reset after updating
Found the solution. 

The lat/long came from a different source and when the location module tried updating the lat/long I got NULL/NULL, because I was on the max amount of requests to google. 

Solution was to set the 'source' column in the location table to 1, so the module thinks the value is overruled by the user. The lat/long doesn't get updated / reset to null anymore.