Custom php in rules how to check users' geolocation lat lon field and return only users in certain area
How about using javascript? You could submit it with ajax and write it to the db.

Here is a good example of a javascript geolocation function to get long and altitude:

window.onload = function(){
    if(navigator.geolocation)
        navigator.geolocation.getCurrentPosition(handleGetCurrentPosition, onError);
}

function handleGetCurrentPosition(location){

    location.coords.latitude;
    location.coords.longitude;
}
function onError(){...}


More complete demo:
http://od-eon.com/labs/geolocation/

For the calculation of the distance with php:
http://www.dzone.com/snippets/distance-calculation-using-1