Anyway to extend Craft Entry query search in Craft 3
The "proper" way to access the element query that gets generated is through a plugin via the ElementQuery's query and subquery properties.

It looks like you're doing a Haversine algorithm to calculate distance between a latitude and longitude.  There's are some similar plugins that already do this you can use as an example.

For example: https://github.com/ethercreative/simplemap/blob/v3/src/services/MapService.php