How to think about - and sometimes impute - geographic distances
I don't think time alone is what you are after, although it would certainly be useful.

It sounds to me like you would also want to consider the cost of transport.

Yes the drive and flight may take the same amount of time, but you could be looking at orders of magnitude difference in the total cost. If you are only looking at extremely wealthy people this may not matter, but otherwise this could have as big an impact as time alone.

My advice would be to use the existing distance column to approximate the minimum cost of transportation, similar to how you would approximate the minimum time of transportation. I would then use both time and cost as inputs to my model.