Was the Earth-Sun distance ever calculated based on the time difference between night and day?
You have a good idea. You mentioned 4 effects already (clock accuracy, latitude, time of year, and clear horizon), but there is another effect that is larger than those: atmospheric refraction. Refraction causes a rising object to appear to be half a degree higher than reality. Refraction depends on atmospheric pressure and temperature, so it may be harder to compensate for it.
This article on Sky &amp; Telescope indicates that the amount of refraction is not well understood: We Don’t Really Know When the Sun Rises

But Michigan Tech dissertation work by Teresa Wilson (now at the U.S. Naval Observatory) suggests that our estimates are often off by 1 to 5 minutes.
She found that, overall, predicted times varied in accuracy by location and season, with sunrise times over land generally being early in the summer months and late in the winter. Summer showed the largest discrepancies, probably due to the pronounced refractive effect that the large temperature difference in the atmosphere has during those months. Mirage effects due to cold air topped by warm over water horizons also exacerbated lags in sunset times throughout the year, sometimes by up to 5 minutes.