How does the ideal free distribution work?
The order of settling depends on the resource availability in different patches (in your case the difference between high and low quality habitat), but generally speaking, the pattern you observe conforms to the ideal free distribution.

The key factor in the ideal free distribution is that habitat patches are filled according to the current resource availablity (~per capita resource access), given the current density of individuals in the patch (so the marginal resource availability). If the high quality patch has much higher resource levels than the low quality patch, this means that many individuals can settle in the high quality patch before the marginal resource availability becomes equal to the resource availability in the low quality patch, at which point individuals will start settling in both patches. Therefore, the two observations you make; 1) lower overall density in low quality habitat and 2) the high quality patch being settled first, both agree with the ideal free distribution.

This graph (from wikipedia.org) illustrates this pattern nicely, for the standard version of the ideal free distribution:


Here, the marginal, apparent value of a patch is shown on the y-axis and the number of individuals in the patch on the x-axis, and the numbers refer to the order that individuals will settle in the two patches (so the first 6 settle in the high-quality Patch B, ind. 7-9 in Patch A, ind. 10 in Patch B etc).

However, I encourage you to read about other theories on habitat choice, and the pattern you describe can probably be explained by other processes as well. The ideal free distribution also has some rather restrictive assumptions (e.g. equal quality of individuals and no dominance/territorial/interference behaviours among individuals), which you have have to consider.