Finding the k-th smallest rational number efficiently
It is perhaps more natural to consider the set $S$. Let us further consider the case $A = B = n$ and the set $S \cap [0,1] = \mathcal{F}_n$, the $n$th Farey sequence; if we know $|\mathcal{F}_n|$ and can solve the problem for $\mathcal{F}_n$ then we can also solve it for the entire $S$. The set $\mathcal{F}_n$ has size $\Theta(n^2)$.

The fastest algorithm known for finding order statistics (i.e., the $k$th smallest element) in $\mathcal{F}_n$, due to Patrășcu and Pawlewicz, runs in time $\tilde{O}(n^{2/3})$. This is much slower than the complexity you're interested in, $O(\log n)$.

It is possible that considering the multiset instead of the set makes the problem easier.