How to perfrom modular division while numerator is lesser than the denominator?
I'll assume we are working modulo $p$, which most likely is some large prime. When evaluating $S={P_y-Q_y\over P_x-Q_x}$, one really is finding a representative integer of the residue class of the equivalence relation modulo $p$; that is, the result is defined within addition or subtraction of $p$ any number of times.

The same holds for numerator $P_y-Q_y$ and denominator $P_x-Q_x$: addition or subtraction of $p$ to these quantities, any number of times, is immaterial. That makes comparing numerator and denominator moot; and even ambiguously defined: when working modulo 7, we can't tell which is the smaller of 2 or 6; that's because we could as well be comparing 2 to -1, or 9 to 6.

As long as $P_x-Q_x$ is not a multiple of $p$, one evaluates $S={P_y-Q_y\over P_x-Q_x}$ without considering a comparison of numerator and denominator. The result is, depending on definition


any integer $S$ such that $S\cdot(P_x-Q_x)-(P_y-Q_y)$ is a multiple of $p$;
or perhaps the single non-negative integer $S$ less than $p$ with this property; this would be obtained by "reducing modulo $p$" some $S$ with this property; that is:


if $S$ is negative, change $S$ to $S+p\cdot\lfloor{p-1-S\over p}\rfloor$, where $\lfloor{p-1-S\over p}\rfloor$ is the quotient of the Euclidian division of the positive integer $p-1-S$ by the positive integer $p$, and $\cdot$ stands for ordinary multiplication;
otherwise, change $S$ to $S-p\cdot\lfloor{S\over p}\rfloor$ (or equivalently, change $S$ to the remainder of the Euclidian division of $S$ by $p$).



If we stick to the convention that $S$ and intermediary results are kept as non-negative integers less than $p$ (or, equivalently, as kept reduced modulo $p$), the evaluation of $S={P_y-Q_y\over P_x-Q_x}$ could go:


evaluate the denominator $U=P_x-Q_x$, reducing the result modulo $p$;
abort if that's zero;
compute the modular inverse $I$ of $U$ modulo $p$ using the extended Euclidian algorithm, reducing the result modulo $p$ ($I$ is well defined if $p$ is a prime);
evaluate the numerator $V=P_y-Q_y$, reducing the result modulo $p$;
evaluate $S=I\cdot V$, and reduce the result modulo $p$.


The same applies for $S$ as used in point doubling, except for what numerator and denominator are (the formula to be used will depend on the curve).