What is the difference between the adversary and the algorithm? I recently saw a sketch relating to provable security in a book, regarding the amount of time it takes to factor N where N = pq and p,q are primes.

It says that "There is a trivial algorithm which always factors a number in time $\sqrt N$ , so there is an adversary $A$ such that $Adv_b (A, 2^{b/2}) = 1$" b is the amount of bits of $N$.

Here there is a distinction drawn between the adversary and the algorithm. However, if the algorithm can always factor a number in $\sqrt N$ why does it matter, how strong the adversary is?
This statement is just stating that since there exists an algorithm that can factor a number in time sqrt(N), we can use what we know about this algorithm to calculate the probability that the adversary can win an indistinguishability game against a challenger using this encryption scheme. 

In other words, the algorithm is used to calculate the advantage of the adversary. Since the algorithm can factor $N$ in time sqrt(N), we must pick big enough primes, so that no probabilistic polynomial time adversary can attack the system.