What is the difference between the adversary and the algorithm?
This statement is just stating that since there exists an algorithm that can factor a number in time sqrt(N), we can use what we know about this algorithm to calculate the probability that the adversary can win an indistinguishability game against a challenger using this encryption scheme. 

In other words, the algorithm is used to calculate the advantage of the adversary. Since the algorithm can factor $N$ in time sqrt(N), we must pick big enough primes, so that no probabilistic polynomial time adversary can attack the system.