General factoring and one-way functions
What you are wondering about is the difference between weak one-way function, and strong one-way function. Google should give you the definitions (e.g. here is the first link I found). The factoring function for general inputs is not a strong one-way function, since there are outputs for which it is easy to invert. But it is a weak OWF, because no polytime adversary can have more than a probability $1-1/\mathsf{poly}$ of inverting it (as far as we know).

Furthermore, even though a weak OWF is not a strong OWF in general, we know a generic way of building strong OWFs from weak OWFs - in other words, both assumptions are formally equivalent. See e.g. this online course, that deals specifically with the product function which you care about.