Were predatory journals widespread before publishers of reputable journals introduced the "author pays" model?
Anecdotally, a significant proportion of journals on post-Soviet space in 90s were quite happy to publish literally anything as long as author covers the "publication costs". The peer-review was either very light or non-existent (e.g. authors could invite their friends to act as a referee). This was used by some high-rank officials to secure the publications required for their academic degree, which gives a bearer a certain level of prestige even if they never practice science.

This happened before any OA was introduced in Europe/USA and it is still happening now. The existence of predatory journals is probably more driven by the publish or perish principle, than by OA.