Can I open a PDF file that is possibly locked on mac?
Some vendors of PDFs do encrypt them in third-party DRM, which requires the use of a special reader, or a plug-in for Adobe Reader/Acrobat. (Such as FileOpen).
Realistically, you need to ask the vendor about whether the files they supply can be read on standard PDF readers, and what to do about recovering access to your account.
It's arguable whether such files are 'true' PDFs. The first four characters of a 'normal' PDF file should be %PDF, so you can open a PDF file in a text editor and see if the files you've found are indeed PDFs. (Of course, it's possible for them to be incomplete or damaged even if they have this header.)
I'm not sure whether iBooks can handle DRM: It's likely just the same PDF engine as is used in Preview.