Is it better to use ePub or mobi files in iBooks?
ePub is the newer standard and is supported by the Apple iBooks app, but Apple only supports it's own FairPlay DRM wrapper used to encrypt the ePub files to protect the books from being copied or read on unauthorised devices. 

Most other (non-Apple) eBook stores sell ePub books encrypted with Adobe's Adept DRM wrapper, which you can't read with the iBooks app (although certain other eBook apps do support the Adobe DRM system)

Mobi is for other reader systems (I believe the Kindle uses this format) and is not supported by the iBooks app. I expect there are other iPhone apps that support mobi files.