How to install ViTables?
I don't know about the pip error and cannot check it because I've no OS X boxes available.

Regarding the conda error it is almost sure the well known conflict between pyqt5 pip package and pyqt5 conda package. A workaround is simply remove the PyQt5 dependency from the file Anaconda3\pkgs\vitables-3.0.0-0\Lib\site-packages\ViTables-3.0.0.egg-info\requires.txt (I don't know if the path is the same on Mac OS X).

You can found it explained with some more detail in the ViTables FAQ.