Does Mac OS X have something like SELinux?
To answer your question as stated: Yes, MAC OS X has something similar to SELinux providing mandatory access control security polices and roles. As you have discovered it's called MAC and I believe it's based on the TrustedBSD implementation of the same name.

To answer your implicit question: No. MAC should not interfere with the ability to serve documents out of your home directory. Most likely either your permissions or your virtual host configuration is incorrect.

If I remember correctly the logs should be in /var/log/apache2 or /var/log/httpd. Please edit your question to contain the relevant parts of your virtual host configuration and logs.