Trouble with Memtest86 4.0a on 15" MacBook Pro
An easy way to get memtest running is the Ubuntu install cd which has memtest as a menu entry.

Note that you previously had to run Boot Camp first to make the system capable of booting Linux or Windows.  I do not know if this is still the case with OS X 10.8.