memtest86(+) not working on MacBook Pro 8,3 17" 2011?
I had the same weird behavior on a 17" Macbook Pro (Late 2011). So this is probably not a sign of trouble. I wanted to be sure that the memory was actually OK, and I remembered that there are two versions of memtest86:
memtest86 (by PassMark, memtest86.com) and memtest86+ (GPL from memtest86.org)

The Passmark versions seems to have more active development at the moment. I gave it a try, and it actually works on the MBP 17"! Be sure to boot in in EFI mode, otherwise an older version of it will start which exhibits the same reboot behavior as the memtest86.org version.