What is the recommended amount of video memory to assign to OS X El Capitan running in VirtualBox?
Your allocated video memory depends on what you are going to be doing with your VM.  For instance, I use 8MB for my headless FreeBSD machines and 256MB for my Windows 10 VMs.  Windows Servers 2008 or 2012 get anywhere from 16 to 128MB of RAM depending on what they are used for.

However, if you set it to 256MB (max for one monitor) you should have ample memory to run El Capitan.

I do recommend maxing out the memory in your iMac, however (32GB RAM).  That has made all the difference for me.