How to increase graphics memory in Mac OS running as a client in VMware?
VMware's knowledge base article, Setting Custom Video Memory Size in Fusion, shows the setting to change:


  Setting Custom Video Memory Size in Fusion
  
  There is no way to choose a different video memory size through the
  user interface. It must be done by editing the virtual machine
  settings (.vmx) file. For more information, see Editing the .vmx file
  for your Fusion virtual machine (1014782).
  
  Add this line to the .vmx file:
  
  svga.vramSize = value_in_bytes
  
  where value_in_bytes is the desired memory size in bytes


The .vmx mentioned in the article is hidden within your virtual machine's bundle. To find the file to edit, right-click (or Control-click) on your virtual machine in the Finder and 
 select Show Package Contents.