How do I get music labeled under "various artists" to appear under the artist themselves?
Besides the Compilation checkbox, look under the Sorting tab.  The songs are probably configured with sort artist as "various artist".  Clear that field for all of the tracks and they will default back to the appropriate artist value.