Can't modify partitions
Thanks @Tetsujin. The listing looked a little screwy so I was a little hesitant to go mucking around with the command-line diskutil. But I ended up doing a diskutil corestorage list and then 

diskutil corestorage delete [UUID of 50GB Logical Volume Group]

GUI Disk Utility correctly saw the 50 GB of empty space as part of the other Logical Volume Group and I was able to delete it and merge it back in fine.