Install macOS High Sierra as VirtualBox guest (on macOS High Sierra)?
Your final HighSierra.iso is missing the complete InstallESD content (AKA the real High Sierra installer). You just copy a probably defunct and thus non-bootable base macOS system to the image.

I use the following shell script to create a working macOS installer iso file. Basically it's the same method as Apple's recommended way to create a bootable macOS installer thumb drive. The destination is a sparse dmg though.

#!/bin/bash

hdiutil create -o /tmp/HighSierra -size 5600m -layout SPUD -fs HFS+J -type SPARSE
hdiutil attach /tmp/HighSierra.sparseimage -noverify -mountpoint /Volumes/install_build
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/install_build
hdiutil detach /Volumes/Install\ macOS\ High\ Sierra/
hdiutil convert /tmp/HighSierra.sparseimage -format UDTO -o /tmp/HighSierra.iso
mv /tmp/HighSierra.iso.cdr ~/Desktop/HighSierra.iso
rm /tmp/HighSierra.sparseimage


The script assumes the default location of the High Sierra installer in /Applications.

If you don't want to create a script, copy &amp; paste &amp; execute the commands in the script line by line in Terminal.



Before booting apply some mods to the VM:

VBoxManage modifyvm "name_of_vm" --cpuidset 00000001 000306a9 00020800 80000201 178bfbff



Remove the Floppy in System > Mainboard > Boot devices
enable USB3 in Ports > USB
set video memory in Display > Screen > Video Memory 128 MB
Set RAM and number of CPUs in System > Motherboard: 4GB; Processors: 2 CPUs