How to open a serial connection on High Sierra?
You need a specific driver for the USB-Serial device. MacOS support for these - especially in recent versions of the OS - is spotty, at best. You need to check even ones that claim MacOS support because their support might have worked up until about 10.6, when Apple started getting serious about kernel and driver security, sandboxing, etc.

FWIW, USB-Serial devices based on the Prolific chipset seem to have most luck.

Serial.app works because it uses its own driver, not the /dev device. For most uses, I'd say this app is required if you want to maintain sanity.