How dependent are stylus devices on software? I am considering acquiring a tablet or phablet with integrated stylus. One of the first things I expect to do with is flashing the latest vanilla Android release. That done, is the stylus expected to work properly? Can I then install alternative note taking programmes (e.g. Quill)? Or is the stylus functionality fully dependent on vendor software?
For a start, you can't install "vanilla Android" on a non-Nexus device. See Flash a smartphone with vanilla Android ROM for more. The closest you can get is a custom ROM that was originally based on AOSP and then ported to your device, such as Cyanogenmod.

The custom ROM's documentation should say whether it has the drivers for the stylus, and if so, it'll work just like a more precise touchscreen. You still won't get any special stylus features (like extra buttons or pressure-sensitivity) in this case, though. There's no standard Android interface for such features, so they rely on the vendor's proprietary library and interfaces for apps to support them. Again, you should check the custom ROM's documentation to find out how much is supported.