Automatically smooth bevel faces on apply
Never mind, I found an easy way. Set a material for the bevel modifier, thankfully Blender keeps the material assignment after applying. So it's easy to select and set faces to smooth.