Is there really a difference by throwing the vanilla pod in as well?
I think you've answered your question yourself. You use the bean itself to make vanilla sugar, so obviously there is much flavour in it as well and not just in the seeds.

My experience is that you get much more flavour out of the pod if you let it simmer in warm milk/fluid.