Is there any diminishing return on grinding spices for a "tea"?
If you're boiling them it isn't necessary to break them down at all unless time is a factor for you.

Nutmeg also works well boiled whole as you can see in many traditional hot beverage recipes.  However, if you want to be able to add all of the spices at once, grinding some of them is advisable.  Otherwise you'll have to add the spices to your mixture at different times to avoid flavor imbalances.

The finer that you grind the spices the quicker the volatile organic compounds will be released.  In this way you can control the strength of flavor in the final product if all of the spices are added at once.

I've not been able to find any reference to boiling a spice for an extended period changing the character of the flavor(just the potency) which makes sense for a spice since we usually eat the whole thing in ground form anyway.

The same does not apply to herbs.