Class for modifying stats of units
IMHO The names are fine. You could maybe want to modify Modifiers in StatModifiersSummary as it's, after all, a summary "view" of the actual modifiers.

You should make its fields final or private, though.

In that case, your calculator would become StatModifiersSummaryCalculator.

W.r.t. extending / having one map inside, composition is in general preferable to inheritance, so stick with the current design: after all you happen to use a Map, but there is no reason why you couldn't use another representation and you don't want your "users" to be able to pass your calculator around where a map is needed.