Name of the bias towards not seeing small harm of many as important? I observe this kind of thinking especially in politics, for instance when someone brings up that "It is fine to take a few dollars from a lot of people if it helps the country". I think the reason we see things like this is that we can better imagine the value of one thing with awesome qualities than million things of a small quality because we are more used to compare the quality, not the amount, in our daily lives.
Is there a name for a similar cognitive bias/fallacy?
Yes. This is a special case of the identifiable victim effect: the cognitive bias implicated in the quote, "A single death is a tragedy; a million deaths is a statistic."

The identifiable victim effect might be paraphrased as the tendency to view harm of many as unimportant. A direct corollary is the tendency to view small harm of many as unimportant. Going off AliceD's example, we might expect taking \$1 from a million to sound preferable even to taking just \$100k from 1 person. 

A potential objection might be, "sure, but in this case, 'one person' isn't really an identifiable victim. It's really about the small amount of harm since both victims are just 'statistics'." Fair point, but even if you wanted to classify this as a bias distinct from the identifiable victim effect, they both result from the same cognitive heuristic: experiential thinking.

Experiential thinking, in this context, is evaluating a scenario by imagining yourself in it and judging what that experience would be like. For example, when I imagine losing \$10, it is far preferable to losing \$100k, but the fact that it is taken from a million people has little emotional relevance. You can't imagine experiencing the loss of $10 as more than one person at a time. Similarly, an identifiable victim makes my experiential thinking more vivid because I have a concrete situation in which to imagine myself. I can imagine what it's like being mugged at 11pm in an alley near 130th Street, but "1,000 muggings in the past month" is not something I can relate to experientially.