Colleague claiming first co-autorhip of my work
First, I know the struggle is real. One of my three wishes would certainly be for us to somehow separate career and prestige from the collective pursuit of knowledge. 

There's an expression I recall from redneck NASCAR culture that seems relevant here: "Rubbing is racing." There will always be some pushing and shoving concerning who did what, particularly as scholars are trying to jump from fledgling academics to something tenured. If it's a little irritating, I'd just keep pursuing knowledge. 

If it's more than a little and you feel as though you are not being treated fairly, then I encourage you to not care about how others perceive your bringing it up. The higher we climb, the less accurately we seem to predict how supervisors will respond. So why worry about that? If he's getting undue credit, speak up; if he kind of deserves some credit, why split hairs?