What rivets should be used for gutter work?
It depends on your gutter brand's seam overlap. A narrow riveting area should prompt a small rivet (1/8") because you do not want to drill a large hole (for a thicker rivet) which could weaken the gutter material in the riveted area, causing failure of the gutter material under stress.

The grip range required is determined by combined material thickness.