Users can spam votes and users will get userpoints
One simple way to avoid this is to add a rule that removes points whenever a downvote happens. Same rule, different condition (if the value is -1) and different action: Decrease points.