What sort of nails should I be using on Sheet Vinyl underlayment?
Unless installing a rigid material such as new plywood, luan, cement board, etc - you generally do not nail it down. When you do, ring shank is the right type of nail. But it has to be a decent quality nail vs. the cheapest bidder, also, the condition of the flooring it is being nailed into is quite important. Just nailing into plywood isn't going to cut it because it is multiple layers of glued sheets, and the glue dries over time and plywood separates. This gives very little for the nails to bite into.

Most underlayments are cut into place and taped down (example: installing underlayment) or glued with flooring glue. Even most nowadays don't use that. Depending on the product you buy, many have self-adhesive backings. While many many products have gotton cheaper in design, thickness, etc. over the years, engineering has been thought out for the full product installation. (sometimes for the worse). Your product should have guidance on the methods to be used.

Once you have divots, dimples, nail heads, or other imperfections in the underlayment which almost always happens, you will need floor leveler. Its a simple cement product you add water to, spread it, dry, clean and sand as/if necessary, then you can add the new flooring. If this is not done, the nails will show through even in many thick flooring products. (Here is a quick googled example on youtube: floor leveling)