Why Does My Samsung Note 4 Shut Off When it Reaches Around 20% Battery Life?
Since no one has responded in the time it took me to do my own little experiment I wanted to post what I found.

I found that I needed a new battery.   The new battery can maintain the needed voltage down to single percentages where as, depending on what I was doing on my phone, the old battery would allow my phone to die with little to no warning like I described in the question above.

So if your phone is not draining much lower than 20-30% and dies try buying a new battery.

Also thank you benjaminS for supporting my thoughts in comments of the question.