Noise in audio amplifier
If it is a Bzzz or Hmmm and not a Shhhh (thanks to good input from a commenter) I think you have a classical ground loop.
When you touch your phone you form a loop from ground, via your feet to the phone to the amp and back to ground. This large loop captures significant EMI from the mains and other low frequency circuits.
You can confirm that it's not related to the phone by touching the unconnected amp inputs (e.g. at your 3.5mm audio jack when it is not plugged in to the phone). If you touch the ground ring you might hear a soft buzz, and when you touch the tip or middle ring (L/R signals) you'll hear a loud buzz.
To prevent this you have to isolate the amp input from any ground loops. A sub-100Hz decoupler could be all you need. Advanced audio equipment use a opto-coupler right at the input to prevent ground loops.
More: https://www.google.com/search?client=safari&amp;rls=en&amp;q=audio+ground+loop+isolator&amp;ie=UTF-8&amp;oe=UTF-8
This is a classical topic, and there are plenty of good videos and articles.