Reducing noise in elementary audio circuit
It's really hard to tell from your description.  A good description of the noise and when it happens would be a good start.

You already said that the noise happens when you play a note.  What happens if the volume is turned down?  Either using the control on the electronic piano or by using the pot in your circuit.  If the noise goes away then you can assume that the problem is either with the signal level on the input, or that you're asking the amp to put out more than the rated 3.5 watts.  

If I were a betting man, I would guess this is the case.  You're using a 3.5 watt amp to power a 15 watt speaker.  Normally for a 15 watt speaker you would be using a 15 to 30 watt amp.  So your amp is undersized.  

Adding a cap like what @Manmanguruman suggests isn't a bad idea, but I would hope that there is already some caps inside the amp module.  Still, you would use the largest cap that is somewhat practical.  Start with about 470 uF and higher.

The pot should have a log taper, not a linear taper.  Changing the pot will make the volume control more useful-- but will do nothing for your noise problem.

It is also possible that you have a signal loading problem with the piano.  Basically, the piano might not be able to handle having an amp plugged into it.  I would hope the piano was designed better, but I have seen some products do some stupid things.  Turning the volume down with your linear/log pot should help this issue.

Other than that, the only thing I can think of would be a component failure.  Something like the speaker, amp, battery, or piano actually being broken in some way.