How to speed up/slow down audio along with video?
Unfortunately, kazanaki's answer is your only choice.

There are multiple ways to change the speed of a clip in premiere, namely:


The Speed/Duration panel that can be access via the context menu of a clip (right-click).
The Rate Stretch Tool (Shortcut R)
Time remapping using keyframes. 


The first two do the same thing, that is change the speed of an entire clip by a constant amount. They are also intertwined, so if you change the speed and duration of a clip using the Rate Stretch Tool, the new values will also appear in the Speed/Duration panel.

Time remapping works differently from those tools, as it allows for speed changes over time. That is, you can set keyframes for 400%, 200% and 100% speed (as in your example) and Premiere will interpolate between those keyframes, creating a smooth speed transition. However, this has no effect on the audio stream of the clip. From the documentation:


  When you vary the speed of a clip with linked audio and video, the
  audio remains linked to the video, but remains at 100% speed. The
  audio does not remain synchronized with the video.


Source

I don't know if there is a reason why Adobe has designed it this way (I assume there are technical reasons for this), but at the moment there is no way to adjust audio speed alongside a video using time remapping.

If this behaviour bothers you, you can always write up a Feature Request for Adobe to consider for future releases of the software.

An alternative would be to cut your audio in Adobe Audition and use a Dynamic Link to import it back into your Premiere Pro projekt. If I recall correctly, adjusting the audio alongside the video as described above is also possible in After Effects, but don't quote me on that ...