Is iTunes Match supposed to slow my phone down so much?
The size of your iTunes Match is a possible cause for your slowdown in the Music.

Since you cannot have your entire library cached to your device, your device is probably trying to download all of the meta-data, and cover art thumbnails.  I only have around 12GB of music in my iTunes Match library, and I notice some sluggishness whenever things are added or I setup a new device.

Another cause of slowdown is actually the amount of images in the view.  Unfortunately, it is common for the device to become less responsive when there are a large number of images in a single, scrollable view.