MicroSD card performance deteriorates after long-term read-only usage
Hm, my usual suspicion would be that your read access goes through a file system and that updates something, e.g. a file access time, in the file system, but that doesn't seem to be the case here.
Next thing I'm guessing: You're experiencing read disturb, a phenomenon where reading the same cell over and over again charges neighboring memory cells ever so slightly every time, or the consequence of the flash controller avoiding it:
You're probably aware that between your SD card interface and the physical isolated gate charges in your NAND flash, there's multiple layers of abstraction:

There's a higher-reliability table that maps &quot;external&quot; block addresses to internal ones, so to provide wear leveling, i.e. to avoid writing to a single block until it fails
There's (in many cases) copious amounts of error-correcting codes (ECC): Flash memory is cheap. Can't make a billion nanometer-sized MOSFETs without having a couple of less reliable ones. Also, in case of multi-level memory, there's more than one bit stored in the gate charge's value, so there's also &quot;soft-ish&quot; boundary on what means what. You'll need to add an algorithm that takes the raw things stored in the flash cells, checks whether that looks correct, and if not, correct it. That's ECC's job, and typically, reading only &quot;good&quot; cells is much faster than correcting &quot;partially wrong&quot; data.
Word sizes on the memory don't necessarily match word sizes on the external bus (SPI/MMC/xSD, whatever), so there's often some &quot;getting more and ignoring all that wasn't asked for&quot; or &quot;getting multiple and assembling to what was asked for&quot; operation.

and, here:

To avoid charging neighboring cells through read disturb, the controller takes oft-read cells and redistributes them after a threshold number of reads. That redistribution takes time.

You reading the same cells with CMD17 (which will most certainly be very suboptimal in terms of 3., use CMD18 maybe?) very much sounds either you're corrupting their neighbors (thus leading to increased time in 2.) or leading to copious re-mapping of blocks (in 4.).
I'm a bit surprised: reading audio does sound like a low-throughput operation, compared to, say, reading camera photos, but if your access pattern is very inefficient, say, you read a byte using CMD17; the flash controller has to fetch a full 2048 B block¹, then correct it, throw away all but one byte of the result, give that byte back to the SPI interface, and you then read another byte, which requires the flash controller to read 2048 B, correct.. and so on, you might be doing a very high multiple of the actual read access you think you're doing! That's unfortunate, because audio reading is probably pretty sequential.
So, my recommendation is to write a cleverer SPI SD card interface that fetches long blocks of data, and caches these in MCU RAM. This

can fully eradicate problems with randomly varying latency, because you could start fetching the next large block as soon as the first is finished being fetched via SPI, so that you always have a lot of readahead buffer
makes better use of bus bandwidth
allows the flash controller to really only access raw memory cells the necessary amount of times
converts an operation that reads a small amount of data often to an operation that reads a large amount of data rarely, allowing a lot of things to return to sleep in between, which can help save battery


¹number taken from random guessing, but order of magnitude would fit the size of error correction block codes I'd expect on modern cheap NAND flash.