Is there an inherent recency bias in deep learning?
Your question is very interesting, however I feel you are overlooking a key point in your reasoning:
You usually take a model checkpoint at the point that it performs best on the validation set. This means that the instance of the model you keep is inherently the most robust and generalizable version of the model that you have evaluated, thus suffering the least from recency bias.
Suppose though, you don't checkpoint the model but stop it at a point arbitrarily. Naturally, you'd think that the samples in the final batch influenced the current state of the model much more than the first batches of the epoch. In practice this would show up as regular overfitting, however, instead of recency bias.
Some ways to deal with this:

relatively small learning rate
equivalently regularization as parameter norm penalties (i.e. L1, L2, ...)
ensembling
other more specialized techniques such as SGDA