Why do small mirror imperfections matter with modern computers
correct for the distortion

An imperfect mirror does not produce a distorted image - it produces a blurry image. With light-field sensors and phase imaging, one could possibly correct for the blur, but it is much more challenging problem than normal lens distortion correction.
Distortion refers to a systematic change in how shapes are projected in an image. It results from a lens or mirror with good, accurate geometry that just does not produce a rectilinear projection.
Random imperfections in a mirror do not cause distortion. Every point in the surface of a mirror contributes to every pixel in the result image. If a single part of the mirror is at slightly wrong angle, it does not cause a distortion in one point of the image. Instead, it projects the same image at a slightly different alignment on the same sensor. (1)
In the case of a starfield, this would cause ghost images of very dim stars to appear next to the real stars. Repeat this for a thousand imperfections, and the result is just blurry dots. Deconvolution is a process that can be used to remove blurriness, but noise and other uncertainties limit its effectiveness.
(1) This may be a bit unintuitive if you think about funhouse mirrors where the image is distorted. Those work differently because they act the part of a planar mirror, where indeed each part of image is reflected by a single part of a mirror. But planar mirrors cannot form an image by themselves, instead the lens in your eye is the critical component of the image accuracy.