Magnet encoder accuracy problem
I know nothing about these chips but a quick look at the AS5147 datasheet bottom of page 8 shows:

Figure 1. AS5147 RMS output noise.
Assuming that 14-bits is 360° then 0.082° is \$ \frac {0.082}{360} = 0.000228 \$ or an error of 0.0228%. Note that this is the RMS noise value. You'll have to figure out how that relates to the max and min values.
Meanwhile you're seeing 3-bits noise on your 14-bit signal. Therefore 11 bits are remaining stable and the 12th bit is unstable. \$ \frac {1}{2^{12}} = 0.000244 \$ or an error of 0.0244%.
Someone will correct the details of my calculations here but it seems to me that you're running close to specification.

Analog Devices AN615 may help.

Figure 2. Histogram for a Grounded Analog Input.

Noise
Figure 1 shows a typical histogram obtained from a
sigma-delta ADC when the analog input is grounded.
Ideally, for this fixed dc analog input, the output code
should be zero. However, due to noise, there will be a
spread of codes for a constant analog input. This noise
is due to thermal noise within the ADC and quantization
noise due to the analog-to-digital conversion process.
The code spread is generally Gaussian in nature.The rms
noise is calculated using the curve that results from the
histogram, the width of the curve determining the rms
noise. A Gaussian curve goes from –infinity to +infinity.
However, 99.99% of the codes occur within 6.6 \$ \times \$ rms
noise.Therefore, the peak noise is 6.6 \$ \times \$ rms noise.