Quadratic residues are so much fun! Definitions
Quadratic residues
An integer \$r\$ is called a quadratic residue modulo \$n\$ if there exists an integer \$x\$ such that:
$$x^2\equiv r \pmod n$$
The set of quadratic residues modulo \$n\$ can be simply computed by looking at the results of \$x^2 \bmod n\$ for \$0 \le x \le \lfloor n/2\rfloor\$.
The challenge sequence
We define \$a_n\$ as the minimum number of occurrences of the same value \$(r_0-r_1+n) \bmod n\$ for all pairs \$(r_0,r_1)\$ of quadratic residues modulo \$n\$.
The first 30 terms are:
$$1, 2, 1, 1, 1, 2, 2, 1, 1, 2, 3, 1, 3, 4, 1, 1, 4, 2, 5, 1, 2, 6, 6, 1, 2, 6, 2, 2, 7, 2$$
This is A316975 (submitted by myself).
Example: \$n=10\$
The quadratic residues modulo \$10\$ are \$0\$, \$1\$, \$4\$, \$5\$, \$6\$ and \$9\$.
For each pair \$(r_0,r_1)\$ of these quadratic residues, we compute \$(r_0-r_1+10) \bmod 10\$, which leads to the following table (where \$r_0\$ is on the left and \$r_1\$ is on the top):
$$\begin{array}{c|cccccc}
&amp; 0 &amp; 1 &amp; 4 &amp; 5 &amp; 6 &amp; 9\\
\hline
0 &amp; 0 &amp; 9 &amp; 6 &amp; 5 &amp; 4 &amp; 1\\
1 &amp; 1 &amp; 0 &amp; \color{blue}7 &amp; 6 &amp; 5 &amp; \color{green}2\\
4 &amp; 4 &amp; \color{magenta}3 &amp; 0 &amp; 9 &amp; \color{red}8 &amp; 5\\
5 &amp; 5 &amp; 4 &amp; 1 &amp; 0 &amp; 9 &amp; 6\\
6 &amp; 6 &amp; 5 &amp; \color{green}2 &amp; 1 &amp; 0 &amp; \color{blue}7\\
9 &amp; 9 &amp; \color{red}8 &amp; 5 &amp; 4 &amp; \color{magenta}3 &amp; 0
\end{array}$$
The minimum number of occurrences of the same value in the above table is \$2\$ (for \$\color{green}2\$, \$\color{magenta}3\$, \$\color{blue}7\$ and \$\color{red}8\$). Therefore \$a_{10}=2\$.
Your task

You may either:

take an integer \$n\$ and print or return \$a_n\$ (either 0-indexed or 1-indexed)
take an integer \$n\$ and print or return the \$n\$ first terms of the sequence
take no input and print the sequence forever


Your code must be able to process any of the 50 first values of the
sequence in less than 1 minute.

Given enough time and memory, your code must theoretically work for any positive integer supported by your language.

This is code-golf.
MATL, 14 bytes

:UG\u&amp;-G\8#uX&lt;


Try it online! Or verify the first 30 values.

Explanation

:      % Implicit input. Range
U      % Square, element-wise
G      % Push input again
\      % Modulo, element-wise
u      % Unique elements
&amp;-     % Table of pair-wise differences
G      % Push input
\      % Modulo, element-wise
8#u    % Number of occurrences of each element
X&lt;     % Minimum. Implicit display