How do I reassign numbering of Displays when you have more than 2 monitors connected to mac mini
This is a very annoying problem and I have not found any official documentation on how to control the numbering of the Desktop on each monitor. However, through a lot of trial and error, I've discovered these rules that will help you order the Desktop numbers.
First, a few key points on how the numbering behaves

Your 'Home Screen' (the display with the Dock) will always have the lowest number (e.g. Desktop 1, Desktop 2 if it has 2 desktops)
When you drag the Dock (in Displays &gt;&gt; Arrangement) from one display to another, it will swap the numbering between the two monitors that you dragged it to and from. For example if you have 3 monitors that is setup as follow


Monitor A (Desktop 1, Desktop 2) ==DOCK LOCATION==
Monitor B (Desktop 3, Desktop 4)
Monitor C (Desktop 5, Desktop 6)

If you drag the DOCK from Monitor A to B, you'll end up with the following:

Monitor A (Desktop 3, Desktop 4)
Monitor B (Desktop 1, Desktop 2) ==DOCK LOCATION==
Monitor C (Desktop 5, Desktop 6)

Therefore, by planning the sequence of DOCK position swaps, you can reach the final ordering that you desire.
The ordering may get messed up again if you disconnect the monitors or return from sleep or restart. I have no idea how to 'save' the settings. But at least this will help you reconfigure quickly.