How do you calculate RMSE in degrees for a DOA estimator system with multiple incoming signals? Let's say you have a system with 1 transmitter and 5 receivers. We'll use the MUSIC algorithm to determine DOA of incoming signals.

If we have 1 signal (at 0° for example), it's easy to calculate RMSE vs SNR. Simply take the peak of the MUSIC spectrum (an angle) and find the difference between the actual target angle.

However, what if we have 2 signals? One at 0° and a second at 10° for example. You can no longer look at the MUSIC spectrum and simply pick the 2 greatest points, since the absolute greatest points may be at 0.0° and 0.1°, even if there's a clear (lower) peak at 10°.

What would be the/an appropriate way to measure the error when multiple signals are present? Or perhaps another metric (besides RMSE) is needed, such as probability of observing 2 peaks, etc.
The way to do it is to find peaks and not maximum values. Like the function findpeaks in MATLAB (local peaks) you can define resolution (distance between peaks and more...)