Name of this rearranging/sorting problem?
Note: It is a hardness proof, and I think there are practical algorithms like integer programming, etc.

Given a BIN_PACKING instance where you want to pack $K$ numbers $n_1,\ldots,n_K$ into $L$ bins of size $m_1,\ldots,m_L$, and it is ensured that $\sum n_i=\sum m_j=N$, then we could design a instance of your problem as follows:


There are $K+(N+1)(L-1)$ classes;
The first $K$ classes have size $n_1,\ldots,n_K$ respectively, and each of the rest classes have size $N+1$;;
The array is partitioned into slots of size:
$$m_1,(N+1)^2,m_2,(N+1)^2,m_3,\ldots,(N+1)^2,m_L$$
where each slot of size $(N+1)^2$ is packed with $N+1$ classes, arranged contiguously, and the rest are arbitrarily arranged.


Now a key observation is that it is meaningless to keep at least one class in a $(N+1)^2$ slot unmoved and move other ones (because it won't change the size of a 'bin'). So the original bin packing is available if and only if the minimum number of swaps is no larger than $N$. Since BIN-PACKING is known to be strongly NP-complete, your problem is NP-hard.