What is the purpose of manipulating trained layers? In my course there are a number of exercises which amount to taking a trained residual network and deleting, swapping or repeating layers (without retraining).
Is there any real application for doing this? An exercise might say to swap two specific layers and I will find that the performance will increase (on a very small test set) but if I try to swap any other two layers the performance decreases.
To me it seems that operations in the exercises are chosen to increase the performance on the specific test set that we have but I don't see how it would improve generalisation. In answering the exercises I can say that the performance increased but I can't say why (because to me it seems to be an arbitrary manipulation of the network).
No.  I've never heard of this being used for a published task.  Neural nets are trained with optimization (SGD) rather than by swapping/deleting/repeating layers.