Can the ARM processor be switched to unprivileged mode inside handler mode?
So can I switch back to unprivileged here?


Yes. The simplest method would be to set up another stack for the unprivileged "thread". 

Now you can simply switch stacks in the interrupt handler and use the standard "exception return" by moving a special value into the PC.