Sorting multi-dimensional grouped arrays
Not really an answer that fixes the question per sé if anyone else has the same issue, but none the less the answer that worked for me was simple:

Split up your sorting where possible and simplify things! 

I wanted my main array to be sorted by condition A, grouped by condition B and than sort the sub arrays alphabeticaly. All done to the same array in a few lines of code so I could use this one array in my template to get all information from it. Bad idea.

So what I did was sort everything by condition A in PhP in my plugin, in my template I than group everything by condition B and only when I needed to iterate over my array to reveal my sub arrays, only then I sort the values alphabetically.

So short answer: keep it simple damnit! :)