Do all the engines/propellers of a VTOL UAV need to be identical?
MIT CSAIL has researched apparently quite thoroughly on this and the end result it would seem is that you can have ANY different size of motors be able to fly given that it satisfies certain conditions such as thrustep etc. For this I would recommend following developments in two areas (as far as my knowledge goes)
First of ckurse MIT CSAIL's video demo of their custom software to simulate all combos of motor thrust and positioning


                
             

The next 'guy' you want to follow is ofcourse
Raffaello D'Andrea  and his research team. His TED talk explores drones at mechanical disadvantages, here you go 

https://www.ted.com/talks/raffaello_d_andrea_meet_the_dazzling_flying_machines_of_the_future

That's as far as my enthusiast knowledge could help