Is there any way to sort by Image Quality?
The only way that I see is calculate image quality when it's saving and use it later. 
How to do it:


Extend your file entity via module File entity
Add special field to store value for image quality. 
Save value of quality into this field each time when file is saving. For example via hook_entity_presave
Sort by this field in your views


How to calculate image quality? Here is good example 
but you can use your own approach to do it.

P.S. Yes it looks a little tricky but I don't think that it's possible to do it easier. But if it's possible I'll be happy to hear it.