Disable image autoscale when dimensions are bigger than my settings
You don't want to return the message "your image is too big" as people nowadays are uploading directly from their mobile phones and do not have (or know) a way of resizing/cropping themselves before uploading.

So you should try it via another way: do not impose restrictions on the original image, but use an image style to display the uploaded images and add some smart/user/custom crop effect to that image style. There are a number of modules that let users/editors custom crop images, e.g:


Imagefield crop
Manual Crop
Smart Crop
...


Also see Comparison of image cropping and resizing modules.