Automator: Is it possible to resize an image to a maximum file size?
I'm a little confused with the approach on how to properly compress the images you want to work with. As mentioned, there is sips, a working example based on pulling a directory of images and stepping through the list with a resize if the width isn't 800px:

property theWidth : 800

tell application "Finder"
    try
        ## Choose directory
        set imageDir to choose folder with prompt "Please select directory."

        ## Get list of items in diretory
        set the imageList to list folder imageDir without invisibles

        ## Step through images
        repeat with theImage in imageList

            ## Turn path to posix path
            set imagePos to quoted form of POSIX path of ((imageDir as text) &amp; contents of theImage)

            ## Get image width
            set imgWidth to (do shell script "sips -g pixelWidth " &amp; " " &amp; imagePos &amp; " " &amp; "| tail -n1 | cut -d' ' -f4") as integer

            ## if not equal to set variable width resize
            if imgWidth ≠ theWidth then do shell script "sips --resampleWidth " &amp; theWidth &amp; " " &amp; imagePos

        end repeat
    on error error_message
        display dialog error_message
    end try


That is just an example of what you can do with the width. As mentioned in the comment you could check to see which is longer the width or height and resize from there. If you do man sips in the Terminal you can pull all types of info. The above code uses tail and cut to trim the return. Resizing is done with --resampleWidth. You could resample based on height with --resampleHeight. 

Regarding the compression aspect, if these are PNGs you can use Optipng. If you want something you can run with PNGs and JPGs you could use Trimage. AppleScript/Automater could also be scripted into using a compression utility. You would just need to call from bin a check if a compression exists:

try
    set checkBin to "(ls usr/local/bin/optipng &gt;&gt; /dev/null 2&gt;&amp;1 &amp;&amp; echo yes) || echo no"
    return do shell script checkBin
on error error_message
    return error_message
end try


The above is checking for OptiPNG. If you wanted to check wether something was installed with HomeBrew you could use a similar command on /usr/local/Cellar.

All the above mentioned could be completely done in a Shell Script. It just depends on what you want and how you want to execute the code.