Any way to get more powerful rules for Mail.app?
There is no way I can think of to use AppleScript to filter messages directly, but I guess you could go the ugly route of redirecting "All Messages" to an AppleScript and then handling them from there back to Mail.app.

==== Update:

To give you a better idea of what I´m talking about, you´ll have to create a new rule with condition "All Messages" and execute an Applescript. For example to check for the subject, you´ll want something similar to this:

using terms from application "Mail"
    on perform mail action with messages theMessages for rule theRule
        tell application "Mail"

            repeat with eachMessage in theMessages
                set theSubject to subject of eachMessage

                if theSubject contains "test" then
                    -- do something here, like move message to a folder
                end if
            end repeat

        end tell
    end perform mail action with messages
end using terms from


Note that of course you could easily use a do shell script statement here to use whatever command line tool you want to compare the subject with a regexp, like if (do shell script "*your regexp that returns 'true' if matched*" ) is true … and so on.