Mail.app smart mailbox, how to negate a search filter clause (_not_ flagged)?
I would create a smart mailbox containing all Flagged items, and then (this is the one you want) a smart mailbox with the 'Not In Mailbox' option to return all the items that are not Flagged, and thus not in that mailbox.  

Create this first: 

Then for the mailbox you want, create this one:


Worked for me in Mail 4.6, OSX 10.6.8.