Can I revert Mail.app to handle notifications in the pre-10.9.2 way?
Unfortunately this does not work out of the box. But there is a nice and free plugin called Herald which gives you this exact feature back, and many more on top.

(See Version 4.0.8 (Mac OS X 10.9) by clicking on "Old Versions").

Please close your Mail.app before installing.