Why does a Java based app in OS X have 2 different user interface designs?
The same Java code (mostly) runs on all OS/architectures. It has several different  APIs to draw forms/windows. Some of these APIs e.g. Swing allow you to change look and feel between emulating different widget sets. So a program can be written so that it allows the user to change the look and feel.

The look and feel that is supplied includes OSX (as in the top screenshot) and Windows(several forms) or Motif or GTK. The way to set this in Swing and for more information see this Oracle document Note that even the OSX Look and feel does not full match Apple's guidelines e.g. positioning of the menu in the window and not at the top of the display.

Like any Windowing system you can always write programs that behave and look different to the expected standard. It happens that for OSX most developers stick to Apple's guidelines, but even using Apple's supplied APIs you can have a different feel between Carbon(originally Mac OS &lt;= 9) and Cocoa programs(originally NeXT). The AppStore allows Apple to have a bit more control but I do not know if meeting display guidelines is actually checked.