I was able to reproduce the problem in the shipping 3.7.11 version of MarsEdit, which for various reasons is still built using an older version of Xcode, against the 10.6 SDK. The MarsEdit 4 Beta, which is built against the 10.12 SDK, does not exhibit the problem.
It’s not unusual for the behavior of Apple’s frameworks to vary based on the version of SDK an application was built against. The idea is usually to preserve the old behaviors of frameworks, so that any changes do not defy the expectations of a developer who has not been able to build and test their app against a later SDK. Sometimes, the variations in behavior lead to bugs like this one.
Using a totally straightforward demo app, consisting only of an NSTextView and a button to bring up the color panel, I was able to confirm that the bug affects an app that links against the macOS 10.9 SDK, but does not affect an app that links against the 10.10 SDK.
I filed Radar #34757710: “NSColorPanel swatches don’t work on apps linked against 10.9 or earlier.” I don’t know of a workaround yet, other than compiling against a later SDK.