How to fix poor font rendering in Mac iBooks?
It seems that there is are others that report a problem with font rendering in the iBooks app when using epub files under OS X Mavericks (10.9.x). As in your experience, the problem seems isolated to Macs that don't have a Retina Display, such as your Mac Pro.

The solution is fortunately a simple one: Go to

System Preferences &gt; General

and uncheck the box that says Use LCD font smoothing when available.

This should clear up the jagged text you've been experiencing in iBooks on your Mac Pro.

(Note: You may need to quit and restart iBooks and/or restart your Mac for these changes to take effect.)