Mavericks Server Open Directory behind AirPort Extreme
Open Directory insists that the host name it lives on resolves to that host for both "forward" (e.g. normal) DNS lookups and "reverse" (e.g. PTR record) lookups. If this is not the case, it will insist that the "problem" be fixed or that the host name be changed to one that meets both these criteria.

In many cases, the simplest way to make that happen is to also turn on the DNS service and populate it with the appropriate zones and host entries for your LAN. Of course, you also have to configure your server and your client machines to point to your new name server, and the DNS service to forward requests to your ISP's name servers, etc.

This sounds tricky and error-prone, but so long as you make all the client changes in a new Location in System Preferences > Network it's really not bad.

An alternative approach would be to set the Open Directory server's host name to whatever.local… (At least, I think that is allowed.) I don't really recommend this route, though, unless you only want machines to be able to bind to the directory when they're on your LAN.