Can an iPhone use OS X Server's Calendar service without enabling iCloud calendars?
Yes, you can add an OS X Server calendar as a separate account through Settings → Mail, Contacts, Calendars → Add Account → Other → Add OS X Server Account.

This is a completely separate account to your iCloud account. You can allow sync outside of your LAN by forwarding the relevant ports.