Custom validation rules on frontend form
If your validation rules are truly custom and aren't covered by the default Craft validation rules, the only way I can think of getting them into the Guest Entries EntryModel is to fork the plugin and add your own MyPluginEntryModel that extends EntryModel adding a rules method with your custom validation logic.

Read here for more details on that.