Error When Expanding Relationship Section on Advanced Search
So this all boiled down to this issue, which has to do specifically with custom fields that are integer selects, and it was impacting both the custom fields and the relationships sections of the advanced search since I was using the same custom field on a custom relationship.

The short-term fix is I went into the database and made the custom field in question not searchable and set is_search_range to 0 as well, which resolved the issue with the relationships section on advanced search throwing an error. 

That does mean that for the time being I can't search by that custom field, which is unfortunate since it's kind of a key piece of what I might want to sort/filter by, but I'll keep an eye on the bug and if push comes to shove I can always change that custom field into a text field and rework things since in this case although the data is integers it doesn't have to be since I'm not running math on it or anything, it's essentially just a numeric label.

Hope that helps someone else who runs into this with some thought process or at least raises awareness about the bug.