How might big Matrix/Relationship fields affect front-end CP performance?
Matrix fields are going to suffer from similar scalability issues if you have a large number of blocks in a single field, affecting memory consumption, MySQL/Javascript performance, and usability. We consider that a bug (and it has been logged in our bug tracker as such), and plan on addressing it at some point in the future.

Relational fields have similar issues if you plan on selecting hundreds of elements. If you simply have a section with hundreds of entries, but don’t intend on actually selecting more than a handful, you don’t have anything to worry about; the element selector modal that comes up when you want to select additional elements is perfectly scalable, because it only loads 50 elements at a time, and lazy-loads additional elements as needed via an infinite scrolling implementation.