Display a new entry modal when clicking on a relation field
I'd rather look into nesting Super Table fields within your Matrix block types than relating Entries. The Entry element type and its relationship field come with a lot of inappropriate UI for the use case you're trying to solve.

Another approach would be to write a plugin that provides its own element type and relationship field / modal UI.