How can I limit and show only the first referenced node per node in a view?
I can suggest you one solution which will probably help you with performance.

You can make use of views field view module.

You can use the view which you have created to render the fields of content type A and now create a separate view (as block) of content type B with current node ID as contextual filter and required relationship for the reference field and set limit to 1 or use delta as filter and call this view as field in the view in which you have rendered content type A fields.

EDIT: Second view can be called conditionaly wherever required, so is more handy if you require it only once in the parent view (by result count or once in header) OR get rendered it with every result of parent view.