How to query with relatedTo of a Relationship Field in a Matrix Field block
You were right about the issue being that the relationship is made within a Matrix field. Because of this, you have to explicitly pass the Matrix field's handle to the relatedTo parameter.

{% for instrument in craft.entries.section('instrument') %}
      &lt;h3&gt;{{ instrument.instrumentName }}&lt;/h3&gt;
      {% set people = craft.entries.section('person').relatedTo({
         targetElement: instrument,
         field: 'roles.instrument'
      }) %}
      {% if people|length %}
         {% for person in people %}
           &lt;p&gt;{{ person.personName }}&lt;/p&gt;
         {% endfor %}
      {% else %}
         &lt;p&gt;Nobody plays {{ instrument.instrumentName }} :(&lt;/p&gt;
      {% endif %}
{% endfor %}


Note: Using a dot notation to specify the sub-field is only necessary if you have multiple relational fields within a Matrix, but it can't hurt.