Why does my Twig for loop not get a Key?
I'm guessing that you'd like {{ categoryTitle }} to be the current groups documentType.title? 

If so, I would output document.documentType for the first document in each group using {% if loop.first %}.

Complete code suggestion would be:

{% set grouped = entry.documents|group('documentType.first().title') %}

{% for groupedRows in grouped %}
  {% for document in groupedRows %}
    {% if loop.first %}
      {{ document.documentType }}
    {% endif %}
    {{ document.documentTitle }}
  {% endfor %}
{% endfor %}


EDIT: updated to use carlcs group filter suggestion.