How to break up user roles checkboxes into groups?
One way to do this would be to use some JS to add a header element in front of the role that starts the group.  Each role has a unique id, e.g. input#edit-roles-2.  You could use some jQuery like:

$("input#edit-roles-2").before("&lt;h3&gt;My Group&lt;/h3&gt;");


You would have to order the roles the way you want and find the right ids.

Also, this will probably require using a custom admin theme or a module in order to inject the JS since user editing generally uses that theme and not the main theme.