Is there any way to get only the custom roles?
As @stevekeiretsu said in a comment ->

The Role entity doesn't really make any distinction. Looking at the roles in my own db (filter config table by name LIKE user.role%) the only potential contender I can see is that the three default ones have a dependency on _core, so you could check that via getDependencies, but that feels arguably flakier than a static filter to me.

So for Drupal 8 there is no function defined.