How do I sort the list of modules by the number of open issues? Is there any option to sort the list of modules on drupal.org by the number of issues?
No, there isn't.

The number of issues isn't really relevant because you need to see what type of issues there are (cosmetic / blocking), how actively the module is maintained, how many maintainers, etc.

Big modules can have a lot of issues, but if they are frequently updated it's no problem. If a module has only one (blocking) issue but it isn't resolved due to lack of maintenance than it's worse.