Where is links__system_main_menu defined? I searched for links__system_main_menu, but just find usages of this theme. Where is it defined?
Drupal 7 core doesn't actually define this function. 

links__system_main_menu is a theme hook pattern of the form [base hook]__[context]. When links are themed with theme('links__system_main_menu', $vars), theme() will search for and use theme_links__system_main_menu() if it has been defined. If not, it will use theme_links(). 

See theme() for the official documentation of this behavior. Chapter 3 of Drupal 7 Module Development does an excellent job of explaining this and provides several examples.

BTW, the same principle applies to template files. If, for example, we call theme('node__article__1', $vars), theme() will search for a node--article--1.tpl.php file, and then for node--article.tpl.php, finally falling back to node.tpl.php if neither of them are defined.