Overriding core forum function is there any clean way to override _forum_topics_unread function call from template_preprocess_forum_list without hacking the core?
No, hardcoded function calls can not be replaced.

The only thing you can do is add your own preprocess function for "forum_list "(yourmodule_preprocess_forum_list()), and then call another function to overwrite the value returned by the function.