How to override the core function format_xml_elements()? format_xml_elements() is defined in the core includes/common.inc file. I want to manipulate some parts of this code, but there is no way to hook into this.

How could I achieve this without hacking core?
You cannot directly make Drupal use a different function instead of format_xml_elements(), but you can replace those functions that (directly or not) calls format_xml_elements().
For example, to alter how the RSS page for a feed is printed, you just need to replace theme_aggregator_page_rss() with a function implemented from your module using hook_theme_registry_alter(); to alter the output for the 'blog/%user/feed, or the blog/feed, or the taxonomy/term/%taxonomy_term/feed, you implement hook_menu_alter() to replace the page callback Drupal uses with your own page callback.

What changes is the code you need to write in each case.
For example, taxonomy_term_feed(), the page callback for taxonomy/term/%taxonomy_term/feed, calls node_feed(), which calls format_rss_item(), which calls format_xml_elements(). In this case, you can use the same code taxonomy_term_feed() uses, except it calls another function (e.g. mymodule_node_fee()) instead of node_feed(), which has the same code, except it doesn't call format_rss_item() nor format_rss_channel(), but your own functions. For the latter two functions, you can use the same code they use, except your code will code your replacement for format_xml_elements() instead of format_xml_elements().

In the case of replacing theme_aggregator_page_rss(), you write less code.