Cannot use my drupal installation after enabling new custom theme
theme.inc on line 98 is trying to collect theme's ancestors.

My bet is that your custom theme is wrong and it's telling Drupl that is a subtheme of itself, making Drupal the enter in an infinte loop that exhausts memory.

Double check your custom theme configuration or just move it away from Drupal directories so Drupal can't process it.