Processing file inclusion directives using mutual recursion I am preprocessing a set of files where one file is included in another file with an include tag like shown below:

file A

include file B
include file C
 contents of file A
include file D
 contents of file A


Here I need to replace the include tags with contents of each of the respective files, just like what a compiler does. I have two functions for that:

def parseContent(self, rdbFilePath, content):
    finalContent = self.removeComments(content)
    includeFileSearch = re.compile(r'(?P&lt;tag&gt;(\s)*include+(\s*)\"(\s*)(?P&lt;file&gt;[a-zA-Z0-9\.\_/]*)\")')
    for includes in includeFileSearch.finditer(finalContent):
        finalContent = re.sub(includes.group('tag'),self.parseIncludes(rdbFilePath, includes.group('file')), finalContent)
    return finalContent

def parseIncludes(self, rdbFilePath, file):
    path = rdbFilePath + "/" + file
    f = open(path)
    pathDir = os.path.dirname(path)
    includedFileContent = self.parseContent(pathDir, f.read())
    return includedFileContent


As you can see, functions parseContent and parseIncludes call each other recursively to replace all the include tags in every file. The logic works fine. But it takes a bit long time to execute. Is there any better way to do the same with lesser execution time?
parseContent() calls both re.finditer() and re.sub(), which is inefficient, because the re.sub() has to find the insertion point that re.finditer() had already found. What you want is just re.sub():


  repl can be a string or a function… If repl is a function, it is called for every non-overlapping occurrence of pattern. The function takes a single match object argument, and returns the replacement string.


Going further, you should make just one substitution pass that handles both comments and includes. (You didn't show us the removeComments() code, so you'll have to adapt the regex yourself.)

The double-quotes in the regex have no special significance, and do not need to be escaped with backslashes. Also, the dot and the underscore characters do not need to be escaped within a character class.

DIRECTIVE_RE = re.compile(r'(?P&lt;comment&gt;#.*)|(?P&lt;include&gt;(\s)*include+(\s*)"(\s*)(?P&lt;file&gt;[a-zA-Z0-9._/]*)")')

def process_content(self, rdb_file_path, content):
    def directive_handler(match):
        if match.group('comment'):
            return ''
        elif match.group('include'):
            path = os.path.join(rdb_file_path, match.group('file'))
            with open(path) as f:
                return self.process_content(rdb_file_path, f.read())
    return DIRECTIVE_RE.sub(directive_handler, content)


open() should almost always be called using a with block to ensure that the filehandle gets closed automatically.

Note that I have used os.path.join() as a portable alternative to + "/" +.