Reading a file line by line - Two tokens per entry
1) Control flow clarity

if a and b: ...
elif a and not b: ...
elif not a and b: ...
elif not a and not b: ...


can be improved to

if a:
    if b: ...
    else: ...
else:
    if b: ...
    else: ...


2) Remove unnecessary function calls

' '.join([a, b]) can be simplified to a + ' ' + b. In your case, since you're appending b, you can do a += ' ' + b.

3) Implicit slicing notation

line[0:-2] can be line[:-2].

4) Consolidate constants

The regex string is a constant and is used multiple times. It can be moved to the top as a constant (or even outside the function).

5) Remove unnecessary comments

Assume the reader is familiar with built-in functions like str.split(). Thus, # remove \n and other spaces is unnecessary.

6) Readability


Include exactly one space before and after an assignment. e.g line = line.strip().
No need for if a == True or if a == False. Instead, do if a or if not a.
Remove unnecessary line breaks near the top.
Add spaces after commas in a function's parameters. e.g: f(a, b) instead of f(a,b).
Pick either camelCase or under_scores for variable names. Don't use both.


Final result:

def get_tokens(file_paths):
    REGEX = r'(?&lt;!\\\n)^(0[xX][0-9a-fA-F]+)([^\n]*)'

    for path in file_paths:
        rule_capture = False
        with open(path) as f:
            for line in f:
                line = line.strip()
                if line.startswith('#'): # ignore comments
                    continue

                if line.endswith('\\'):
                    if rule_capture:
                        buffer += ' ' + line[:-2]
                    else:
                        buffer = line[:-2]
                        rule_capture = True
                else:
                    if rule_capture:
                        buffer += ' ' + line
                        rule_capture = False
                        event_rule_match = re.search(REGEX, buffer)
                        yield event_rule_match.group(1), event_rule_match.group(2)
                        buffer = ''
                    else:
                        event_rule_match = re.search(REGEX, line)
                        if event_rule_match:
                            yield event_rule_match.group(1), event_rule_match.group(2)


Notice I didn't change any of the logic, but just moved things around or removed things. See if you can simplify the logic (e.g. removing unnecessary assignments, etc).