Reading a file line by line - Two tokens per entry Problem

Read a file and capture two tokens for every multi-line entry.

First token is hexcode with which the entry starts.

Second token is the remaining line(multi-line) of the entry. Trailing backslash indicates the continuation of line.

Typical entries of a file is as shown below:

0x0512ff01 R 5330 R XY.Zion, \
        "parseexp( {f 7},{S \"OWAUTO\"})", "0x0222ff02 -:-", \
        "parseexp( {g 7},{S \"TWFAB\"})", "0x0222ff02 -:-",  \
        "setfault", "0x0222ff03 -:-"
# whatever
0x0436ff02 I 5330  R XY.Yion, " \
                    ! ( strcat( {H 19.19.121.180}, {attr 0x11137f })  \
                            "default", "0x0222ff03 -:-"
0x0236ff03 G 50
0x0588ff03 H 49
0x0777ff03 R 34


There are such ~100 files to read



Solution

Input to this function is a generator that provides path, one at a time

def get_tokens(file_paths):

    for path in file_paths:
        ruleCapture = False
        with open(path) as f:

            for line in f:

                line=line.strip() # remove \n and other spaces
                if line.startswith('#'): # comments
                    continue
                elif (not line.endswith('\\')) and (ruleCapture == False):
                    event_rule_match =  re.search(r'(?&lt;!\\\n)^(0[xX][0-9a-fA-F]+)([^\n]*)',line)
                    if event_rule_match:
                        yield event_rule_match.group(1), event_rule_match.group(2)
                elif line.endswith('\\') and ruleCapture == False:
                    buffer = line[0:-2]
                    ruleCapture = True
                elif line.endswith('\\') and ruleCapture == True:
                    buffer = ' '.join([buffer, line[0:-2]])
                elif (not line.endswith('\\')) and ruleCapture == True:
                    buffer = ' '.join([buffer, line])
                    ruleCapture = False
                    event_rule_match =  re.search(r'(?&lt;!\\\n)^(0[xX][0-9a-fA-F]+)([^\n]*)',buffer)
                    yield event_rule_match.group(1), event_rule_match.group(2)
                    buffer=''




1) Can this code avoid regex?

2) Can elif structure still be optimized?

3) Can this code avoid reading the file line by line? Instead use f.read() and compile regex once....
1) Control flow clarity

if a and b: ...
elif a and not b: ...
elif not a and b: ...
elif not a and not b: ...


can be improved to

if a:
    if b: ...
    else: ...
else:
    if b: ...
    else: ...


2) Remove unnecessary function calls

' '.join([a, b]) can be simplified to a + ' ' + b. In your case, since you're appending b, you can do a += ' ' + b.

3) Implicit slicing notation

line[0:-2] can be line[:-2].

4) Consolidate constants

The regex string is a constant and is used multiple times. It can be moved to the top as a constant (or even outside the function).

5) Remove unnecessary comments

Assume the reader is familiar with built-in functions like str.split(). Thus, # remove \n and other spaces is unnecessary.

6) Readability


Include exactly one space before and after an assignment. e.g line = line.strip().
No need for if a == True or if a == False. Instead, do if a or if not a.
Remove unnecessary line breaks near the top.
Add spaces after commas in a function's parameters. e.g: f(a, b) instead of f(a,b).
Pick either camelCase or under_scores for variable names. Don't use both.


Final result:

def get_tokens(file_paths):
    REGEX = r'(?&lt;!\\\n)^(0[xX][0-9a-fA-F]+)([^\n]*)'

    for path in file_paths:
        rule_capture = False
        with open(path) as f:
            for line in f:
                line = line.strip()
                if line.startswith('#'): # ignore comments
                    continue

                if line.endswith('\\'):
                    if rule_capture:
                        buffer += ' ' + line[:-2]
                    else:
                        buffer = line[:-2]
                        rule_capture = True
                else:
                    if rule_capture:
                        buffer += ' ' + line
                        rule_capture = False
                        event_rule_match = re.search(REGEX, buffer)
                        yield event_rule_match.group(1), event_rule_match.group(2)
                        buffer = ''
                    else:
                        event_rule_match = re.search(REGEX, line)
                        if event_rule_match:
                            yield event_rule_match.group(1), event_rule_match.group(2)


Notice I didn't change any of the logic, but just moved things around or removed things. See if you can simplify the logic (e.g. removing unnecessary assignments, etc).