Scan files and pick overlapped hex codes Problem

Two folders having files with nameEfile that store lines starting with hex codes.

Print those hex codes that overlap in both folders.



Solution

import os
import sys
import re

def process_cache(cache):
    event_code_set = set()
    for file_name, multi_line_content in cache.items():
        if file_name.endswith('Efile'):
            for line in multi_line_content.splitlines():
                line = line.rstrip('\\') # trailing backslash, if exist
                if bool(re.search(r'^0[xX][0-9a-fA-F]+', line)):
                    # Take the hexcode
                    obj = re.search(r'^0[xX][0-9a-fA-F]+', line)
                    event_code_set.add(obj.string[obj.start():obj.end()])
    return event_code_set



def scan_files(dir):
    cache = {}
    for root, dirs, files in os.walk(dir):
        for name in files:
            if name in ('Efile'):
                path = os.path.join(root, name)
                with open(path,'r') as file:
                    cache[path] = file.read()
    return cache

cache1 = scan_files(sys.argv[1])
cache2 = scan_files(sys.argv[2])

cache1_event_code_set = process_cache(cache1)
cache2_event_code_set = process_cache(cache2)

overlap_event_codes = cache1_event_code_set &amp; cache2_event_code_set 
print(overlap_event_codes)




For a typical three entries in a file, with comments(#),


0x00010d35 D 11 G 3,0x10009,N R XY.Condition,   "({a 0x40001} == {H 0x166}) &amp;&amp; ({a 0x11ff8} == {I 15})","0x0763ffc2 "
# Below event codes are from vendor xyz
0x84900c5 M 22 Y 1,0x03330469,4,5,6,7,8
0x04b60ff6 L 50 U \
          0x04c60e07,102 &amp;&amp; ({a 0x11ff8} == {I 15})","0x0763ffc2 "



Picking 0x00010d35, 0x04b60ff6 &amp; 0x84900c5 is the task. Rest of the line is supposed to be ignored

Some entries are multi-line with trailing backslash.

Each file is in megabytes. Total files in both folders - 80



1) Please suggest optimization in the below code, because pattern check is done twice.

        if bool(re.search(r'^0[xX][0-9a-fA-F]+', line)):
            # Take the hexcode
            obj = re.search(r'^0[xX][0-9a-fA-F]+', line)
            event_code_set.add(obj.string[obj.start():obj.end()])


2) Please suggest coding style optimizations for tree walk, cache and command line arguments.
Repeating the match can be avoided by storing it in a variable first:

match = re.search(r'^0[xX][0-9a-fA-F]+', line)
if match:
    event_codes.add(match.group())


Note also that you don't need to strip a trailing \, because it is ignored afterwards anyways and you are only interested in hex codes at the beginnings of the line and all your multilines seem to have some whitespace at the beginning of the continued line. Note that you could have used re.match(r'0[xX][0-9a-fA-F]+', line), instead of re.search, because it always matches only the beginning of the line.

Python also caches your regex, once you have used it, but you may still squeeze out a bit of performance with re.compile.

I also used the match.group() method, which immediately returns the matching part if the string.



I would also not suggest caching the file content of all the files in the directories. This seems like a huge waste of memory. Instead process them one at a time and add it to your event code sets, or even better, just generate a stream of event codes that you can consume with set:

import os
import sys
import re

def event_codes_file(path):
    with open(path) as file:
        for line in file:
            match = re.match(r'0[xX][0-9a-fA-F]+', line)
            if match:
                yield match.group()

def event_codes_dir(dir):
    for root, dirs, files in os.walk(dir):
        for name in files:
            if name.endswith('Efile'):
                path = os.path.join(root, name)
                yield from event_codes_file(path)

if __name__ == "__main__":
    event_codes1 = set(event_codes_dir(sys.argv[1]))
    event_codes2 = set(event_codes_dir(sys.argv[2]))

    overlap_event_codes = event_codes1 &amp; event_codes2
    print(overlap_event_codes)


yield from x was introduced in Python 3.3 and is mostly equivalent to for i in x: yield i.

Also note that name in ('Efile') is almost the same as name == 'Efile', but matches a bit more (like 'E'). Unless of course you meant what you wrote in process_cache, name.endswith('Efile'). I am assuming the latter.

On a file "test", containing your example content, this produces:

set(event_codes_file("test"))
# {'0x00010d35', '0x04b60ff6', '0x84900c5'}