Python: Iterative subdirectory line count
Question: I'm attempting to count the number of lines in two separate directories that
are formatted differently and compare the results. The text files needed are
all within a single directory in the the first dir, but the second one has
subdirectories that I need to iterate through, save the name of, and then pull
and count all txt files that begin with the name.

At the moment, I'm having trouble comparing the subdirectory name with the txt
files that begin with it. My traceback is as follows:

    
    
    be29X1(149)% ./SeriesCount.py
    Traceback (most recent call last):
      File "./SeriesCount.py", line 23, in <module>
        for fn in files('subdir_name*.txt'):
    TypeError: 'list' object is not callable
    

I don't need to permanently save the subdirectory name, since all I care about
is storing the txt file names and their counts to a dict. For example, if the
directory name is "regprices", I want to pull out the line counts for all text
files within the directory that begin with "regprices". Code is below:

    
    
    #!/usr/bin/env python
    
    import csv
    import copy
    import os
    import sys
    import glob
    import dircmp
    
    #set dicts 
    dict1 = {}
    dict2 = {}
    final_dict = {}
    
    #parses through directory 1, counts lines, saves to a dict
    for fn in glob.glob('/data/*.txt'):
        with open(fn) as f:
            dict1[fn] = [1 for line in f if line.strip() and not line.startswith('#')]
    
    #parses through subdirectories in directory 2, counts lines, saves to a dict
    for subdir, dirs, files in os.walk('/docs/prod/count/'):
        subdir_name = os.getcwd()
        for fn in files('subdir_name*.txt'):
            dict2[fn] = [1 for line in f if line.strip() and not line.startswith('#')]
    
    #compare dicts, overwrite counts from dict1 with dict2, save to final dict
    
    save final dictionary with key/val pairs to a csv
    with open('seriescount.csv', 'w') as f:
        w = csv.DictWriter(f, final_dict) 
        w.writeheader()
        w.writerow({k:sum(v) for k, v in final_dict.items()})
    

Bonus points if you can help with syntax comparing the two dictionaries,
overwriting the counts from dir2 to dir1, and saving them to final_dict


Answer: Example is all over the place with differently named variables. Definitely not
a working example. It's hard to work out what you're trying to achieve.

Not sure how you're trying to compare the dictionary keys based on filename.
Here's an attempt at guessing what you're trying to achieve.

    
    
    import glob
    import os
    
    
    def count_lines(filename):
      with open(filename,'r') as f:
        count = sum(1 for line in f if line.strip() and line[0] != '#')
      return count
    
    def directory1_count(path='/data/*.txt'):
      counts = {}
      for filepath in glob.glob(path):
        directory, filename = os.path.split(filepath)
        name, extension = os.path.splitext(filename)
        counts[name] = count_lines(filepath)
      return counts
    
    def directory2_count(path='/docs/prod/count/'):
      counts = {}
      for directory, dirs, files in os.walk(path):
        _, subdir = os.path.split(directory)
        for filename in [x for x in files if x.startswith(subdir) and x.endswith('.txt')]:
          name, extension = os.path.splitext(filename)
          filepath = os.path.join(directory,filename)
          counts[name] = count_lines(filepath)
      return counts
    
    counts = directory1_count()
    counts.update(directory2_count())
    

