title: Python search filenames in subfolders using regex and append to list for further use
body: <p>My file system contains a large number of folders, with the folder names in the format of 4 digits. Within each folder are varying numbers of files in different formats, with names corresponding to the folder name, some with letters and/or dates appended to it. What I would like to do is search through each folder and if a year in the format "19??" or "20??" is found, create a folder labelled with that year. If the filename further contains the expression "ED*", create a subfolder with that label. Once all the folders are created, copy all files into their corresponding folders, and strip the names down to the first 6 characters. This file structure will be created on a separate drive.</p>
  
  <p>Example: The folder MAIN contains many subfolders such as 2742, 2830, 3417 etc. The folder 2742 contains the files </p>
  
  <ul>
  <li>2742AB.txt</li>
  <li>2742AB_1999.tif</li>
  <li>2742BB_1999_ED1.txt</li>
  <li>2742AC_1999_ED2_FINAL.txt</li>
  <li>2742CC_2001_ED4.tif </li>
  </ul>
  
  <p>After  running the script, the folder structure should be  as follows:</p>
  
  <ul>
  <li>MAIN\2742\2742AB.txt</li>
  <li>MAIN\2742\1999\2742AB.tif</li>
  <li>MAIN\2742\1999\ED1\2742BB.txt</li>
  <li>MAIN\2742\1999\ED2\2742AC.txt</li>
  <li>MAIN\2742\2001\ED4\2742CC.tif</li>
  </ul>
  
  <p>Using various snippets of code after searching for similar examples on here, I have come up with the following:
      import os
      folder = 'C:\MAIN'
      os.chdir(folder)</p>
  
  <pre><code>dirlist = []
  for dirname, dirnames, filenames in os.walk(folder):
      for subdirname in dirnames:
          dirlist.append(subdirname)
      for filename in filenames:
          dirlist.append(filename)
  </code></pre>
  
  <p>My problem is the next part, i.e amending the above search to check first if the value is already in the list. If it is in the list, continue searching, else check if the filename contains the regex and if it does, add it to the array. In (sort of) psuedocode:</p>
  
  <pre><code>initialise list years
  #search through folders etc
  for filenames in folders
      if filename contains either regex
          for values in years
              if filename exists
                  skip it and continue searching
              else
                  append filename to years
  </code></pre>
  
  <p>I've looked at the various answers on here about searching folders using a regex etc, but I'm not entirely sure if the approach I am using is correct. Any advice and/or comments on the approach I have used would be appreciated.</p>
  
  <p><strong>edit</strong> My edited code, using Michal's suggestions below and modifying it a bit myself:</p>
  
  <pre><code>import os, re, shutil
  tfolder = 'C:\\MAIN'
  os.chdir(tfolder)
  
  re_year19xx = re.compile('(19[0-9][0-9])')
  re_year20xx = re.compile('(20[0-9][0-9])')
  re_ed = re.compile('(ed[0-9]\d+)', re.IGNORECASE)
  destPath = 'C:\\DEST'
  
  filePath, coords = os.path.split(fname)
  coordsFolder = coords[:4]
  coordsFname = coords[:6]
  coordsExt = os.path.splitext(fname)
  year = ' ' #create variable year
  ed = ' ' #create variable ed to store the edition number if necessary
  bname = fname #the original file name
  for re_year in (re_year19xx, re_year20xx):
      rx = re_year.search(fname) #search for regex in the file name and store it in rx
      if rx:
          year = rx.group(1) #if the regex is found, store the year
          print(year)
          bname.replace(year, ' ')
          re_searchEd = re_ed.search(fname)
          if re_searchEd:
              ed = re_searchEd.group(1)
              bname.replace(ed, ' ')
      os.chdir(destPath)  
      fname2 = os.path.join(destPath, coordsFolder,year, ed) + coordsFname + coordsExt[1]
      print('%s -&gt; %s' % (fname, fname2))
      dirn, _ = os.path.split(fname2)
      if not os.path.exists(dirn):
          os.makedirs(dirn)
      shutil.copy(fname, fname2)
  
  for root, dirs, files in os.walk(tfolder):
      for name in files:
          fn = os.path.join(root, name)
          analyse_file_name(fn)
  </code></pre>
  
  <p>Looking at my code again, everything seems to be working correctly until it must create the subfolders i.e. once the <code>coordsFolder</code> has been created, if <code>year</code> is not blank, create subfolders, if <code>ed</code> is not blank, create subfolders, then copy the original files to the respective subfolders and rename. The result of the debug <code>print</code> statement for the folder <code>2742</code> is as follows:</p>
  
  <pre><code>C:\MAIN\2742\2742AB.txt -&gt; C:\DEST\2742\year\ed2742AB.txt
  C:\MAIN\2742\2742AB.txt -&gt; C:\DEST\2742\year\ed2742AB.txt
  1999
  C:\MAIN\2742\2742AB_1999.tif -&gt; C:\DEST\2742\1999\ed2742AB.tif
  C:\MAIN\2742\2742AB_1999.tif -&gt; C:\DEST\2742\1999\ed2742AB.tif
  1999
  C:\MAIN\2742\2742AC_1999_ED2_final.tif -&gt; C:\DEST\2742\1999\ed2742AC.tif
  C:\MAIN\2742\2742AC_1999_ED2_final.tif -&gt; C:\DEST\2742\1999\ed2742AC.tif
  1999
  C:\MAIN\2742\2742BB_1999_ED1.txt -&gt; C:\DEST\2742\1999\ed2742BB.txt
  C:\MAIN\2742\2742BB_1999_ED1.txt -&gt; C:\DEST\2742\1999\ed2742BB.txt
  C:\MAIN\2742\2742CC_2001_ED4.tif -&gt; C:\DEST\2742\year\ed2742CC.tif
  2001
  C:\MAIN\2742\2742CC_2001_ED4.tif -&gt; C:\DEST\2742\2001\ed2742CC.tif
  1999
  </code></pre>
  
  <p><strong>edit</strong> After adding in a check for the value of <code>year</code>:</p>
  
  <pre><code>if year is 'year':
      fname2 = os.path.join(destPath, coordsFolder) + '\\' + coordsFname + coordsExt[1]
  else:
      fname2 = os.path.join(destPath, coordsFolder,year) + '\\' + coordsFname + coordsExt[1]
  </code></pre>
  
  <p>The paths are npw created correctly and the files are being copied over to the correct places and renamed properly! So the last part of the question now applies to the <code>ed</code> subfolders. At the moment, the code is just ignoring the part of the filename that has "ED1", "ED2" etc in it. Where do I place the code to create the subfolders if the value of <code>ed</code> is not the initial value of "ed"></p>