title: Using Python to Convert Fixed Width Hierarchical File to CSV
body: <p>I have a large fixed width file with lines that follow different parsing rules based on the line identifier (FH for File Header, H2 for Header 2...) Here is some sample data:</p>
  
  <pre><code>H20123456789 Tuesday Eastern
  L1John Smith's Apple Store   1234 Wall Street    New York, New York 01234
  D17777777               20191126 2507   0000000000001
  D28888888           6789012             9876520000002
  D17777777               20191126 3424   0000000000002
  D28888888           6789012             9876520000002
  HT00000000000000000000000000000000000000000000000000000001
  L1Mark Sloan's Flower Store  1235 23rd Street    Dallas,   Texas    01234
  D17777777               20191126 1307   0000000000003
  D26666666           6789012             9876520000002
  HT00000000000000000000000000000000000000000000000000000001
  FT0382927439297843+099876333453
  </code></pre>
  
  <p>Using the line identifier (FH,H2,L1,D1,D2,HT,FT) I would like to ignore the FH, HT and FT lines, and combine all detail lines with their associated header line and convert this data into a csv. I would like the output text to follow the pattern:</p>
  
  <pre><code>H2,L1,D1,D2
  </code></pre>
  
  <p>For the sample data above, I would like the output to be:</p>
  
  <pre><code>0123456789 Tuesday Eastern,John Smith's Apple Store,1234 Wall Street,New York,New York,01234,7777777,20191126,2507,0000000000001,8888888,6789012,9876520000002
  0123456789 Tuesday Eastern,John Smith's Apple Store,1234 Wall Street,New York,New York,01234,7777777,20191126,3424,0000000000002,8888888,6789012,9876520000002
  0123456789 Tuesday Eastern,Mark Sloan's Flower Store,1235 23rd Street,Dallas,Texas,01239,7777777,20191126,1307,0000000000003,6666666,6789012,9876520000002
  </code></pre>
  
  <p>I have am new to Python, but I have searched stack overflow and found ways to identify lines based on their characters using this code:</p>
  
  <pre><code>import re  # import RegEX
  
  lists = []  # create an array called lists
  linenum = 0  # start line number count at 0
  pattern = re.compile(
      r"^D1"
  )  # define the RegEX patter we are looking for, ^ means line must start with
  with open("readthis.txt", "rt") as myfile:  # open the text file in read mode
      for line in myfile:  # for each line in this file
          if pattern.search(line) != None:  # If the pattern is found above
              lists.append(
                  (linenum, line.rstrip("\n"))
              )  # list the full line and remove any extra whitespace between lines
  for sub in lists:  # Iterate over the list of tuples
      print(sub[1])  # print the lines that match
  </code></pre>
  
  <p>This finds and prints the rows I need, but it doesn't help with parsing. I also am using this code to parse out a file based on the "config.txt" file, but I can't find a way to store lines as variables and carry the L1 records through the file:</p>
  
  <pre><code>"""
  This script will convert Fixed width File into Delimiter File, tried on Python 3.5 only
  Sample run: (Order of argument doesnt matter)
  python ConvertFixedToDelimiter.py -i SrcFile.txt -o TrgFile.txt -c Config.txt -d "|"
  Inputs are as follows
  1. Input FIle - Mandatory(Argument -i) - File which has fixed Width data in it
  2. Config File - Optional (Argument -c, if not provided will look for Config.txt file on same path, if not present script will not run)
      Should have format as
      FieldName,fieldLength
      eg:
      FirstName,10
      SecondName,8
      Address,30
      etc:
  3. Output File - Optional (Argument -o, if not provided will be used as InputFIleName plus _Delimited.txt)
  4. Delimiter - Optional (Argument -d, if not provided default value is "," (comma))
  """
  from collections import OrderedDict
  import argparse
  from argparse import ArgumentParser
  import os.path
  import sys
  
  # Define how to parse the file
  def slices(s, args):
      position = 0
      for length in args:
          length = int(length)
          yield s[position:position + length]
          position += length
  
  # Check to make sure argparse is loaded
  def extant_file(x):
      """
      'Type' for argparse - checks that file exists but does not open.
      """
      if not os.path.exists(x):
          # Argparse uses the ArgumentTypeError to give a rejection message like:
          # error: argument input: x does not exist
          raise argparse.ArgumentTypeError("{0} does not exist".format(x))
      return x
  
  # Define arguments
  parser = ArgumentParser(description="Please provide your Inputs as -i InputFile -o OutPutFile -c ConfigFile")
  parser.add_argument("-i", dest="InputFile", required=True,    help="Provide your Input file name here, if file is on different path than where this script resides then provide full path of the file", metavar="FILE", type=extant_file)
  parser.add_argument("-o", dest="OutputFile", required=False,    help="Provide your Output file name here, if file is on different path than where this script resides then provide full path of the file", metavar="FILE")
  parser.add_argument("-c", dest="ConfigFile", required=False,   help="Provide your Config file name here,File should have value as fieldName,fieldLength. if file is on different path than where this script resides then provide full path of the file", metavar="FILE",type=extant_file)
  parser.add_argument("-d", dest="Delimiter", required=False,   help="Provide the delimiter string you want",metavar="STRING", default=",")
  
  # Use arguments defined above
  args = parser.parse_args()
  
  #Input file madatory
  InputFile = args.InputFile
  #Delimiter by default ","
  DELIMITER = args.Delimiter
  
  #Check to see if an Output file name has been specified, if not default to filename + _Delimited
  if args.OutputFile is None:
      OutputFile = str(InputFile) + "_Delimited.txt"
      print ("Setting Ouput file as "+ OutputFile)
  else:
      OutputFile = args.OutputFile
  
  # Check to make sure the Config file exists in the same directory as the python script
  if args.ConfigFile is None:
      if not os.path.exists("Config.txt"):
          print ("There is no Config File provided exiting the script")
          sys.exit()
      else:
          ConfigFile = "Config.txt"
          print ("Taking Config.txt file on this path as Default Config File")
  else:
      ConfigFile = args.ConfigFile
  
  # Create an array for field names, field length
  fieldNames = []
  fieldLength = []
  myvars = OrderedDict()
  
  # Read the configuration file and define the variables in the file in the arrays created above
  with open(ConfigFile) as myfile:
      for line in myfile:
          name, var = line.partition(",")[::2]
          myvars[name.strip()] = int(var)
  for key,value in myvars.items():
      fieldNames.append(key)
      fieldLength.append(value)
  
  # Create the output file
  # Use the field length to slice the input file and join the sliced pieces with commas in new lines in the output file
  with open(OutputFile, 'w') as f1:
      fieldNames = DELIMITER.join(map(str, fieldNames))
      f1.write(fieldNames + "\n")
      with open(InputFile, 'r') as f:
          for line in f:
              rec = (list(slices(line, fieldLength)))
              myLine = DELIMITER.join(map(str, rec))
              f1.write(myLine + "\n")
  </code></pre>