title: Python Generic Data Engine
body: <p>I have been working on Python for about 1.5yrs and looking for some direction. This is the first time I can't find what I need after doing a lot of searching and must be missing something- most likely searching the wrong terms.</p>
  
  <p><strong>Problem</strong>: I am working on an app that has many processes (Could be hundreds or even thousands). Each process may have a unique input and output data format - could be <em>multiline strings</em>, <em>comma separated strings</em>, <em>excel</em> or <em>csv</em> with or without varying headers and many others. I need something that will format the input correctly and handle the output based upon the process. New processes also need to be easily added/defined. I am open to whatever is the best approach, but my thoughts are to use a database that stores the template/data definition and use that to know the format given a process. However, I'm struggling to come up with exactly how, if this is really the best approach, but it needs to be a solution that is scalable. Any direction would be appreciated. Thank you.</p>
  
  <p>A couple simple examples of data</p>
  
  <p><strong>Process 1 example data (multi line string with Header)</strong>
  Input of</p>
  
  <pre class="lang-py prettyprint-override"><code> [ABC123, XYZ453, CDE987]
  </code></pre>
  
  <p>and the resulting data input below would be created:</p>
  
  <pre><code>Barcode
  ABC123
  XYZ453
  CDE987
  </code></pre>
  
  <p>This code below works, but is not reusable for the example 2. </p>
  
  <pre class="lang-py prettyprint-override"><code>list = [ABC123, XYZ453, CDE987]
  
  input = "Barcode /r/n"
  
  for l in list:
      input = input + l + '/r/n'
  </code></pre>
  
  <p><strong>Process 2 example input template (comma separated with Header):</strong></p>
  
  <pre><code>Barcode,Location,Param1,Param2
  Item1,L1,11,A
  Item1,L1,22,B
  Item2,L1,33,C
  Item2,L2,44,F 
  Item3,L2,55,B
  Item3,L2,66,P
  </code></pre>
  
  <p><strong>Process 2 example resulting input data (comma separated with Header):</strong>
  Input of </p>
  
  <pre class="lang-py prettyprint-override"><code>{'Barcode':['ABC123', 'XYZ453', 'CDE987', 'FGH487', 'YTR123'], 'Location':['Shelf1', 'Shelf2']}
  </code></pre>
  
  <p>and using the template to create the input data below:</p>
  
  <pre><code>Barcode,Location,Param1,Param2
  ABC123,Shelf1,11,A
  ABC123,Shelf1,22,B
  XYZ453,Shelf1,33,C
  XYZ453,Shelf2,44,F 
  CDE987,Shelf2,55,B
  CDE987,Shelf2,66,P
  FGH487,Shelf1,11,A
  FGH487,Shelf1,22,B
  YTR123,Shelf1,33,C
  YTR123,Shelf2,44,F 
  </code></pre>
  
  <p>I know how to handle each process with hardcoded loop/dataframe merge, etc. Ive done some abstraction in other cases with dicts. However, how to define/store each format that vary so much and create reusable abstracted code is where I am stuck. </p>