title: How to structure the code of a python data pre-processing pipeline (for production)
body: <p>I have a large dataset with a defined structure of columns for which I have built a script/pipeline that generally does: <strong>first</strong>, ingests the data (formatting, cleaning, etc.), and <strong>second</strong>, it transform values and creates a new column with these new transformed values (final result), more or less like this:</p>
  
  <ol>
  <li>Imports csv into pandas framework, fills nans, cleans some values in some columns, homogenizes text, names, etc.
  1.1. Creates a new column (cleaned names)</li>
  <li>Transform/convert values in another column via look ups in dictionaries, doing groupbys, etc. 
  2.2 Creates 1 new column (transformed values)</li>
  </ol>
  
  <p>My script is divided into two files (~150 lines of code, each) and is composed of many methods: <code>.where, .replace, .map, .apply, .etc</code>. Given that pandas allows for method chaining and is very flexible, the dataset can be processed without defining any function (except a few for ) <code>df.apply(func)</code>. My code gets the csv into a df and naturally starts processing it with the mentioned methods <code>.where, .replace, .map, .apply, .etc</code> without using any function or a <code>.pipeline</code> method. My project tree looks like:</p>
  
  <pre><code>/project
      table.csv
      ingest.py (outputs a clean intermediate_table.csv)
      transform.py (reads previous table.csv and outputs a final_table.csv)
      final_table.csv
  </code></pre>
  
  <p>The thing is, I need to send this code over to other people who will run my script in more datasets, so I will need to comment and <strong>test</strong> it. Given the above, here are my questions in terms of the code structure.</p>
  
  <ul>
  <li>Should I have a function for each of the steps above?</li>
  <li>If so, with what granularity? </li>
  </ul>
  
  <p>E.g. Should I have multiple functions like below?:</p>
  
  <pre><code>df = pd.read_csv('file.csv')
  
  def uppercase_column_A(dataframe, col) 
  def clean_column(dataframe, col) 
  def calculate_mean_here(dataframe, col)
  def transform_values_there(dataframe, col)
  
  df
  .pipe(uppercase_column_A)
  .pipe(clean_column)
  .pipe(calculate_mean_here)
  .pipe(transform_values_there)
  .pipe(etc)
  )
  
  </code></pre>
  
  <p>or, maybe, just two big functions ?</p>
  
  <pre><code>df = pd.read_csv('file.csv')
  def ingest(df): returns intermediate_df
  def transform(intermediate_df)
  
  df
  .pipe(ingest)
  .pipe(transform)
  </code></pre>
  
  <ul>
  <li>Do I actually need to use .pipe? at all</li>
  <li>Should I use classes? separate into modules?</li>
  </ul>
  
  <p>I know the question is broad but I think common practices are important as well as the code itself. In academia (my background), this does not matter much as there is not a 'production' side. So, in general, what would be a recommended industry-way of building data pipelines in terms of code/structure?</p>