title: dynamically skiprows in reading multiple csv files
body: <p>I would like to read a csv file every month from the government census website <a href="https://www.census.gov/econ/currentdata/datasets/" rel="nofollow noreferrer">here</a>, more specifically, the one named VIP-mf.zip. To save your eyes from a cumbersome df, you can download the zip file using <a href="https://www.census.gov/econ/currentdata/datasets/VIP-mf.zip" rel="nofollow noreferrer">this</a> (its 1.5MB)</p>
  <p>I only want to read the csv after the row that says 'DATA', which for this specific file is row 309.
  I can do that easily using:</p>
  <pre><code>import pandas as pd
  df = pd.read_csv('VIP-mf.csv', skiprows=310)
  </code></pre>
  <p>the problem is next month, when the new csv is updated on the website — that <code>skiprows</code> parameter will have to be 311, or else it reads it incorrectly. I would like to have a dynamic <code>skiprows</code> parameter to be able to capture this change every month so I can automatically download and read it correctly.</p>
  <p>I tried implementing a solution from <a href="https://stackoverflow.com/a/63127985/11588364">this answer</a> using <a href="https://towardsdatascience.com/import-csv-files-as-pandas-dataframe-with-skiprows-skipfooter-usecols-index-col-and-header-fbf67a2f92a" rel="nofollow noreferrer">this article</a> by creating a function for the <code>skiprows</code> parameter using the following:</p>
  <pre><code>def fetch_skip(index):
      if index == 'DATA':
          return True
      return False
  df = pd.read_csv('VIP-mf.csv', skiprows= lambda x: fetch_skip(x))
  </code></pre>
  <p>but I get this error:</p>
  <pre><code>ParserError: Error tokenizing data. C error: Expected 4 fields in line 311, saw 7
  </code></pre>
  <p>Which I'm assuming is because the csv has &quot;mini-tables&quot; within a single csv. Even though, I only need the final &quot;table&quot; which has the column names:</p>
  <pre><code>['per_idx', 'cat_idx', 'dt_idx', 'et_idx', 'geo_idx', 'is_adj', 'val']
  </code></pre>
  <p>Thank you for your help.
  P.S If there is another way to do this than fiddling with the <code>skiprows</code> parameter that also works.</p>