title: Concatenate multiple CSV files in a windows machine by running a shell command through python
body: <p>Objective: Concatenate multiple CSV files given some specific conditions.</p>
  <ul>
  <li>Conditions: Windows 10 machine + ANY shell command (e.g., <code>awk</code>) + Python &lt;= 3.7.</li>
  <li>Disclaimer: I know that are plenty of ways of doing this which includes: using pandas dataframe concatenation; OR using python's <code>shutil</code>; OR using system command (e.g., <code>awk</code>) directly without python on a shell. But again, I want an answer that fits these exact above conditions.</li>
  </ul>
  <p>Here is a shell command example (which works if I run directly on a shell) that I am currently trying to run</p>
  <pre><code>$ awk &quot;(NR == 1) || (FNR &gt; 1)&quot; &quot;..\\results\\raw*.csv&quot; &gt; &quot;..\\results\\consolidated.csv&quot;
  </code></pre>
  <p>And this is how I run it through python's <code>subprocess</code>:</p>
  <pre class="lang-py prettyprint-override"><code>src_files = os.path.join('..', 'results', 'raw*.csv')
  dest_file = os.path.join('..', 'results', 'consolidated.csv')
  
  result = subprocess.run('awk &quot;(NR == 1) || (FNR &gt; 1)&quot; &quot;{}&quot; &gt; &quot;{}&quot;'.format(src_files, dest_file),
                          shell=True,
                          capture_output=True)
  </code></pre>
  <p>However, I keep getting the following error:</p>
  <pre class="lang-py prettyprint-override"><code>print(result.stderr)
  
  b&quot;awk: fatal: cannot open file `..\\results\\raw*.csv' for reading (No such file or directory)\n&quot;
  </code></pre>
  <p>I should point out that if I run this python code on a Unix machine it works (if the quotes around the format <code>{}</code> are removed). The python code just not works on a Windows machine.</p>
  <p>Thus, I believe this may have something to with properly escaping the command string because when I specify the name of some files, this command works. Still, I can't manage to find the proper way of escaping the <code>*</code> symbol and make all of this work.</p>
  <p>PS: Generating the data for a working example</p>
  <pre class="lang-py prettyprint-override"><code>for i in range(1, 4):
      pd.DataFrame([[i*1, i*2, i*3]], columns=['a', 'b', 'c']).to_csv(os.path.join('results', 'raw-{}.csv'.format(i)), index=False, sep=';')
  </code></pre>
  <p>PS2: Employed folder structure</p>
  <pre><code>|-- script_folder/
  |   |-- consolidation_script.py
  |-- results/
  |   |-- raw-1.csv
  |   |-- raw-2.csv
  |   |-- raw-3.csv
  </code></pre>