title: Is there any way to redirect stdout to a parquet file on bash?
body: <p>I want to generate a parquet file for later processing with Spark. This parquet consists in a list of all nested files (json or xml) within a certain root folder together with their corresponding timestamps and paths, and excluding some folders from search.</p>
  <p>So far I was generating it with the following python script:</p>
  <pre><code>import subprocess
  from io import StringIO
  
  import pandas as pd
  
  PATH = input(&quot;Enter path: &quot;) or &quot;.&quot;
  
  proc = subprocess.Popen(
      'find %(path)s \( -path &quot;%(path)s/venv/*&quot; -o -path &quot;%(path)s/_forms/*&quot; -o -path &quot;%(path)s/failed/*&quot; -o -path &quot;%(path)s/logs/*&quot; -o -path &quot;%(path)s/OLD/*&quot; \) -prune -o \( -type f -name *.xml -o -name *.json \) -exec du --time {} +'
      % {&quot;path&quot;: PATH},
      shell=True,
      stdout=subprocess.PIPE,
  )
  buff = StringIO(proc.communicate()[0].decode())
  
  df = pd.read_csv(buff, sep=&quot;\t&quot;, header=None, names=[&quot;size&quot;, &quot;timestamp&quot;, &quot;path&quot;])
  df.to_parquet(&quot;paths.parquet&quot;)
  </code></pre>
  <p>The problem with this approach is that, when output is very big, process eventually is killed (I suppose due to OOM issues when buffering), so I've decided to move it to a pure bash script. It is something like:</p>
  <pre><code>cat &lt;(echo -e 'size\ttimestamp\tpath') &lt;(find . \( -path &quot;./venv/*&quot; -o -path &quot;./_forms/*&quot; -o -path &quot;./failed/*&quot; -o -path &quot;./logs/*&quot; -o -path &quot;./OLD/*&quot; \) -prune -o \( -type f -name *.xml -o -name *.json \) -exec du --time {} +)
  </code></pre>
  <p>But this way how can I construct a valid parquet compressed file as in python-pandas approach? Can I just redirect it to <code>&gt; paths.parquet</code> or an external package is needed for that?</p>