title: How to use awk and grep on 300GB .txt file?
body: <p>I have a huge .txt file, 300GB to be more precise, and I would like to put all the distinct strings from the first column, that match my pattern into a different .txt file.</p>
  
  <pre><code>awk '{print $1}' file_name | grep -o '/ns/.*' | awk '!seen[$0]++' &gt; test1.txt
  </code></pre>
  
  <p>This is what I've tried, and as far as I can see it works fine but the problem is that after some time I get the following error:</p>
  
  <pre><code>awk: program limit exceeded: maximum number of fields size=32767
      FILENAME="file_name" FNR=117897124 NR=117897124
  </code></pre>
  
  <p>Any suggestions?</p>