title: How can I make this PowerShell script parse large files faster?
body: <p>I have the following PowerShell script that will parse some very large file for <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load" rel="nofollow noreferrer">ETL</a> purposes. For starters my test file is ~ 30&nbsp;MB. Larger files around 200&nbsp;MB are expected. So I have a few questions.</p>
  
  <p>The script below works, but it takes a very long time to process even a 30&nbsp;MB file.</p>
  
  <h2>PowerShell Script:</h2>
  
  <pre><code>$path = "E:\Documents\Projects\ESPS\Dev\DataFiles\DimProductionOrderOperation"
  $infile = "14SEP11_ProdOrderOperations.txt"
  $outfile = "PROCESSED_14SEP11_ProdOrderOperations.txt"
  $array = @()
  
  $content = gc $path\$infile |
      select -skip 4 |
      where {$_ -match "[|].*[|].*"} |
      foreach {$_ -replace "^[|]","" -replace "[|]$",""}
  
  $header = $content[0]
  
  $array = $content[0]
  for ($i = 1; $i -le $content.length; $i+=1) {
      if ($array[$i] -ne $content[0]) {$array += $content[$i]}
  }
  
  $array | out-file $path\$outfile -encoding ASCII
  </code></pre>
  
  <h2>DataFile Excerpt:</h2>
  
  <pre><code>---------------------------
  |Data statistics|Number of|
  |-------------------------|
  |Records passed |   93,118|
  ---------------------------
  02/14/2012                                                                                                                                                           Production Operations and Confirmations                                                                                                                                                              2
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Production Operations and Confirmations
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  |ProductionOrderNumber|MaterialNumber                       |ModifiedDate|Plant|OperationRoutingNumber|WorkCenter|OperationStatus|IsActive|     WbsElement|SequenceNumber|OperationNumber|OperationDescription                    |OperationQty|ConfirmedYieldQty|StandardValueLabor|ActualDirectLaborHrs|ActualContractorLaborHrs|ActualOvertimeLaborHrs|ConfirmationNumber|
  |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  |180849518            |011255486L1                          |02/08/2012  |2101 |            9901123118|56B30     |I9902          |        |SOC10MA2302SOCJ31|              |0140           |Operation 1                             |          1 |               0 |              0.0 |                    |                499.990 |                      |        9908651250|
  |180849518            |011255486L1                          |02/08/2012  |2101 |            9901123118|56B30     |I9902          |        |SOC10MA2302SOCJ31|14            |9916           |Operation 2                             |          1 |               0 |            499.0 |                    |                        |                      |        9908532289|
  |181993564            |011255486L1                          |02/09/2012  |2101 |            9901288820|56B30     |I9902          |        |SOC10MD2302SOCJ31|14            |9916           |Operation 1                             |          1 |               0 |            499.0 |                    |                399.599 |                      |        9908498544|
  |180885825            |011255486L1                          |02/08/2012  |2101 |            9901162239|56B30     |I9902          |        |SOC10MG2302SOCJ31|              |0150           |Operation 3                             |          1 |               0 |              0.0 |                    |                882.499 |                      |        9908099659|
  |180885825            |011255486L1                          |02/08/2012  |2101 |            9901162239|56B30     |I9902          |        |SOC10MG2302SOCJ31|14            |9916           |Operation 4                             |          1 |               0 |            544.0 |                    |                        |                      |        9908858514|
  |181638583            |990104460I0                          |02/10/2012  |2101 |            9902123289|56G99     |I9902          |        |SOC11MAR105SOCJ31|              |0160           |Operation 5                             |          1 |               0 |          1,160.0 |                    |                        |                      |        9914295010|
  |181681218            |990104460B0                          |02/08/2012  |2101 |            9902180981|56G99     |I9902          |        |SOC11MAR328SOCJ31|0             |9910           |Operation 6                             |          1 |               0 |            916.0 |                    |                        |                      |        9914621885|
  |181681036            |990104460I0                          |02/09/2012  |2101 |            9902180289|56G99     |I9902          |        |SOC11MAR108SOCJ31|              |0180           |Operation 8                             |          1 |               0 |              1.0 |                    |                        |                      |        9914619196|
  |189938054            |011255486A2                          |02/10/2012  |2101 |            9999206805|5AD99     |I9902          |        |RS08MJ2305SOCJ31 |              |0599           |Operation 8                             |          1 |               0 |              0.0 |                    |                        |                      |        9901316289|
  |181919894            |012984532A3                          |02/10/2012  |2101 |            9902511433|A199399Z  |I9902          |        |SOC12MCB101SOCJ31|0             |9935           |Operation 9                             |          1 |               0 |              0.5 |                    |                        |                      |        9916914233|
  |181919894            |012984532A3                          |02/10/2012  |2101 |            9902511433|A199399Z  |I9902          |        |SOC12MCB101SOCJ31|22            |9951           |Operation 10                            |          1 |               0 |           68.080 |                    |                        |                      |        9916914224|
  </code></pre>