title: Dataflow Pipeline Slow
body: <p>My Dataflow pipeline is running extremely slow. Its processing approximately 4 elements/2 with 30 worker threads. A single local machine running the same operations (but not in the dataflow framework) is able to process 7 elements/s. The script is written in Python. Data is read from BigQuery.</p>
  
  <p>The workers are n1-standard, and all look to be at 100% CPU utilization.</p>
  
  <p>The operations contained within the combine are:</p>
  
  <ol>
  <li>tokenizes the record and applies stop word filtering (nltk)</li>
  <li>stem the word (nltk)</li>
  <li>lookup the word in a dictionary</li>
  <li>increment the count of said word in a dictionary</li>
  </ol>
  
  <p>Each record is approximately 30-70 KB.
  Total number of records is ~ 9,000,000 from BigQuery (Log shows all records have been exported successfully)</p>
  
  <p>With 30 worker threads, I expect the throughput to be a lot higher than my single local machine and certainly not half as fast.</p>
  
  <p>What could the problem be?</p>
  
  <p><a href="https://i.stack.imgur.com/tHMnY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/tHMnY.png" alt="GCP Console Screenshot"></a></p>