Importing text into PANDAS and counting certain words Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 

Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns contain text posts of var length. I then loop through each of these rows and query the post against an imported function that returns a specific counter of variable length. The counter tells me about the presence of certain words in the text. On average the func takes less than  1 ms to return this counter. (Timer for the "Func" inserted at the end to prove this)

Overheads: The code i'm looking to improve is the loop. I accept a certain level of overhead with the "func" which can't be improved at this minute. I have considered looking at Spark or Dask to parallelize the loop and speed up the process. Suggestions are welcome

#Import data
import pandas as pd 
from func import func 
data = pd.read_csv('Dataset.csv')

print(len(data))
&gt;&gt; 1900000

print(data.columns)
&gt;&gt; Index(['type', 'body'], dtype='object')


#Create new DF
data2 = pd.Dataframe()

for post in data['post']:
    post = str(post)
    scores = func.countWords(posts)
    data2 = data2.append(scores,ignore_index=True)

print(scores)
&gt;&gt; Counter({0: 306,
     1: 185,
     2: 61,
     45: 31,
     87: 23,
     92: 5,
     94: 3,
     102: 30,})


 import time
 start = time.time()
 score = func.countWords("Slow down Sir, you're going to give yourself skin faliure!")
 end = time.time()
 print(end - start)
 &gt;&gt; 0.0019948482513427734
Several errors:


  I then loop through each of these loops


I take it you mean "I then loop through each of these columns".


  for post in data['post]:


missing end quote mark


  scores = Func.countWords(posts)


You imported func (lowercase) and now you're calling Func (uppercase)


  data2 = data2.append(scores,ignore_index=True)


append should take a row-type object. If the function returns a numeric, then you shouldn't be appending it. Instead you can do:

def post_to_count(post):
      return func.countWord(str(post))

scores = data['post'].apply(post_to_count)