title: I'm trying to get word frequency on tweet text from a csv file using pandas value counts
body: <p>Here's my code:</p>
  <pre><code>import nltk
  from nltk.corpus import stopwords
  from nltk.tokenize import RegexpTokenizer
  from nltk.stem import WordNetLemmatizer
  import pandas as pd
  import numpy as np
  import openpyxl
  import string
  
  tokenizer = RegexpTokenizer(r'\w+')
  lemmatizer = WordNetLemmatizer()
  
  
  def remove_stopwords(df_text):
      words = [w for w in df_text if w not in stopwords.words('english')]
      return words
  
  def word_lemmatizer(df_text):
      lem_text = [lemmatizer.lemmatize(i) for i in df_text]
      return lem_text
  
  #works fine from here
  df = pd.read_csv('amazonfresh-test.csv', encoding='utf-8', converters={'text': str})
  
  
  df['text'].apply(lambda x: tokenizer.tokenize(x.lower()))
  df['text'].apply(lambda x: remove_stopwords(x))
  df['text'].apply(lambda x: word_lemmatizer(x))
  
  #to here
  
  #this is where I have issues
  data_count = df['test'].apply(pd.value_counts())
  
  data_count.to_excel(&quot;amazonfresh-test.xlsx&quot;)
  </code></pre>
  <p>It takes forever to run and I'm simply trying to strip and split the strings in each row of a text column and then have the overall word counts to show word frequency.</p>
  <p><a href="https://i.stack.imgur.com/sIUhm.png" rel="nofollow noreferrer">Here is what the csv looks like:</a></p>
  <p><a href="https://i.stack.imgur.com/B6fIC.png" rel="nofollow noreferrer">Here is the new csv with words in their own cell, still struggling to get the value_count on this.</a></p>