title: Get each unique word in a csv file tokenized
body: <p><a href="https://i.stack.imgur.com/53rEL.png" rel="nofollow noreferrer">Here is the CSV table</a>There are two columns in a CSV table. One is summaries and the other one is texts. Both columns were typeOfList before I combined them together, converted to data frame and saved as a CSV file. BTW, the texts in the table have already been cleaned (removed all marks and converted to lower cases):</p>
  
  <p>I want to loop through each cell in the table, split summaries and texts into words and tokenize each word. How can I do it?</p>
  
  <p>I tried with python CSV reader and df.apply(word_tokenize). I tried also newList=set(summaries+texts), but then I could not tokenize them. 
  Any solutions to solve the problem, no matter of using CSV file, data frame or list. Thanks for your help in advance!</p>
  
  <p>note: The real table has more than 50,000 rows.</p>
  
  <p>===some update== </p>
  
  <p>here is the code I have tried. </p>
  
  <pre><code>import pandas as pd
  data= pd.read_csv('test.csv')
  
  data.head()
  
  newTry=data.apply(lambda x: " ".join(x), axis=1)
  type(newTry)
  
  print (newTry)
  
  import nltk
  
  for sentence in newTry: 
      new=sentence.split() 
  
      print(new)
   print(set(new))
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/7g1MN.png" rel="nofollow noreferrer">enter image description here</a></p>
  
  <p>Please refer to the output in the screenshot. There are duplicate words in the list, and some square bracket. How should I removed them? I tried with set, but it gives only one sentence value.</p>