title: how to use column of tokenized sentences to further tokenize into words
body: <p>i have tokenized a text in a column into a new column 'token_sentences' of sentence tokens.
  i want to use 'token_sentences' column to create a new column 'token_words' containing tokenized words.</p>
  
  <p>df i am using</p>
  
  <pre><code>article_id      article_text                                       
  1           Maria Sharapova has basically no friends as te...   
  2           Roger Federer advance...    
  3           Roger Federer has revealed that organisers of ...   
  4           Kei Nishikori will try to end his long losing ...
  </code></pre>
  
  <p>added <code>token_sentences</code> column</p>
  
  <pre><code>article_id      article_text                                      token_sentences                          
  1           Maria Sharapova has basically no friends as te...    [Maria Sharapova has basically no friends as te    
  2           Roger Federer advance...                             [Roger Federer advance...
  3           Roger Federer has revealed that organisers of ...    [Roger Federer has revealed that organisers of...
  4           Kei Nishikori will try to end his long losing ...    [Kei Nishikori will try to end his long losing...
  
  </code></pre>
  
  <p>which is a list of sentences in every row.
  i am unable to flatten the list in <code>token_sentences</code> column to be able to used in the next step</p>
  
  <p>i want use <code>token_sentences</code> column 
  to make the df look like </p>
  
  <pre><code>article_id  article_text    token_sentences                         token_words                       
  1           Maria...        ["Maria Sharapova..",["..."]]           [Maria, Sharapova, has, basically, no, friends,...]       
  2           Roger...        ["Roger Federer advanced  ...",["..."]] [Roger,Federer,...]
  3           Roger...        ["Roger Federer...",["..."]]            [Roger ,Federer,...]
  4           Kei ...         ["Kei Nishikori will try...",["..."]]   [Kei,Nishikori,will,try,...]
  
  </code></pre>