title: How to create a pandas dataframe of word tokens from existing dataframe column of strings?
body: <p>I have a pandas dataframe <strong>df</strong> of the form:</p>
  
  <pre><code>df = pd.DataFrame.from_dict({'ID':[1,2,3], \
  'Strings':['Hello, how are you?', 'Nice to meet you!', 'My name is John.']})
  </code></pre>
  
  <p>I want to tokenize the <em>Strings</em> column and create a new data frame <strong>new_df</strong>:</p>
  
  <pre><code>Sentence    Word
     0        Hello
     0        ,
     0        how
     0        are
     0        you
     0        ?
     1        Nice
     1        to
     1        meet
     1        you
     1        .
     2        My
     2        name
     2        is
     2        John
     2        .
  </code></pre>
  
  <p>I know for tokenization I can possibly use <a href="https://www.nltk.org/api/nltk.tokenize.html" rel="nofollow noreferrer">nltk.word_tokenize()</a> for evert string in <strong>df</strong>, but how do I get from that point to <strong>new_df</strong> in a manner that is efficient?</p>