title: Transform dataframes into matrix in Python for hashtag sets
body: <p>Here is my dataset 'new.csv'. Also I post a glance overview here:</p>
  <p><a href="https://drive.google.com/file/d/17xbwgp9siPuWsPBN5rUL9VSYwl7eU0ca/view?usp=sharing" rel="nofollow noreferrer">https://drive.google.com/file/d/17xbwgp9siPuWsPBN5rUL9VSYwl7eU0ca/view?usp=sharing</a></p>
  <pre><code>    Dominant_Topic  Hashtags
  0       4.0         [#boycottmulan, #blacklivesmatter]
  1       8.0         []
  2       8.0         [#blacklivesmatter, #protests]
  3       4.0         [#blacklivesmatter, #swoleinsolidarity]
  4       4.0         [#starlink]
  ... ... ...
  15995   4.0         [#verizon5gaccess, #oscars, #verizon5gaccess, #sweepstakes, #glennclose]
  15996   5.0         [#blacklivesmatter, #lifewithkellykhumalo, #bushiri, #vodacomnxtlvl, #amakhosi4life]
  15997   8.0         [#blm, #blacklivesmatter, #trumpkillsamericans]
  15998   3.0         [#tmobiletrucks, #str, #allin, #tmogoeslocal]
  15999   0.0         [#tdcsvm, #officialtdcent, #dmv, #blacklivesmatter, #fox5dc]
  </code></pre>
  <p>My target is to drop the all hashtag counts less than, let's say, 80 times in the datasets 'new' and use the rest hashtags to form matrix (from column 4) like follows:</p>
  <p><a href="https://i.stack.imgur.com/J1OzS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/J1OzS.png" alt="enter image description here" /></a></p>
  <p>As you can see, from column 4, it starts to code 1 or 0 to indicate if a given hashtag (the remained hashtags that appeared more than 80 times in the dataset) exist in column &quot;Hashtags&quot;</p>
  <p>I made a hashtage counts as follows:</p>
  <pre><code>hashtags_list_new = new.loc[new.Hashtags.apply(lambda hashtags_list:hashtags_list !=[]),['Hashtags']]
  
  # which hashtags were popular? 
  # create dataframe where each use of hashtag gets its own row
  flattened_hashtags_new = pd.DataFrame([hashtag for hashtags_list in hashtags_list_new.Hashtags for hashtag in hashtags_list],columns=['Hashtags'])
  
  # count of appearances of each hashtag
  popular_hashtags_new =flattened_hashtags_new.groupby('Hashtags').size().reset_index(name='counts')\
                                    .sort_values('counts',ascending=False)\
                                      .reset_index(drop=True)
  popular_hashtags_new
  </code></pre>
  <p>The result is :</p>
  <pre><code>        Hashtags            counts
  0       #blacklivesmatter   11379
  1       #blm                1022
  2       #jacobblake         565
  3       #verizon5gaccess    510
  4       #sweepstakes        496
  ... ... ...
  1484    #idolportraits      11
  1485    #augmentedreality   11
  1486    #smartphones        11
  1487    #ar                 11
  1488    #cx                 11
  </code></pre>
  <p>But I have no idea on how to get my target. Can anyone help me to solve it? Thank you for attention.</p>