How to cluster label (in a multilabel classification problem) which mostly appear together in a class To cluster label (in a multilabel classification problem) which mostly appear together in a dataframe? For example, I have this dataframe:
text     |   genre
===========================
text 1   | [action,mistery,horror,thriller]
text 2   | [drama,romance]
text 3   | [comedy,drama,romance]
text 4   | [scifi,mystery,horror,thriller]
text 5   | [horror,mystery,thriller]

How can I cluster the tag that often appear together? For example, genre &quot;mystery&quot;,&quot;horror&quot;,&quot;thriller&quot; often appear together (3 times), genre &quot;drama&quot;,&quot;romance&quot; often appears together (2 times).
The difficulty here is the design: if one just represents every &quot;class&quot; as an instance with a vector of its co-occurrences with other &quot;classes&quot;, then the clustering will only group the full group of instances/classes by how similar they are with other groups.
Instead I think that an instance should represent a pair of &quot;classes&quot;, using the PMI as a measure of how strongly they are associated.
An advanced option would be to represent each &quot;class&quot; as a node in a graph and add an edge between any two classes which occur together, weighted by their PMI value. Then use a graph-clustering algorithm to group the &quot;classes&quot;.