Best approach to doing analysis of sets of tags?
Question: I have thousands of survey responses that have been tagged according to the
content of the response. Each response can have one tag or many (up to 20),
and the tags are independent of one another rather than being structured into
category-subcategory or something.

I want to be able to do analysis like the following:

  * How many instances of a given tag are there?
  * Which tags occur most frequently overall?
  * Where tag X is present, which other tags appear along with it most frequently?
  * List of all tags with the count of each next to it
  * Select subsets of the data to do similar analysis on (by country, for example)

The people I'm working with have traditionally tackled everything in Excel
(general business strategy consulting work), and that won't work in this case.
Their response is to change the project framework to something that Excel can
handle in a pivot table, but it would be so much better if we could use more
robust tools that allow for more sophisticated relationships.

I've been learning SQLite but am starting to fear that the kinds of things I
want to do will be pretty complicated.

I've also been learning Python (for unrelated reasons) and am kind of
wondering if an ORM tool and some Python code might be the better way to go.

And then there's something like Access (which I don't have but would possibly
be willing to get if it's a sweet spot for this kind of thing).

In summary, I'd love to know how hard these kinds of analysis would be to do
overall and which tools would best be suited for the job. I'm completely open
to the idea that I'm thinking about some of or all of the problem in a way
that's backwards and would welcome any advice on any aspect of what I've
written here.


Answer: Collect all tags into a list and use the python collections.Counter and
associated methods to get the frequencies and a host of other statistics. Just
like this

    
    
    >>> from collections import Counter
    >>> x=['java', 'python', 'scheme', 'scheme', 'scheme', 'python', 'go', 'go', 'c',
    ... 'c']
    >>> freqs = Counter(x)
    >>> freqs.most_common(1)
    [('scheme', 3)]
    >>> 
    

