<p>I haven't been able to find an understandable explanation of how to actually use Python's <code>itertools.groupby()</code> function.  What I'm trying to do is this:</p>
<ul>
<li>Take a list - in this case, the children of an objectified <code>lxml</code> element</li>
<li>Divide it into groups based on some criteria</li>
<li>Then later iterate over each of these groups separately.</li>
</ul>
<p>I've reviewed <a href="https://docs.python.org/3/library/itertools.html#itertools.groupby" rel="noreferrer">the documentation</a>, but I've had trouble trying to apply them beyond a simple list of numbers.</p>
<p>So, how do I use of <code>itertools.groupby()</code>?  Is there another technique I should be using?  Pointers to good &quot;prerequisite&quot; reading would also be appreciated.</p>

[{'answer_id': 783, 'author': 'Seb', 'author_id': 189, 'author_profile': 'https://Stackoverflow.com/users/189', 'pm_score': 6, 'selected': False, 'text': "<p>The example on the Python docs is quite straightforward:</p>\n\n<pre><code>groups = []\nuniquekeys = []\nfor k, g in groupby(data, keyfunc):\n    groups.append(list(g))      # Store group iterator as a list\n    uniquekeys.append(k)\n</code></pre>\n\n<p>So in your case, data is a list of nodes, <code>keyfunc</code> is where the logic of your criteria function goes and then <code>groupby()</code> groups the data.</p>\n\n<p>You must be careful to <strong>sort the data</strong> by the criteria before you call <code>groupby</code> or it won't work. <code>groupby</code> method actually just iterates through a list and whenever the key changes it creates a new group.</p>\n"}, {'answer_id': 7286, 'author': 'James Sulak', 'author_id': 207, 'author_profile': 'https://Stackoverflow.com/users/207', 'pm_score': 11, 'selected': True, 'text': "<p><strong>IMPORTANT NOTE:</strong> You have to <strong>sort your data</strong> first.</p>\n<hr />\n<p>The part I didn't get is that in the example construction</p>\n<pre><code>groups = []\nuniquekeys = []\nfor k, g in groupby(data, keyfunc):\n   groups.append(list(g))    # Store group iterator as a list\n   uniquekeys.append(k)\n</code></pre>\n<p><code>k</code> is the current grouping key, and <code>g</code> is an iterator that you can use to iterate over the group defined by that grouping key. In other words, the <code>groupby</code> iterator itself returns iterators.</p>\n<p>Here's an example of that, using clearer variable names:</p>\n<pre><code>from itertools import groupby\n\nthings = [(&quot;animal&quot;, &quot;bear&quot;), (&quot;animal&quot;, &quot;duck&quot;), (&quot;plant&quot;, &quot;cactus&quot;), (&quot;vehicle&quot;, &quot;speed boat&quot;), (&quot;vehicle&quot;, &quot;school bus&quot;)]\n\nfor key, group in groupby(things, lambda x: x[0]):\n    for thing in group:\n        print(&quot;A %s is a %s.&quot; % (thing[1], key))\n    print(&quot;&quot;)\n    \n</code></pre>\n<p>This will give you the output:</p>\n<blockquote>\n<p>A bear is a animal.<br />\nA duck is a animal.</p>\n<p>A cactus is a plant.</p>\n<p>A speed boat is a vehicle.<br />\nA school bus is a vehicle.</p>\n</blockquote>\n<p>In this example, <code>things</code> is a list of tuples where the first item in each tuple is the group the second item belongs to.</p>\n<p>The <code>groupby()</code> function takes two arguments: (1) the data to group and (2) the function to group it with.</p>\n<p>Here, <code>lambda x: x[0]</code> tells <code>groupby()</code> to use the first item in each tuple as the grouping key.</p>\n<p>In the above <code>for</code> statement, <code>groupby</code> returns three (key, group iterator) pairs - once for each unique key. You can use the returned iterator to iterate over each individual item in that group.</p>\n<p>Here's a slightly different example with the same data, using a list comprehension:</p>\n<pre><code>for key, group in groupby(things, lambda x: x[0]):\n    listOfThings = &quot; and &quot;.join([thing[1] for thing in group])\n    print(key + &quot;s:  &quot; + listOfThings + &quot;.&quot;)\n</code></pre>\n<p>This will give you the output:</p>\n<blockquote>\n<p>animals: bear and duck.<br />\nplants: cactus.<br />\nvehicles: speed boat and school bus.</p>\n</blockquote>\n"}, {'answer_id': 37252, 'author': 'nimish', 'author_id': 3926, 'author_profile': 'https://Stackoverflow.com/users/3926', 'pm_score': 6, 'selected': False, 'text': '<p>A neato trick with groupby is to run length encoding in one line:</p>\n\n<pre><code>[(c,len(list(cgen))) for c,cgen in groupby(some_string)]\n</code></pre>\n\n<p>will give you a list of 2-tuples where the first element is the char and the 2nd is the number of repetitions.</p>\n\n<p>Edit: Note that this is what separates <code>itertools.groupby</code> from the SQL <code>GROUP BY</code> semantics: itertools doesn\'t (and in general can\'t) sort the iterator in advance, so groups with the same "key" aren\'t merged.</p>\n'}, {'answer_id': 1573195, 'author': 'pedromanoel', 'author_id': 83284, 'author_profile': 'https://Stackoverflow.com/users/83284', 'pm_score': 3, 'selected': False, 'text': "<p>@CaptSolo, I tried your example, but it didn't work.</p>\n\n<pre><code>from itertools import groupby \n[(c,len(list(cs))) for c,cs in groupby('Pedro Manoel')]\n</code></pre>\n\n<p>Output:</p>\n\n<pre><code>[('P', 1), ('e', 1), ('d', 1), ('r', 1), ('o', 1), (' ', 1), ('M', 1), ('a', 1), ('n', 1), ('o', 1), ('e', 1), ('l', 1)]\n</code></pre>\n\n<p>As you can see, there are two o's and two e's, but they got into separate groups. That's when I realized you need to sort the list passed to the groupby function. So, the correct usage would be:</p>\n\n<pre><code>name = list('Pedro Manoel')\nname.sort()\n[(c,len(list(cs))) for c,cs in groupby(name)]\n</code></pre>\n\n<p>Output:</p>\n\n<pre><code>[(' ', 1), ('M', 1), ('P', 1), ('a', 1), ('d', 1), ('e', 2), ('l', 1), ('n', 1), ('o', 2), ('r', 1)]\n</code></pre>\n\n<p>Just remembering, if the list is not sorted, the groupby function <strong>will not work</strong>!</p>\n"}, {'answer_id': 14443477, 'author': 'user650654', 'author_id': 650654, 'author_profile': 'https://Stackoverflow.com/users/650654', 'pm_score': 5, 'selected': False, 'text': "<p>Another example:</p>\n<pre><code>for key, igroup in itertools.groupby(xrange(12), lambda x: x // 5):\n    print key, list(igroup)\n</code></pre>\n<p>results in</p>\n<pre><code>0 [0, 1, 2, 3, 4]\n1 [5, 6, 7, 8, 9]\n2 [10, 11]\n</code></pre>\n<p>Note that <code>igroup</code> is an iterator (a sub-iterator as the documentation calls it).</p>\n<p>This is useful for chunking a generator:</p>\n<pre><code>def chunker(items, chunk_size):\n    '''Group items in chunks of chunk_size'''\n    for _key, group in itertools.groupby(enumerate(items), lambda x: x[0] // chunk_size):\n        yield (g[1] for g in group)\n\nwith open('file.txt') as fobj:\n    for chunk in chunker(fobj):\n        process(chunk)\n</code></pre>\n<p>Another example of <code>groupby</code> - when the keys are not sorted.  In the following example, items in <code>xx</code> are grouped by values in <code>yy</code>.  In this case, one set of zeros is output first, followed by a set of ones, followed again by a set of zeros.</p>\n<pre><code>xx = range(10)\nyy = [0, 0, 0, 1, 1, 1, 0, 0, 0, 0]\nfor group in itertools.groupby(iter(xx), lambda x: yy[x]):\n    print group[0], list(group[1])\n</code></pre>\n<p>Produces:</p>\n<pre><code>0 [0, 1, 2]\n1 [3, 4, 5]\n0 [6, 7, 8, 9]\n</code></pre>\n"}, {'answer_id': 16427674, 'author': 'kiriloff', 'author_id': 1141493, 'author_profile': 'https://Stackoverflow.com/users/1141493', 'pm_score': 4, 'selected': False, 'text': '<p>I would like to give another example where groupby without sort is not working. Adapted from example by James Sulak</p>\n\n<pre><code>from itertools import groupby\n\nthings = [("vehicle", "bear"), ("animal", "duck"), ("animal", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]\n\nfor key, group in groupby(things, lambda x: x[0]):\n    for thing in group:\n        print "A %s is a %s." % (thing[1], key)\n    print " "\n</code></pre>\n\n<p>output is</p>\n\n<pre><code>A bear is a vehicle.\n\nA duck is a animal.\nA cactus is a animal.\n\nA speed boat is a vehicle.\nA school bus is a vehicle.\n</code></pre>\n\n<p>there are two groups with vehicule, whereas one could expect only one group</p>\n'}, {'answer_id': 20013133, 'author': 'RussellStewart', 'author_id': 2237635, 'author_profile': 'https://Stackoverflow.com/users/2237635', 'pm_score': 5, 'selected': False, 'text': "<p>WARNING:</p>\n\n<p>The syntax list(groupby(...)) won't work the way that you intend. It seems to destroy the internal iterator objects, so using</p>\n\n<pre><code>for x in list(groupby(range(10))):\n    print(list(x[1]))\n</code></pre>\n\n<p>will produce:</p>\n\n<pre><code>[]\n[]\n[]\n[]\n[]\n[]\n[]\n[]\n[]\n[9]\n</code></pre>\n\n<p>Instead, of list(groupby(...)), try [(k, list(g)) for k,g in groupby(...)], or if you use that syntax often,</p>\n\n<pre><code>def groupbylist(*args, **kwargs):\n    return [(k, list(g)) for k, g in groupby(*args, **kwargs)]\n</code></pre>\n\n<p>and get access to the groupby functionality while avoiding those pesky (for small data) iterators all together.</p>\n"}, {'answer_id': 31660194, 'author': 'Russia Must Remove Putin', 'author_id': 541136, 'author_profile': 'https://Stackoverflow.com/users/541136', 'pm_score': 3, 'selected': False, 'text': "<blockquote>\n  <p><strong>How do I use Python's itertools.groupby()?</strong></p>\n</blockquote>\n\n<p>You can use groupby to group things to iterate over. You give groupby an iterable, and a optional <strong>key</strong> function/callable by which to check the items as they come out of the iterable, and it returns an iterator that gives a two-tuple of the result of the key callable and the actual items in another iterable. From the help:</p>\n\n<pre><code>groupby(iterable[, keyfunc]) -&gt; create an iterator which returns\n(key, sub-iterator) grouped by each value of key(value).\n</code></pre>\n\n<p>Here's an example of groupby using a coroutine to group by a count, it uses a key callable (in this case, <code>coroutine.send</code>) to just spit out the count for however many iterations and a grouped sub-iterator of elements:</p>\n\n<pre><code>import itertools\n\n\ndef grouper(iterable, n):\n    def coroutine(n):\n        yield # queue up coroutine\n        for i in itertools.count():\n            for j in range(n):\n                yield i\n    groups = coroutine(n)\n    next(groups) # queue up coroutine\n\n    for c, objs in itertools.groupby(iterable, groups.send):\n        yield c, list(objs)\n    # or instead of materializing a list of objs, just:\n    # return itertools.groupby(iterable, groups.send)\n\nlist(grouper(range(10), 3))\n</code></pre>\n\n<p>prints</p>\n\n<pre><code>[(0, [0, 1, 2]), (1, [3, 4, 5]), (2, [6, 7, 8]), (3, [9])]\n</code></pre>\n"}, {'answer_id': 44617583, 'author': 'Arko', 'author_id': 7933904, 'author_profile': 'https://Stackoverflow.com/users/7933904', 'pm_score': 2, 'selected': False, 'text': '<p>One useful example that I came across may be helpful:</p>\n\n<pre><code>from itertools import groupby\n\n#user input\n\nmyinput = input()\n\n#creating empty list to store output\n\nmyoutput = []\n\nfor k,g in groupby(myinput):\n\n    myoutput.append((len(list(g)),int(k)))\n\nprint(*myoutput)\n</code></pre>\n\n<p>Sample input: 14445221</p>\n\n<p>Sample output: (1,1) (3,4) (1,5) (2,2) (1,1)</p>\n'}, {'answer_id': 45431237, 'author': 'Satyajit Das', 'author_id': 8137464, 'author_profile': 'https://Stackoverflow.com/users/8137464', 'pm_score': 3, 'selected': False, 'text': "<blockquote>\n  <p>Sorting and groupby</p>\n</blockquote>\n\n<pre><code>from itertools import groupby\n\nval = [{'name': 'satyajit', 'address': 'btm', 'pin': 560076}, \n       {'name': 'Mukul', 'address': 'Silk board', 'pin': 560078},\n       {'name': 'Preetam', 'address': 'btm', 'pin': 560076}]\n\n\nfor pin, list_data in groupby(sorted(val, key=lambda k: k['pin']),lambda x: x['pin']):\n...     print pin\n...     for rec in list_data:\n...             print rec\n... \no/p:\n\n560076\n{'name': 'satyajit', 'pin': 560076, 'address': 'btm'}\n{'name': 'Preetam', 'pin': 560076, 'address': 'btm'}\n560078\n{'name': 'Mukul', 'pin': 560078, 'address': 'Silk board'}\n</code></pre>\n"}, {'answer_id': 45873519, 'author': 'pylang', 'author_id': 4531270, 'author_profile': 'https://Stackoverflow.com/users/4531270', 'pm_score': 7, 'selected': False, 'text': '<p><code>itertools.groupby</code> is a tool for grouping items.</p>\n<p>From <a href="https://docs.python.org/3/library/itertools.html#itertools.groupby" rel="noreferrer">the docs</a>, we glean further what it might do:</p>\n<blockquote>\n<p><code># [k for k, g in groupby(\'AAAABBBCCDAABBB\')] --&gt; A B C D A B</code></p>\n<p><code># [list(g) for k, g in groupby(\'AAAABBBCCD\')] --&gt; AAAA BBB CC D</code></p>\n</blockquote>\n<p><code>groupby</code> objects yield key-group pairs where the group is a generator.</p>\n<p>Features</p>\n<ul>\n<li>A. Group consecutive items together</li>\n<li>B. Group all occurrences of an item, given a sorted iterable</li>\n<li>C. Specify how to group items with a <em>key function</em> <sup>*</sup></li>\n</ul>\n<p>Comparisons</p>\n<pre><code># Define a printer for comparing outputs\n&gt;&gt;&gt; def print_groupby(iterable, keyfunc=None):\n...    for k, g in it.groupby(iterable, keyfunc):\n...        print(&quot;key: \'{}\'--&gt; group: {}&quot;.format(k, list(g)))\n</code></pre>\n    \n<pre><code># Feature A: group consecutive occurrences\n&gt;&gt;&gt; print_groupby(&quot;BCAACACAADBBB&quot;)\nkey: \'B\'--&gt; group: [\'B\']\nkey: \'C\'--&gt; group: [\'C\']\nkey: \'A\'--&gt; group: [\'A\', \'A\']\nkey: \'C\'--&gt; group: [\'C\']\nkey: \'A\'--&gt; group: [\'A\']\nkey: \'C\'--&gt; group: [\'C\']\nkey: \'A\'--&gt; group: [\'A\', \'A\']\nkey: \'D\'--&gt; group: [\'D\']\nkey: \'B\'--&gt; group: [\'B\', \'B\', \'B\']\n\n# Feature B: group all occurrences\n&gt;&gt;&gt; print_groupby(sorted(&quot;BCAACACAADBBB&quot;))\nkey: \'A\'--&gt; group: [\'A\', \'A\', \'A\', \'A\', \'A\']\nkey: \'B\'--&gt; group: [\'B\', \'B\', \'B\', \'B\']\nkey: \'C\'--&gt; group: [\'C\', \'C\', \'C\']\nkey: \'D\'--&gt; group: [\'D\']\n\n# Feature C: group by a key function\n&gt;&gt;&gt; # islower = lambda s: s.islower()                      # equivalent\n&gt;&gt;&gt; def islower(s):\n...     &quot;&quot;&quot;Return True if a string is lowercase, else False.&quot;&quot;&quot;   \n...     return s.islower()\n&gt;&gt;&gt; print_groupby(sorted(&quot;bCAaCacAADBbB&quot;), keyfunc=islower)\nkey: \'False\'--&gt; group: [\'A\', \'A\', \'A\', \'B\', \'B\', \'C\', \'C\', \'D\']\nkey: \'True\'--&gt; group: [\'a\', \'a\', \'b\', \'b\', \'c\']\n</code></pre>\n<p>Uses</p>\n<ul>\n<li><a href="https://stackoverflow.com/questions/8181513/finding-and-grouping-anagrams-by-python">Anagrams</a> (<a href="https://github.com/vterron/EuroPython-2016/blob/master/kung-fu-itertools.ipynb" rel="noreferrer">see notebook</a>)</li>\n<li><a href="https://stackoverflow.com/a/45991797/4531270">Binning</a></li>\n<li><a href="https://naiquevin.github.io/a-look-at-some-of-pythons-useful-itertools.html" rel="noreferrer">Group odd and even numbers</a></li>\n<li><a href="https://stackoverflow.com/questions/5695208/group-list-by-values">Group a list by values</a></li>\n<li><a href="https://stackoverflow.com/questions/56796328/how-to-delete-repeat-elements-in-this-list">Remove duplicate elements</a></li>\n<li><a href="https://stackoverflow.com/a/44792205/4531270">Find indices of repeated elements in an array</a></li>\n<li><a href="https://stackoverflow.com/a/42677465/4531270">Split an array into n-sized chunks</a></li>\n<li><a href="https://stackoverflow.com/a/41993784/4531270">Find corresponding elements between two lists</a></li>\n<li><a href="https://youtu.be/iCrOGS1QlB8?t=27m27s" rel="noreferrer">Compression algorithm</a> (<a href="https://github.com/vterron/EuroPython-2016/blob/master/kung-fu-itertools.ipynb" rel="noreferrer">see notebook</a>)/<a href="https://stackoverflow.com/questions/25537028/run-length-encoding-in-python-with-list-comprehension/25537050#25537050">Run Length Encoding</a></li>\n<li><a href="https://youtu.be/iCrOGS1QlB8?t=29m11s" rel="noreferrer">Grouping letters by length, key function</a> (<a href="https://github.com/vterron/EuroPython-2016/blob/master/kung-fu-itertools.ipynb" rel="noreferrer">see notebook</a>)</li>\n<li><a href="https://youtu.be/iCrOGS1QlB8?t=30m19s" rel="noreferrer">Consecutive values over a threshold</a> (<a href="https://github.com/vterron/EuroPython-2016/blob/master/kung-fu-itertools.ipynb" rel="noreferrer">see notebook</a>)</li>\n<li><a href="https://stackoverflow.com/questions/4628333/converting-a-list-of-integers-into-range-in-python">Find ranges of numbers in a list</a> or <a href="https://stackoverflow.com/questions/2154249/identify-groups-of-continuous-numbers-in-a-list">continuous items</a> (see <a href="https://docs.python.org/release/2.4.4/lib/itertools-example.html" rel="noreferrer">docs</a>)</li>\n<li><a href="https://stackoverflow.com/questions/46928922/pythonic-way-to-find-all-potential-longest-sequence">Find all related longest sequences</a></li>\n<li><a href="https://stackoverflow.com/a/46752214/4531270">Take consecutive sequences that meet a condition</a> (<a href="https://stackoverflow.com/a/46432991/4531270">see related post</a>)</li>\n</ul>\n<p><em>Note: Several of the latter examples derive from Víctor Terrón\'s PyCon <a href="https://www.youtube.com/watch?v=iCrOGS1QlB8" rel="noreferrer">(talk)</a> <a href="https://www.youtube.com/watch?v=4ZIxcdREYVc" rel="noreferrer">(Spanish)</a>, &quot;Kung Fu at Dawn with Itertools&quot;. See also the <code>groupby </code><a href="https://github.com/python/cpython/blob/6cca5c8459cc439cb050010ffa762a03859d3051/Modules/itertoolsmodule.c#L11-L375" rel="noreferrer">source code</a> written in C.</em></p>\n<p><sup>* A function where all items are passed through and compared, influencing the result.  Other objects with key functions include  <code>sorted()</code>, <code>max()</code> and <code>min()</code>.</sup></p>\n<hr />\n<p>Response</p>\n<pre><code># OP: Yes, you can use `groupby`, e.g. \n[do_something(list(g)) for _, g in groupby(lxml_elements, criteria_func)]\n</code></pre>\n'}, {'answer_id': 61048516, 'author': 'Tiago', 'author_id': 9726459, 'author_profile': 'https://Stackoverflow.com/users/9726459', 'pm_score': 3, 'selected': False, 'text': '<p>This basic implementation helped me understand this function. Hope it helps others as well:</p>\n\n<pre class="lang-py prettyprint-override"><code>arr = [(1, "A"), (1, "B"), (1, "C"), (2, "D"), (2, "E"), (3, "F")]\n\nfor k,g in groupby(arr, lambda x: x[0]):\n    print("--", k, "--")\n    for tup in g:\n        print(tup[1])  # tup[0] == k\n</code></pre>\n\n<pre><code>-- 1 --\nA\nB\nC\n-- 2 --\nD\nE\n-- 3 --\nF\n</code></pre>\n'}, {'answer_id': 68091577, 'author': 'andrewdotn', 'author_id': 14558, 'author_profile': 'https://Stackoverflow.com/users/14558', 'pm_score': 3, 'selected': False, 'text': "<p>Sadly I don’t think it’s advisable to use <code>itertools.groupby()</code>. It’s just too hard to use safely, and it’s only a handful of lines to write something that works as expected.</p>\n<pre><code>def my_group_by(iterable, keyfunc):\n    &quot;&quot;&quot;Because itertools.groupby is tricky to use\n\n    The stdlib method requires sorting in advance, and returns iterators not\n    lists, and those iterators get consumed as you try to use them, throwing\n    everything off if you try to look at something more than once.\n    &quot;&quot;&quot;\n    ret = defaultdict(list)\n    for k in iterable:\n        ret[keyfunc(k)].append(k)\n    return dict(ret)\n</code></pre>\n<p>Use it like this:</p>\n<pre><code>def first_letter(x):\n    return x[0]\n\nmy_group_by('four score and seven years ago'.split(), first_letter)\n</code></pre>\n<p>to get</p>\n<pre><code>{'f': ['four'], 's': ['score', 'seven'], 'a': ['and', 'ago'], 'y': ['years']}\n</code></pre>\n"}, {'answer_id': 69783850, 'author': 'Ankit Gupta', 'author_id': 7864006, 'author_profile': 'https://Stackoverflow.com/users/7864006', 'pm_score': 2, 'selected': False, 'text': '<pre><code>from random import randint\nfrom itertools import groupby\n\n l = [randint(1, 3) for _ in range(20)]\n\n d = {}\n for k, g in groupby(l, lambda x: x):\n     if not d.get(k, None):\n         d[k] = list(g)\n     else:\n         d[k] = d[k] + list(g)\n</code></pre>\n<p>the code above shows how groupby can be used to group a list based on the lambda function/key supplied. The only problem is that the output is not merged, this can be easily resolved using a dictionary.</p>\n<p>Example:</p>\n<pre><code>l = [2, 1, 2, 3, 1, 3, 2, 1, 3, 3, 1, 3, 2, 3, 1, 2, 1, 3, 2, 3]\n</code></pre>\n<p>after applying groupby the result will be:</p>\n<pre><code>for k, g in groupby(l, lambda x:x):\n    print(k, list(g))\n\n2 [2]\n1 [1]\n2 [2]\n3 [3]\n1 [1]\n3 [3]\n2 [2]\n1 [1]\n3 [3, 3]\n1 [1]\n3 [3]\n2 [2]\n3 [3]\n1 [1]\n2 [2]\n1 [1]\n3 [3]\n2 [2]\n3 [3]\n</code></pre>\n<p>Once a dictionary is used as shown above following result is derived which can be easily iterated over:</p>\n<pre><code>{2: [2, 2, 2, 2, 2, 2], 1: [1, 1, 1, 1, 1, 1], 3: [3, 3, 3, 3, 3, 3, 3, 3]}\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/773', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/207/']