title: fast and concurrent algorithm of frequency calculation in elixir
body: <p>I have two big lists that their item's lengths isn't constant. Each list include millions items.
  And I want to count frequency of items of <code>first list</code> in <code>second list</code>!</p>
  
  <p>For example:</p>
  
  <pre><code>a = [[c, d], [a, b, e]]
  b = [[a, d, c], [e, a, b], [a, d], [c, d, a]]
  
  # expected result of calculate_frequency(a, b) is %{[c, d] =&gt; 2, [a, b, e] =&gt; 1} Or [{[c, d], 2}, {[a, b, e], 1}]
  </code></pre>
  
  <p>Due to the large size of the lists, I would like this process to be done concurrently.
  So I wrote this function:</p>
  
  <pre><code>  def calculate_frequency(items, data_list) do
      items
      |&gt; Task.async_stream(
        fn item -&gt;
          frequency =
            data_list
            |&gt; Enum.reduce(0, fn data_row, acc -&gt;
              if item -- data_row == [] do
                acc + 1
              else
                acc
              end
            end)
  
          {item, frequency}
        end,
        ordered: false
      )
      |&gt; Enum.reduce([], fn {:ok, merged}, merged_list -&gt; [merged | merged_list] end)
    end
  </code></pre>
  
  <p>But this algorithm is slow. What should I do to make it fast?</p>
  
  <p><strong>PS</strong>: Please do not consider the type of inputs and outputs, the speed of execution is important.</p>